Decoding apparatus and method for decoding transform block from bitstream, encoding apparatus and method for encoding transform block in bitstream, and storage medium

Abstract

The present disclosure discloses a decoding apparatus and method for decoding a transform block from a bitstream, an encoding apparatus and method for encoding a transform block in a bitstream, and a storage medium. A system and method for decoding a sub-block of residual coefficients of a transform block from a video bitstream is also disclosed. The method comprises: determining whether sign bit hiding is used for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block; if no symbol bit hiding is used, decoding symbol bits of which the number is equal to the number of significant coefficients in the sub-block; and decoding the sub-block by reconstructing residual coefficients of the sub-block using the decoded sign bits.

Description

[0001] (This application is a divisional application of an application with a filing date of November 23, 2020, application number 2020800961394, and invention name “Method, device and system for encoding and decoding video sample blocks”.) Technical Field

[0002] The present invention generally relates to digital video signal processing, and more particularly to a method, device and system for encoding and decoding video sample blocks. The present invention also relates to a computer program product comprising a computer readable medium having recorded thereon a computer program for encoding and decoding video sample blocks. Background Art

[0003] There are currently many applications for video encoding, including applications for transmitting and storing video data. Many video encoding standards have also been developed and other video encoding standards are currently under development. Recent advances in video coding standardization have led to the formation of a group known as the "Joint Video Experts Group" (JVET). The Joint Video Experts Group (JVET) includes members of Study Group 16, Question 6 (SG16 / Q6) of the Telecommunication Standardization Sector (ITU-T) of the International Telecommunication Union (ITU), also known as the "Video Coding Experts Group" (VCEG); and members of the International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Group Association 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the "Moving Picture Experts Group" (MPEG).

[0004] The Joint Video Experts Team (JVET) issued a call for proposals (CfP) and analyzed the responses at its 10th meeting in San Diego, USA. The responses submitted showed that the video compression capabilities are significantly better than those of the current state-of-the-art video compression standard, namely "High Efficiency Video Coding" (HEVC). Based on this excellent performance, it was decided to start a project to develop a new video compression standard named "Versatile Video Coding" (VVC). It is expected that VVC will address the continued demand for even higher compression performance, especially as the capabilities of video formats increase (e.g., with higher resolutions and higher frame rates), and the growing market demand for service provision over WANs, where bandwidth costs are relatively high. At the same time, VVC must be implementable in contemporary silicon processes and provide an acceptable trade-off between the achieved performance and the implementation cost (e.g., in terms of silicon area, CPU processor load, memory utilization, and bandwidth).

[0005] Video data includes a sequence of frames of image data, each of which includes one or more color channels. Typically, one primary color channel and two secondary color channels are required. The primary color channel is often referred to as the "brightness" channel, and (one or more than one) secondary color channels are often referred to as the "chrominance" channel. Although video data is usually displayed in an RGB (red-green-blue) color space, this color space has a high degree of correlation between the three corresponding components. The video data representation seen by the encoder or decoder usually uses a color space such as YCbCr. YCbCr concentrates the luminous brightness (mapped to "brightness" according to the transfer function) in the Y (primary) channel, and concentrates the chrominance in the Cb and Cr (secondary) channels. In addition, the Cb and Cr channels can be spatially sampled at a lower rate (subsampling) than the brightness channel (for example, half in the horizontal direction and half in the vertical direction (referred to as the "4:2:0 chroma format")). The 4:2:0 chroma format is often used in (such as Internet video streaming, broadcast television, and Blu-ray TM 4:2:2 chroma format is used in "consumer" applications (such as storage on disks). Subsampling the Cb and Cr channels horizontally at half the rate, rather than vertically, is referred to as the "4:2:2 chroma format." The 4:2:2 chroma format is commonly used in professional applications (including capturing scripts for film production, etc.). The higher sampling rate of the 4:2:2 chroma format makes the resulting video more resilient to editing operations such as color grading. Before distribution to consumers, 4:2:2 chroma format material is typically converted to 4:2:0 chroma format and then encoded for distribution to consumers. In addition to the chroma format, video is also characterized by resolution and frame rate. Example resolutions are ultra-high definition (UHD) with a resolution of 3840×2160 or "8K" with a resolution of 7680×4320, and example frame rates are 60 or 120 Hz. Luma sampling rates can range from approximately 500 megasamples per second to several gigasamples per second. For the 4:2:0 chroma format, the sampling rate of each chroma channel is one quarter of the luma sampling rate, and for the 4:2:2 chroma format, the sampling rate of each chroma channel is half of the luma sampling rate.

[0006] The VVC standard is a "block-based" codec in which a frame is first divided into an array of square areas called "coding tree units" (CTUs). CTUs typically occupy a relatively large area, such as 128×128 luma samples. However, the area of ​​the CTUs at the right and lower edges of each frame may be smaller. Associated with each CTU is a "coding tree" for the luma channel and an additional coding tree for the chroma channel. The coding tree defines the decomposition of the area of ​​the CTU into a set of blocks (also called "coding blocks" (CBs)). A single coding tree can also specify blocks for both luma and chroma channels, in which case a set of juxtaposed coding blocks is called a "coding unit" (CU), i.e., each CU has a coding block for each color channel. CBs are processed in a specific order for encoding or decoding. Due to the use of a 4:2:0 chroma format, a CTU with a luma coding tree for a 128×128 luma sample area has a corresponding chroma coding tree for a 64×64 chroma sample area (juxtaposed with a 128×128 luma sample area). When a single coding tree is used for the luma channel and the chroma channel, a set of collocated blocks for a given region is often referred to as a "unit", such as the above-mentioned CU, as well as a "prediction unit" (PU) and a "transform unit" (TU). When a separate coding tree is used for a given region, the above-mentioned CB, as well as a "prediction block" (PB) and a "transform block" (TB) are used.

[0007] Despite the above distinction between a "unit" and a "block", the term "block" may be used as a general term for an area or region of a frame for which operations are applied to all color channels.

[0008] For each CU, a prediction unit (PU) ("prediction unit") of the content (sample values) of the corresponding region of the frame data is generated. If the PU is generated from sample values ​​in a previously signaled frame, the prediction is called inter-frame prediction. If the PU is generated from previous samples in the same frame, the prediction is called intra-frame prediction. In addition, a representation of the difference (or "residual" in the spatial domain) between the prediction and the content of the region seen at the input of the encoder is formed. The differences in the various color channels can be transformed and encoded into residual coefficient blocks, thereby forming one or more TUs for a given CU. The residual coefficients can be transformed by a transform such as a discrete cosine transform (DCT), a discrete sine transform (DST), or other transforms to produce a final block of transform coefficients of substantially decorrelated residual samples. Significant coding gain can be achieved by quantizing the transform coefficients. The quantized transform coefficients are then traversed in an order such as a reverse diagonal scan, and the individual coefficients are encoded by an entropy encoder. Entropy coding includes expressing individual coefficients according to syntax elements, and each syntax element is binarized. The binarized syntax elements may then be further encoded by a Context-Adaptive Binary Arithmetic Coder (CABAC), or passed to the bitstream ("bypass coding").

[0009] In some categories of video content, such as picture content, it may be advantageous to avoid a transform. If a transform is to be avoided, the residual coefficients are quantized, traversed, and encoded. Since the statistics of the residual coefficients are different from the statistics of the transform coefficients, it is often advantageous to encode the residual coefficients using a different process than the encoding process of the transform coefficients. Typical methods for encoding the residual coefficients include a "regular residual coding" (RRC) process and a "transform skip residual coding" (TSRC) process, where a specific one is selected for a block depending on whether a transform is performed.

[0010] In some use cases, it may be desirable to compress video data losslessly (i.e., without coding losses). A CU may be losslessly encoded by skipping both the transform step and the quantization step. In TSRC processing, quantization may be avoided by setting the "quantization parameter" to a value indicating no quantization. However, as indicated above, TSRC processing may only be applicable to video content categories (such as picture content, etc.). Therefore, forcing lossless encoding of video data to use TSRC processing may be suboptimal. It is desirable that lossless encoding have more flexible options available based on the statistics of the video data being encoded, while minimizing the amount of additional logic required to support the additional flexibility. Summary of the invention

[0011] It is an object of the present invention to substantially overcome or at least ameliorate one or more disadvantages of existing arrangements.

[0012] One aspect of the present invention provides a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method comprising: determining whether to use a sign bit hiding for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block; decoding a number of sign bits equal to the number of significant coefficients in the sub-block without using the sign bit hiding; and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0013] According to another aspect, sign bit hiding is used if the sign bit hiding flag has a value of true, the transform skip flag has a value of false, and a difference between a first significant position and a last significant position of the sub-block is greater than three.

[0014] According to another aspect, in a case where the sign bit hiding flag has a value of true and the transform skip flag has a value of true, sign bit hiding is not used.

[0015] According to another aspect, the method further includes: if it is determined that sign bit hiding is used, decoding a number of sign bits equal to the number of significant coefficients in the sub-block minus 1, and determining additional sign bits based on the sum of the parities of the significant coefficients of the sub-block.

[0016] Another aspect of the present invention provides a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method comprising: determining whether to use sign bit hiding for the sub-block, the determination being based on a value of a sign bit hiding flag and a value of a quantization parameter associated with the sub-block; decoding a number of sign bits equal to the number of significant coefficients in the sub-block when sign bit hiding is not used; and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0017] According to another aspect, in a case where the sign bit hiding flag has a value of true and the quantization parameter is equal to 4, sign bit hiding is not used.

[0018] According to another aspect, sign bit hiding is used if the sign bit hiding flag has a value of true, the quantization parameter is not equal to 4, and a difference between a first significant position and a last significant position of the subblock is greater than 3.

[0019] Another aspect of the present invention provides a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method comprising: determining whether to use sign bit hiding for the sub-block, the determination being based on a value of a sign bit hiding flag and a value of a TSRC disable flag; decoding a number of sign bits equal to the number of significant coefficients in the sub-block when sign bit hiding is not used; and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0020] According to another aspect, sign bit hiding is used if the sign bit hiding flag has a value of true, the TSRC disable flag has a value of false, and a difference between a first significant position and a last significant position of the sub-block is greater than three.

[0021] According to another aspect, in a case where the sign bit hiding flag has a value of true and the TSRC disable flag has a value of true, sign bit hiding is not used.

[0022] Another aspect of the present invention provides a non-transitory computer-readable medium having a computer program stored thereon for implementing a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method comprising: determining whether to use a sign bit hiding for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block; decoding a number of sign bits equal to the number of significant coefficients in the sub-block without using sign bit hiding; and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0023] Another aspect of the present invention provides a system comprising: a memory; and a processor, wherein the processor is configured to execute code stored on the memory to implement a method for decoding a sub-block of residual coefficients of a transform block from a video bitstream, the method comprising: determining whether to use sign bit hiding for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block; decoding a number of sign bits equal to the number of significant coefficients in the sub-block without using sign bit hiding; and decoding the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0024] Another aspect of the present invention provides a video decoder configured to: receive a sub-block of residual coefficients of a transform block from a video bitstream, determine whether to use sign bit hiding for the sub-block, the determination being based on a value of a transform skip flag determined for the sub-block and a value of a sign bit hiding flag associated with the sub-block; decode a number of sign bits equal to the number of significant coefficients in the sub-block without using sign bit hiding; and decode the sub-block by reconstructing the residual coefficients of the sub-block using the decoded sign bits.

[0025] Other aspects are also described. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] At least one embodiment of the present invention will now be described with reference to the following drawings and appendices, in which:

[0027] Figure 1 is a schematic block diagram illustrating a video encoding and decoding system;

[0028] Figure 2A and 2B Composition can be practiced Figure 1 A schematic block diagram of a general computer system for one or both of the video encoding and decoding systems;

[0029] Figure 3 is a schematic block diagram showing the functional modules of a video encoder;

[0030] Figure 4 is a schematic block diagram showing the functional modules of a video decoder;

[0031] Figure 5 is a schematic block diagram illustrating possible partitioning of a block into one or more blocks in a tree structure for general video coding;

[0032] Figure 6 A schematic diagram of a data flow for implementing a permissible partitioning of a block into one or more blocks in a tree structure of a general video coding;

[0033] Fig. 7A and 7B shows an example partitioning of a coding tree unit (CTU) into multiple coding units (CUs);

[0034] Fig. 8A Two levels of reverse diagonal scanning are shown;

[0035] Figure 8B Two levels of forward diagonal scanning are shown;

[0036] Fig. 9 A method for encoding a transformed block of residual coefficients is shown;

[0037] Fig.10A method for decoding a transformed block of residual coefficients is shown;

[0038] Fig.11 Shown as Fig. 9 A method for quantizing a transform block of residual coefficients performed by a method according to the present invention;

[0039] Fig.12 Shown as Fig.10 A method for scaling a transform block of quantized coefficients performed by a method according to the present invention;

[0040] Fig.13 Shown as Fig. 9 A method for encoding a sub-block of quantized coefficients according to the method of

[0041] Fig.14 Shown as Fig.10 A method for decoding a sub-block of quantized coefficients performed by a method. DETAILED DESCRIPTION

[0042] Where steps and / or features having the same reference numerals are referenced in any one or more of the accompanying drawings, these steps and / or features have the same function(s) or operation(s) for the purposes of this specification, unless otherwise intended.

[0043] As described above, it may be desirable to leverage existing building blocks of a codec to support lossless encoding. However, using TSRC processing exclusively for lossless encoding may yield suboptimal encoding performance, since there is no guarantee that the various classes of video data encoded in a lossless manner will exhibit the statistical properties for which the TSRC processing was designed. However, greater flexibility in the choice of advanced building blocks that can be used by lossless encoding allows for superior encoding performance with minimal additional complexity to the overall design.

[0044] Figure 11 is a schematic block diagram illustrating the functional modules of a video encoding and decoding system 100. The system 100 includes a source device 110 and a destination device 130. A communication channel 120 is used to communicate encoded video information from the source device 110 to the destination device 130. In some arrangements, one or both of the source device 110 and the destination device 130 may include a mobile phone handset or "smart phone", respectively, in which case the communication channel 120 is a wireless channel. In other arrangements, the source device 110 and the destination device 130 may include video conferencing equipment, in which case the communication channel 120 is typically a wired channel such as an Internet connection. In addition, the source device 110 and the destination device 130 may include any of a wide range of devices, including devices that support over-the-air television broadcasts, cable television applications, Internet video applications (including streaming), and applications that capture encoded video data on some computer-readable storage medium (such as a hard drive in a file server).

[0045] like Figure 1 As shown, source device 110 includes video source 112, video encoder 114, and transmitter 116. Video source 112 typically includes a source of captured video frame data (denoted as 113), such as a camera sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote camera sensor. Video source 112 may also be the output of a computer graphics card (e.g., display operating system and video output of various applications executed on a computing device (e.g., tablet computer)). Examples of source device 110 that may include a camera sensor as video source 112 include smart phones, video camcorders, professional video cameras, and web video cameras.

[0046] As further reference Figure 3 As described, video encoder 114 converts (or "encodes") captured frame data (indicated by arrow 113) from video source 112 into a bitstream (indicated by arrow 115). Bitstream 115 is transmitted by transmitter 116 as encoded video data (or "encoded video information") via communication channel 120. Bitstream 115 may also be stored in non-transitory storage device 122, such as "flash" memory or hard drive, until or in lieu of subsequent transmission via communication channel 120.

[0047] The destination device 130 includes a receiver 132, a video decoder 134, and a display device 136. The receiver 132 receives the encoded video data from the communication channel 120 and passes the received video data to the video decoder 134 as a bit stream (indicated by arrow 133). Then, the video decoder 134 outputs the decoded frame data (indicated by arrow 135) to the display device 136 to reproduce the video data. The decoded frame data 135 has the same chroma format as the frame data 113. Examples of the display device 136 include a cathode ray tube, a liquid crystal display (such as in a smart phone, a tablet computer, a computer monitor, or a stand-alone television). The functions of each of the source device 110 and the destination device 130 can also be embodied in a single device, examples of which include a mobile phone and a tablet computer.

[0048] Although example devices are described above, source device 110 and destination device 130 may each typically be configured within a general purpose computer system via a combination of hardware and software components. Figure 2A Such a computer system 200 is shown, which includes: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 that can be configured as a video source 112, and a microphone 280; and output devices including a printer 215, a display device 214 that can be configured as a display device 136, and a speaker 217. The computer module 201 can use an external modulator-demodulator (modem) transceiver device 216 to communicate with a communication network 220 via a connection 221. The communication network 220, which can represent a communication channel 120, can be a wide area network (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. In the case where the connection 221 is a telephone line, the modem 216 can be a traditional "dial-up" modem. Alternatively, in the case where the connection 221 is a high-capacity (e.g., cable or optical) connection, the modem 216 can be a broadband modem. Wireless modems can also be used to make wireless connections to the communication network 220. The transceiver device 216 may provide the functionality of the transmitter 116 and the receiver 132 , and the communication channel 120 may be embodied in the wiring 221 .

[0049] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 may have a semiconductor random access memory (RAM) and a semiconductor read-only memory (ROM). The computer module 201 also includes a plurality of input / output (I / O) interfaces, wherein the plurality of input / output (I / O) interfaces include: an audio-video interface 207 connected to a video display 214, a speaker 217, and a microphone 280; an I / O interface 213 connected to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and an optional joystick or other human-machine interface device (not shown); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio-video interface 207 to the computer monitor 214 is typically the output of a computer graphics card. In some implementations, the modem 216 may be built into the computer module 201, such as built into the interface 208. The computer module 201 also has a local network interface 211, which allows the computer system 200 to be connected to a local area communication network 222, known as a local area network (LAN), via a connection 223. Figure 2A As shown, the local area communication network 222 can also be connected to the wide area network 220 via the connection 224, wherein the local area communication network 222 usually includes a so-called "firewall" device or a device with similar functions. The local network interface 211 may include an Ethernet (Ethernet TM )Circuit card, Bluetooth TM ) wireless arrangement or IEEE 802.11 wireless arrangement; however, for the interface 211, a variety of other types of interfaces can be implemented. The local network interface 211 can also provide the functions of the transmitter 116 and the receiver 132, and the communication channel 120 can also be embodied in the local communication network 222.

[0050] I / O interfaces 208 and 213 may provide either or both serial and parallel connections, the former of which is typically implemented in accordance with the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided, and the storage device 209 typically includes a hard disk drive (HDD) 210. Other storage devices (not shown) such as floppy disk drives and tape drives may also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. For example, an optical disk (e.g., CD-ROM, DVD, Blu-ray Disc) may be used. TM)), portable memory devices such as USB-RAM, portable external hard drives and floppy disks, etc., as suitable sources of data for computer system 200. Generally, any of HDD 210, optical drive 212, networks 220 and 222 may also be configured to operate as video source 112, or as a destination for decoded video data to be stored for reproduction via display 214. Source device 110 and destination device 130 of system 100 may be embodied in computer system 200.

[0051] The components 205-213 of the computer module 201 typically communicate via an interconnect bus 204 and in a manner that results in conventional modes of operation of the computer system 200 known to those skilled in the relevant art. For example, the processor 205 is connected to the system bus 204 using a connection 218. Similarly, the memory 206 and the optical drive 212 are connected to the system bus 204 via a connection 219. Examples of computers on which the described arrangement may be practiced include IBM-PC and compatible machines, Sun SPARCstation, Apple Mac TM or similar computer system.

[0052] Where appropriate or desired, the video encoder 114 and the video decoder 134 and the methods described below may be implemented using the computer system 200. In particular, the video encoder 114, the video decoder 134 and the methods to be described may be implemented as one or more software applications 233 executable within the computer system 200. In particular, instructions 231 (see FIG. 233 ) executed within the computer system 200 may be used to execute the video encoder 114 and the video decoder 134. Figure 2B ) to implement the video encoder 114, the video decoder 134 and the steps of the method. The software instructions 231 may be formed into one or more code modules, each for performing one or more specific tasks. The software may also be split into two separate parts, with a first part and corresponding code modules performing the method, and a second part and corresponding code modules managing a user interface between the first part and a user.

[0053] For example, the software may be stored in a computer readable medium including a storage device as described below. The software is loaded from the computer readable medium into the computer system 200 and then executed by the computer system 200. A computer readable medium with such software or a computer program recorded on the computer readable medium is a computer program product. The use of the computer program product in the computer system 200 preferably implements an advantageous device for implementing the video encoder 114, the video decoder 134 and the method.

[0054] The software 233 is typically stored in the HDD 210 or the memory 206. The software is loaded into the computer system 200 from a computer readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 read by the optical drive 212.

[0055] In some examples, the application 233 is provided to the user in a manner encoded on one or more CD-ROMs 225 and read via corresponding drives 212, or alternatively, the application 233 can be read by the user from the network 220 or 222. Furthermore, the software can also be loaded into the computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to the computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray Discs, and the like. TM ), a hard disk drive, a ROM or integrated circuit, a USB memory, a magneto-optical disk, or a computer readable card such as a PCMCIA card, etc., regardless of whether these devices are internal or external to the computer module 201. Examples of temporary or non-tangible computer-readable transmission media that can also participate in providing software, applications, instructions and / or video data or encoded video data to the computer module 401 include: radio or infrared transmission channels and network connections to other computers or networked devices, and the Internet or intranet including email transmission and information recorded on websites.

[0056] The second part of the above-mentioned application program 233 and the corresponding code module can be executed to implement one or more graphical user interfaces (GUIs) to be drawn or otherwise presented on the display 214. Users and applications of the computer system 200 can operate the interface in a functionally applicable manner by typically operating the keyboard 202 and the mouse 203 to provide control commands and / or input to the applications associated with these (one or more) GUIs. Other forms of user interfaces that are functionally applicable can also be implemented, such as audio interfaces that utilize voice prompts output via the speaker 217 and user voice commands input via the microphone 280, etc.

[0057] Figure 2B is a detailed schematic block diagram of the processor 205 and the "memory" 234. The memory 234 represents Figure 2A A logical aggregation of all memory modules (including storage device 209 and semiconductor memory 206) that can be accessed by computer module 201 in.

[0058] When the computer module 201 is initially powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is usually stored in Figure 2A 249 of semiconductor memory 206. Hardware devices such as ROM 249 storing software are sometimes referred to as firmware. POST program 250 checks the hardware within computer module 201 to ensure proper operation, and typically checks processor 205, memory 234 (209, 206), and basic input-output system software (BIOS) module 251, which is also typically stored in ROM 249, for correct operation. Once POST program 250 runs successfully, BIOS 251 starts Figure 2A The hard disk drive 210 is started so that the boot loader 252 residing on the hard disk drive 210 is executed via the processor 205. This loads the operating system 253 into the RAM memory 206, where the operating system 253 starts to work on the RAM memory 206. The operating system 253 is a system-level application executable by the processor 205 to implement various high-level functions including processor management, memory management, device management, storage management, software application interface, and general user interface.

[0059] The operating system 253 manages the memory 234 (209, 206) to ensure that each process or application running on the computer module 201 has sufficient memory to execute without conflicting with memory allocated to other processes. In addition, the appropriate use of Figure 2A The different types of memory available in the computer system 200 are described so that each process can run efficiently. Therefore, the aggregate memory 234 is not intended to illustrate how to allocate specific segments of memory (unless otherwise specified), but rather to provide an overview of the memory accessible to the computer system 200 and how to use it.

[0060] like Figure 2B As shown, the processor 205 includes a plurality of functional modules, wherein the plurality of functional modules include a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248 sometimes referred to as a cache memory. The cache memory 248 typically includes a plurality of storage registers 244-246 in a register section. One or more internal buses 241 functionally interconnect these functional modules. The processor 205 also typically has one or more interfaces 242 for communicating with external devices via the system bus 204 using a connection 218. The memory 234 is connected to the bus 204 using a connection 219.

[0061] The application program 233 includes an instruction sequence 231 that may include conditional branch instructions and loop instructions. The program 233 may also include data 232 used when executing the program 233. The instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative size of the instructions 231 and the memory locations 228-230, a particular instruction may be stored in a single memory location, as described by the instruction shown in memory location 230. Alternatively, the instruction may be split into multiple parts, each stored in a separate memory location, as described by the instruction segments shown in memory locations 228 and 229.

[0062] Typically, the processor 205 is given a set of instructions, which are executed within the processor 205. The processor 205 awaits a subsequent input, which the processor 205 reacts to by executing another set of instructions. Each input may be provided from one or more of a plurality of sources, including data generated by one or more of the input devices 202, 203, data received from an external source via one of the networks 220, 222, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212 (all of which are described in detail in the accompanying drawings). Figure 2A Execution of a set of instructions may result in outputting data in some cases. Execution may also involve storing data or variables to memory 234.

[0063] The video encoder 114, the video decoder 134, and the method may use input variables 254 stored in corresponding memory locations 255, 256, 257 within the memory 234. The video encoder 114, the video decoder 134, and the method produce output variables 261 stored in corresponding memory locations 262, 263, 264 within the memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0064] refer to Figure 2B The processor 205, registers 244, 245, 246, arithmetic logic unit (ALU) 240 and control unit 239 work together to perform micro-operation sequences required to perform a "fetch, decode and execute" cycle for each instruction in the instruction set that constitutes the program 233. Each fetch, decode and execute cycle includes:

[0065] A fetch operation for fetching or reading an instruction 231 from a memory location 228 , 229 , 230 ;

[0066] a decode operation in which the control unit 239 determines which instruction was fetched; and

[0067] An execution operation, wherein in the execution operation, the control unit 239 and / or the ALU 240 executes the instruction.

[0068] Thereafter, further fetch, decode and execute cycles for the next instruction may be performed. Likewise, a store cycle may be performed, whereby the control unit 239 stores or writes a value to the memory location 232 .

[0069] To be explained Figures 9 to 14 Each step or sub-process in the method is associated with one or more segments of program 233, and is typically performed by the register units 244, 245, 247, ALU 240 and control unit 239 in the processor 205 working together to perform fetch, decode and execute cycles for each instruction in the instruction set of the segment of program 233.

[0070] Figure 3 A schematic block diagram showing functional modules of the video encoder 114 is illustrated. Figure 4 1 illustrates a schematic block diagram showing the functional modules of the video decoder 134. Typically, data is transferred between the functional modules within the video encoder 114 and the video decoder 134 in groups of samples or coefficients (such as partitioning of a block into fixed-size sub-blocks, etc.) or as an array. Figure 2A and 2B As shown, the video encoder 114 and the video decoder 134 may be implemented using a general purpose computer system 200, wherein various functional modules may be implemented using dedicated hardware within the computer system 200, using software executable within the computer system 200 (such as one or more software code modules of a software application 233 resident on a hard drive 210 and controlled by a processor 205 for execution, etc.). Alternatively, the video encoder 114 and the video decoder 134 may be implemented using a combination of dedicated hardware and software executable within the computer system 200. The video encoder 114, the video decoder 134, and the method may alternatively be implemented in dedicated hardware such as one or more integrated circuits that perform the functions or sub-functions of the method. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application specific standard product (ASSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or one or more microprocessors and associated memory. In particular, the video encoder 114 includes modules 310 - 386 , and the video decoder 134 includes modules 420 - 496 , where each of these modules may be implemented as one or more software code modules of the software application 233 .

[0071] although Figure 3 The video encoder 114 is an example of a universal video coding (VVC) video coding pipeline, but other video codecs may also be used to perform the processing stages described herein. The video encoder 114 receives captured frame data 113, such as a series of frames, each frame including one or more color channels. The frame data 113 may be in any chroma format, such as 4:0:0, 4:2:0, 4:2:2, or 4:4:4 chroma format. The block partitioner 310 first partitions the frame data 113 into CTUs, which are typically square in shape and configured to use a specific size of the CTU. The size of the CTU may be, for example, 64×64, 128×128, or 256×256 luminance samples. The block partitioner 310 also partitions each CTU into one or more CBs according to a luminance coding tree and a chrominance coding tree. CBs have various sizes and may include both square and non-square aspect ratios. In the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of two. Therefore, according to the luma coding tree and chroma coding tree of the CTU, according to the iterative progress on one or more blocks of the CTU, the current CB denoted as 312 is output from the block partitioner 310. Figure 5 and Figure 6 The options for partitioning a CTU into CBs are further described.

[0072] The CTUs resulting from the first partitioning of the frame data 113 may be scanned in a raster scan order and may be grouped into one or more "slices". A slice may be an "intra" (or "I") slice. An intra slice (I slice) indicates that each CU in the slice is intra predicted. Alternatively, a slice may be uni-predicted or bi-predicted ("P" or "B" slices, respectively), indicating the additional availability of uni-prediction and bi-prediction in the slice, respectively.

[0073] For each CTU, the video encoder 114 operates in two stages. In the first stage (referred to as the "search" stage), the block partitioner 310 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated "candidate" CB. The first stage involves testing various candidate CBs to select a CB that provides high compression efficiency and low distortion. The test typically involves Lagrangian optimization, whereby the candidate CBs are evaluated based on a weighted combination of rate (coding cost) and distortion (error relative to the input frame data 113). The "best" candidate CB (the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into the bitstream 115. Included in the evaluation of the candidate CBs is the option of using a CB for a given region, or further splitting the region according to various splitting options and encoding each smaller resulting region with a further CB, or even further splitting the region. Therefore, both the CB and the coding tree itself are selected in the search stage.

[0074] The video encoder 114 generates a prediction block (PB) indicated by arrow 320 for each CB (e.g., CB 312). PB 320 is a prediction of the content of the associated CB 312. Subtractor module 322 generates a difference (or "residual," which means that the difference is in the spatial domain) between PB 320 and CB 312, represented as 324. Residual 324 is the block size difference between corresponding samples in PB 320 and CB 312. Residual 324 is transformed, quantized, and represented as a transform block (TB) indicated by arrow 336. PB 320 and associated TB 336 are typically selected from one of a plurality of possible candidate CBs, for example, based on an assessed cost or distortion.

[0075] A candidate coding block (CB) is a CB obtained from one of the prediction modes available to the video encoder 114 for the associated PB and the resulting residual. Each candidate CB obtains one or more corresponding TBs. TB 336 is a quantized and transformed representation of the residual 324. When combined with the predicted PB in the video encoder 114, TB 336 reduces the difference between the decoded CB and the original CB 312 at the expense of additional signaling in the bitstream.

[0076] Thus, each candidate coding block (CB) (i.e., a prediction block (PB) in combination with a transform block (TB)) has an associated coding cost (or "rate") and an associated difference (or "distortion"). The rate is typically measured in bits. The distortion of the CB is typically estimated as a difference in sample values, such as the sum of absolute differences (SAD) or the sum of squared differences (SSD), etc. The mode selector 386 can determine the estimates obtained from each candidate PB using the residual 324 to determine the prediction mode (indicated by arrow 388). The estimation of the coding cost associated with each candidate prediction mode and the corresponding residual encoding can be performed at a significantly lower cost than entropy encoding of the residual. Therefore, multiple candidate modes can be evaluated to determine the best mode in terms of rate distortion.

[0077] Determining the best mode in terms of rate-distortion is typically implemented using a variation of Lagrangian optimization. Selecting a prediction mode 388 typically involves determining the coding cost of the residual data resulting from applying a particular prediction mode. The coding cost can be approximated by using the "sum of absolute transform differences" (SATD), thereby obtaining an estimated transform residual cost using a relatively simple transform such as a Hadamard transform. In some implementations using relatively simple transforms, the cost obtained from the simplified estimation method is monotonically related to the actual cost that would otherwise be determined from a complete evaluation. In implementations with monotonically related estimated costs, the simplified estimation method can be used to make the same decision (i.e., prediction mode) with reduced complexity of the video encoder 114. In order to allow for possible non-monotonicity in the relationship between the estimated cost and the actual cost, the simplified estimation method can be used to generate a list of best candidates. For example, non-monotonicity may result from further mode decisions that can be used to encode the residual data. The list of best candidates can have any number. The best candidates can be used to conduct a more complete search to establish the best mode selection for encoding the residual data of each candidate, thereby allowing the final selection of the prediction mode 388 and other mode decisions.

[0078] Prediction modes are broadly divided into two categories. The first category is "intra-frame prediction" (also called "intra prediction"). In intra prediction, a prediction of a block is generated, and the generation method can use other samples obtained from the current frame. Types of intra prediction include intra plane, intra DC, intra angle, and matrix weighted intra prediction (MIP). For intra-predicted PBs, different intra prediction modes can be used for luma and chroma, so intra prediction is mainly described in terms of operations on PBs. In addition, chroma CBs can be predicted from co-located luma samples by cross-component linear model prediction.

[0079] The second category of prediction modes is "inter-frame prediction" (also called "inter prediction"). In inter prediction, samples from one or two frames in the bitstream that precede the current frame in the order of the coded frames are used to generate a prediction for the block. In addition, for inter prediction, a single coding tree is typically used for both the luminance channel and the chrominance channel. The order of the coded frames in the bitstream may be different from the order of the frames when captured or displayed. When one frame is used for prediction, the block is called "single prediction" and has one associated motion vector. When two frames are used for prediction, the block is called "double prediction" and has two associated motion vectors. For P slices, each CU can be intra predicted or single predicted. For B slices, each CU can be intra predicted, single predicted, or double predicted. Frames are typically encoded using a "group of pictures" structure, thereby achieving a temporal hierarchy of frames. The temporal hierarchy of frames allows frames to reference previous and subsequent pictures in the order in which the frames are displayed. The images are encoded in the order required to ensure that the dependencies of the individual frames are met.

[0080] A subcategory of inter prediction is called "skip mode". Inter prediction and skip mode are described as two different modes. However, both inter prediction mode and skip mode involve motion vectors that refer to a block of samples from a previous frame. Inter prediction involves an encoded motion vector increment that specifies a motion vector relative to a motion vector predictor. The motion vector predictor is obtained from a list of one or more candidate motion vectors selected with a "merge index". The encoded motion vector increment provides a spatial offset to the selected motion vector prediction. Inter prediction also uses the encoded residual in the bitstream 133. Skip mode only uses an index (also called a "merge index") to select one from several motion vector candidates. The selected candidate is used without any further signal notification. In addition, skip mode does not support the encoding of any residual coefficients. The absence of an encoded residual coefficient when skip mode is used means that no transformation of skip mode is required. Therefore, skip mode does not usually cause pipeline processing problems. Pipeline processing problems may be the case for intra-predicted CUs and inter-predicted CUs. Due to limited signaling of skip mode, skip mode is useful for achieving very high compression performance when relatively high quality reference frames are available.A bi-predicted CU in a higher temporal layer of a random access group of pictures structure typically has high quality reference pictures and motion vector candidates that accurately reflect the underlying motion.

[0081] Samples are selected based on motion vectors and reference picture indices. Motion vectors and reference picture indices apply to all color channels, so inter prediction is described primarily in terms of operations on PUs rather than PBs. Within each category (i.e., intra and inter prediction), different techniques may be applied to generate PUs. For example, intra prediction may use a combination of values ​​and directions of neighboring rows and columns from previously reconstructed samples to generate PUs according to a prescribed filtering and generation process. Alternatively, a small number of parameters may be used to describe the PU. Inter prediction methods may vary in the number of motion parameters and their precision. Motion parameters typically include a reference frame index, which indicates which reference frame(s) from a reference frame list will be used plus a spatial translation of each reference frame, but may include more frames, special frames, or complex affine parameters (such as scaling and rotation, etc.). In addition, a predetermined motion refinement process may be applied to generate a dense motion estimate based on a block of reference samples.

[0082] A Lagrangian or similar optimization process may be employed for both the selection of the best partition of a CTU to a CB (using the block partitioner 310) and the selection of the best prediction mode from multiple possibilities. By applying the Lagrangian optimization process of the candidate modes in the mode selector module 386, the prediction mode with the lowest cost measure is selected as the best mode. The lowest cost mode is the selected prediction mode 388 and is also encoded in the bitstream 115 by the entropy encoder 338. The selection of the prediction mode 388 by the operation of the mode selector module 386 is extended to the operation of the block partitioner 310. For example, the candidates for the selection of the prediction mode 388 may include a mode applicable to a given block and additional modes applicable to multiple smaller blocks that are overall co-located with the given block. In the case of including modes applicable to a given block and smaller co-located blocks, the process of selecting the candidates is implicitly also a process for determining the optimal hierarchical decomposition of the CTU to the CB.

[0083] In the second stage of the operation of the video encoder 114, referred to as the "encoding" stage, iterations of the selected luma coding tree and the selected chroma coding tree, and thus the respective selected CBs, are performed in the video encoder 114. In the iterations, the CBs are encoded into the bitstream 115 as further described herein.

[0084] The entropy encoder 338 supports both variable length encoding of syntactic elements and arithmetic encoding of syntactic elements. Arithmetic coding is supported using context-adaptive binary arithmetic coding (CABAC) processing. The syntactic elements of arithmetic coding consist of a sequence of one or more "bins (binary files)". Like bits, the value of a bin is "0" or "1". Bins are not encoded in the bit stream 115 as discrete bits. Bins have associated predictions (or "likely" or "maximum probability") values ​​and associated probabilities (referred to as "context"). When the actual bin to be encoded matches the predicted value, the "maximum probability symbol" (MPS) is encoded. Encoding the maximum probability symbol is relatively cheap in terms of consuming bits. When the actual bin to be encoded does not match the possible value, the "minimum probability symbol" (LPS) is encoded. Encoding the minimum probability symbol has a relatively high cost in terms of consuming bits. The bin encoding technique enables bins with skewed probabilities of "0" vs "1" to be efficiently encoded. For syntax elements with two possible values ​​(ie, "flag"), a single bin is sufficient. For syntax elements with many possible values, a sequence of bins is required.

[0085] The presence of a later bin in a sequence can be determined based on the value of an earlier bin in the sequence. In addition, each bin can be associated with more than one context. The selection of a particular context can depend on the bin values ​​of an earlier bin in a syntax element, an adjacent syntax element (i.e., an adjacent syntax element from an adjacent block), etc. Each time a context coding bin is encoded, the context selected for that bin (if present) is updated to reflect the new bin value. Therefore, the binary arithmetic coding scheme is said to be adaptive.

[0086] The video encoder 114 also supports bins that lack context ("bypass bins"). Bypass bins are encoded assuming an equal probability distribution between "0" and "1". Thus, each bin occupies one bit in the bitstream 115. The absence of context saves memory and reduces complexity, thus using bypass bins where the distribution of values ​​for a particular bin is not skewed.

[0087] The entropy encoder 338 encodes the prediction mode 388 using a combination of context-encoded and bypass-encoded bins. For example, when the prediction mode 388 is an intra-frame prediction mode, a list of "maximum probability modes" is generated in the video encoder 114. The list of maximum probability modes is typically a fixed length such as three or six modes, and may include modes encountered in early blocks. The context-encoded bin encodes a flag indicating whether the prediction mode is one of the maximum probability modes. If the intra-frame prediction mode 388 is one of the maximum probability modes, further signaling is encoded using the bypass-encoded bin. The encoded further signaling, for example, uses a truncated unary bin string to indicate which maximum probability mode corresponds to the intra-frame prediction mode 388. Otherwise, the intra-frame prediction mode 388 is encoded as a "remaining mode". Encoding as a remaining mode uses an alternative syntax such as a fixed-length code (also encoded using a bypass-encoded bin) to represent an intra-frame prediction mode other than the intra-frame prediction mode present in the maximum probability mode list.

[0088] The multiplexer module 384 outputs the PB 320 according to the determined best prediction mode 388 selected from the prediction modes of the various candidate CBs tested. The candidate prediction modes need not include every conceivable prediction mode supported by the video encoder 114.

[0089] In the case where the PB 320 is determined and selected and subtracted from the original sample block at the subtractor 322, a residual with the lowest coding cost, represented as 324, is obtained and lossily compressed. The lossy compression process includes the steps of transform, quantization and entropy coding. A forward primary transform module 326 applies a forward transform to the residual 324, thereby converting the residual 324 from the spatial domain to the frequency domain and producing primary transform coefficients represented by arrow 328. The primary transform coefficients 328 are passed to a forward secondary transform module 330 to produce transform coefficients represented by arrow 332 by performing a non-separable secondary transform (NSST) operation. The forward major transform is typically separable, e.g., horizontally for block widths of no more than 16 samples and vertically for block heights of no more than 16 samples, typically using a discrete cosine transform type II (DCT-2) (but discrete sine transform type VII (DST-7) and discrete cosine transform type VIII (DCT-8) may also be available) to transform a set of rows and then a set of columns of each block. The transforms of the sets of rows and columns are performed by first applying a one-dimensional transform to each row of the block to produce an intermediate result, and then applying a one-dimensional transform to each column of the intermediate result to produce a final result. The forward minor transform is typically a non-separable transform that is applied only to the residual of the intra-predicted CU, and can nevertheless be bypassed. The forward minor transform operates on 16 samples (arranged as the upper left 4×4 sub-block of the main transform coefficients 328) or 64 samples (arranged as the upper left 8×8 coefficients, arranged as four 4×4 sub-blocks of the main transform coefficients 328). In addition, the matrix coefficients of the forward secondary transform are selected from multiple sets according to the intra prediction mode of the CU, so that two coefficient sets are available for use. Using one of the matrix coefficient sets or bypassing the forward secondary transform is signaled with the "nsst_index" syntax element, which is encoded using truncated unary binarization to express the value zero (no secondary transform is applied), one (the selected first matrix coefficient set) or two (the selected second matrix coefficient set).

[0090] The video encoder 114 may also choose to skip both the primary transform and the secondary transform, referred to as a "transform skip" mode. Skipping the transform is suitable for residual data that lacks sufficient correlation to reduce encoding cost via representation as a transform basis function. Certain types of content, such as relatively simple computer-generated graphics, may exhibit similar behavior. When the transform skip mode is used, the transform coefficients 332 are the same as the residual coefficients 324.

[0091] The transform coefficients 332 are passed to a quantizer module 334. At module 334, quantization according to a "quantization parameter" is performed to produce quantized coefficients represented by arrow 336. The quantization parameter is constant for a given TB, and therefore results in uniform scaling of the resulting residual coefficients for the TB. Non-uniform scaling can also be achieved by applying a "quantization matrix", whereby the scaling factors applied to each residual coefficient are derived from a combination of the quantization parameter and a corresponding entry in a scaling matrix whose size is generally equal to the size of the TB. The scaling matrix may have a size less than the size of the TB, and when applied to the TB, a nearest neighbor method is used to provide scaled values ​​for each residual coefficient from the scaling matrix whose size is less than the size of the TB. The quantized coefficients 336 are supplied to an entropy encoder 338 for encoding in the bitstream 115. Typically, the quantized coefficients of each TB having at least one valid quantized coefficient are scanned according to a scan pattern to produce an ordered list of values. The scan pattern typically scans the TB as a sequence of 4×4 "sub-blocks", thereby providing a conventional scanning operation at a granularity of 4×4 groups of residual coefficients, where the arrangement of the sub-blocks depends on the size of the TB. Additionally, the prediction mode 388 and corresponding block partitions are also encoded in the bitstream 115 .

[0092] As described above, the video encoder 114 needs to access a frame representation that corresponds to the frame representation seen by the video decoder 134. Therefore, the quantized coefficients 336 are also inverse quantized by the dequantizer module 340 to produce reconstructed transform coefficients represented by arrows 342. The reconstructed transform coefficients 342 are passed through the inverse secondary transform module 344 to produce reconstructed primary transform coefficients represented by arrows 346. The reconstructed primary transform coefficients 346 are passed to the inverse primary transform module 348 to produce reconstructed residual samples of the TU represented by arrows 350. The type of inverse transform performed by the inverse secondary transform module 344 corresponds to the type of forward transform performed by the forward secondary transform module 330. The type of inverse transform performed by the inverse primary transform module 348 corresponds to the type of primary transform performed by the primary transform module 326. The reconstructed residual samples 350 are added by the summation module 352 to the PU 320 to produce reconstructed samples of the CU (indicated by arrows 354).

[0093] The reconstructed samples 354 are passed to the reference sample cache 356 and the in-loop filter module 368. The reference sample cache 356, which is typically implemented using static RAM on an ASIC (thus avoiding expensive off-chip memory accesses), provides the minimum sample storage required to satisfy the dependencies for generating intra PBs for subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs for use by the next row of CTUs and a column buffer with a range set by the height of the CTU. The reference sample cache 356 supplies reference samples (represented by arrow 358) to the reference sample filter 360. The sample filter 360 applies a smoothing operation to produce filtered reference samples (indicated by arrow 362). The filtered reference samples 362 are used by the intra prediction module 364 to produce an intra prediction block of samples represented by arrow 366. For each candidate intra prediction mode, the intra prediction module 364 produces a sample block, namely 366.

[0094] The in-loop filter module 368 applies several filtering stages to the reconstructed samples 354. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned with CU boundaries to reduce artifacts caused by discontinuities. Another filtering stage present in the in-loop filter module 368 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. Another available filtering stage in the in-loop filter module 368 is a "sample adaptive offset" (SAO) filter. The SAO filter works by first classifying the reconstructed samples into one or more categories and applying an offset at the sample level according to the assigned category.

[0095] Filtered samples, represented by arrow 370, are output from the in-loop filter module 368. The filtered samples 370 are stored in a frame buffer 372. The frame buffer 372 typically has the capacity to store several (e.g., up to 16) pictures and is therefore stored in the memory 206. Due to the large memory consumption required, the frame buffer 372 is typically not stored using on-chip memory. As such, access to the frame buffer 372 is expensive in terms of memory bandwidth. The frame buffer 372 provides a reference frame (represented by arrow 374) to the motion estimation module 376 and the motion compensation module 380.

[0096] The motion estimation module 376 estimates a plurality of "motion vectors" (denoted as 378), each of which is a Cartesian spatial offset relative to the position of the current CB, thereby referencing a block in one of the reference frames in the frame buffer 372. A filtered block of reference samples (denoted as 382) is generated for each motion vector. The filtered reference samples 382 form a further candidate mode for potential selection by the mode selector 386. In addition, for a given CU, the PB 320 may be formed using one reference block ("uni-prediction"), or may be formed using two reference blocks ("bi-prediction"). For the selected motion vector, the motion compensation module 380 generates the PB 320 according to a filtering process that supports sub-pixel precision in the motion vector. In this way, the motion estimation module 376 (which operates on many candidate motion vectors) can perform a simplified filtering process compared to the motion compensation module 380 (which operates only on the selected candidate) to achieve reduced computational complexity. When the video encoder 114 selects inter-prediction for the CU, the motion vector 378 is encoded into the bitstream 115.

[0097] Although the reference to Versatile Video Coding (VVC) describes Figure 3 114, but other video coding standards or implementations may also use the processing stages of modules 310-386. Frame data 113 (and bit stream 115) may also be obtained from memory 206, hard drive 210, CD-ROM, Blue-ray disk TM ) or other computer-readable storage medium (or written to memory 206, hard drive 210, CD-ROM, Blu-ray disc or other computer-readable storage medium). In addition, frame data 113 (and bit stream 115) can be received from (or sent to) an external source (such as a server or radio frequency receiver connected to communication network 220).

[0098] In Figure 4 The video decoder 134 is shown in FIG. Figure 4 The video decoder 134 of FIG. 1 is an example of a Versatile Video Coding (VVC) video decoding pipeline, but other video codecs may also be used to perform the processing stages described herein. Figure 4 As shown, the bitstream 133 is input to the video decoder 134. The bitstream 133 may be received from the memory 206, the hard drive 210, the CD-ROM, the Blu-ray disc, or the like. TM Or other non-transitory computer-readable storage medium. Alternatively, the bitstream 133 may be received from an external source, such as a server or a radio frequency receiver connected to the communication network 220. The bitstream 133 contains encoding syntax elements representing the captured frame data to be decoded.

[0099] The bitstream 133 is input to the entropy decoder module 420. The entropy decoder module 420 extracts syntax elements from the bitstream 133 by decoding a sequence of "bins" and passes the values ​​of the syntax elements to other modules in the video decoder 134. An example of a syntax element extracted from the bitstream 133 is a quantization coefficient 424. The entropy decoder module 420 uses an arithmetic decoding engine to decode each syntax element into a sequence of one or more bins. Each bin can use one or more "contexts", where the context describes the probability level of the "one" and "zero" values ​​to be used to encode the bin. In the case where multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts to decode the bin. The process of decoding the bin forms a sequential feedback loop. The number of operations in the feedback loop is preferably minimized so that the entropy decoder 420 can achieve a high throughput in bins / second. Context modeling depends on other properties of the bitstream known to the video decoder 134 when selecting the context, that is, the properties before the current bin. For example, the context may be selected based on the quadtree depth of the current CU in the coding tree.The dependencies are preferably based on properties that are known before decoding the bins, or determined without requiring long sequential processing.

[0100] The quantized coefficients 424 are input to a dequantizer module 428. The dequantizer module 428 inverse quantizes (or scales) the quantized coefficients 424 to generate reconstructed intermediate transform coefficients represented by arrow 432 according to the quantization parameters. If the use of a non-uniform inverse quantization matrix is ​​indicated in the bitstream 133, the video decoder 134 reads the quantization matrix from the bitstream 133 as a sequence of scaling factors and arranges the scaling factors into a matrix. Inverse scaling uses the quantization matrix in combination with the quantization parameters to create reconstructed intermediate transform coefficients 432. The reconstructed intermediate transform coefficients 432 are passed to an inverse secondary transform module 436, where a secondary transform can be applied according to the decoded "nsst_index" syntax element. Under execution of the processor 205, the entropy decoder 420 decodes the "nsst_index" from the bitstream 133. The inverse secondary transform module 436 produces a reconstructed transform coefficient 440.

[0101] The reconstructed transform coefficients 440 are passed to an inverse primary transform module 444. Module 444 transforms the coefficients from the frequency domain back to the spatial domain. The result of the operation of module 444 is a block of residual samples represented by arrow 448. The block of residual samples 448 is equal in size to the corresponding CU. The type of inverse primary transform can be discrete cosine transform type II (DCT-2), discrete sine transform type VII (DST-7), discrete cosine transform type VIII (DCT-8), or a "transform skip" mode. The use of transform skip mode is signaled by a transform skip flag, which can be decoded from the bitstream 133 or otherwise inferred. When transform skip mode is used, the residual samples 448 are the same as the reconstructed transform coefficients 440.

[0102] The residual samples 448 are supplied to a summing module 450. At the summing module 450, the residual samples 448 are added to the decoded PB (denoted as 452) to produce a block of reconstructed samples represented by arrow 456. The reconstructed samples 456 are supplied to a reconstructed sample cache 460 and an in-loop filtering module 488. The in-loop filtering module 488 produces a reconstructed block of frame samples denoted as 492. The frame samples 492 are written to a frame buffer 496.

[0103] The reconstructed sample cache 460 operates in a manner similar to the reconstructed sample cache 356 of the video encoder 114. The reconstructed sample cache 460 provides storage for the reconstructed samples needed for intra prediction of subsequent CBs without the memory 206 (e.g., by using data 232, which is typically on-chip memory, instead). Reference samples, represented by arrows 464, are obtained from the reconstructed sample cache 460 and are supplied to a reference sample filter 468 to produce filtered reference samples, represented by arrows 472. The filtered reference samples 472 are supplied to an intra prediction module 476. The module 476 produces a block of intra prediction samples, represented by arrows 480, based on the intra prediction mode parameters 458 represented in the bitstream 133 and decoded by the entropy decoder 420.

[0104] When the prediction mode of the CB is indicated as intra prediction in the bitstream 133, the intra prediction samples 480 form the decoded PB 452 via the multiplexer module 484. Intra prediction produces a prediction block (PB) of samples, i.e., a block in one color component, which is derived using "neighboring samples" in the same color component. Neighboring samples are samples that are adjacent to the current block and have been reconstructed because they are previous in the block decoding order. In the case where the luma block and the chroma block are juxtaposed, the luma block and the chroma block can use different intra prediction modes. However, the two chroma channels each share the same intra prediction mode.

[0105] Intra prediction for luminance blocks consists of four types. "DC intra prediction" involves filling PB with a single value representing the average value of adjacent samples. "Plane intra prediction" involves filling PB with samples according to the plane using DC offset and vertical and horizontal gradients derived from adjacent samples. "Angle intra prediction" involves filling PB with adjacent samples filtered and propagated across PB in a specific direction (or "angle"). In VVC, PB can be selected from up to 65 angles, where rectangular blocks can use different angles that are not available for square blocks. "Matrix intra prediction" involves filling PB by multiplying a reduced set of adjacent samples by one of the multiple available matrices available to the video decoder 134. A reduced set of adjacent samples is generated by filtering and subsampling adjacent samples. Then, a reduced set of predicted samples is generated by multiplying the reduced set of samples by the matrix and adding an offset vector. The matrix and the associated offset vector are selected from multiple possible matrices according to the size of the PB, where the specific selection of the matrix and the offset vector is indicated by the "MIP mode" syntax element. For example, for PBs of size greater than 8 × 8, there are 11 MIP modes, while for PBs of size 8 × 8, there are 19 MIP modes. Finally, the PBs generated by matrix intra prediction are filled by interpolation from the reduced set of prediction samples.

[0106] The fifth type of intra prediction can be used for chroma PBs, whereby the PBs are generated from collocated luma reconstruction samples according to a "cross component linear model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived model is then used to generate a block of samples of the chroma PB from collocated luma samples.

[0107] When the prediction mode of the CB is indicated as inter-frame prediction in the bitstream 133, the motion compensation module 434 selects and filters a block of samples 498 from the frame buffer 496 using the motion vector and the reference frame index to produce a block of inter-frame prediction samples represented as 438. The block of samples 498 is obtained from a previously decoded frame stored in the frame buffer 496. In order to perform bi-prediction, two blocks of samples are generated and blended together to produce samples of the decoded PB 452. The frame buffer 496 is populated with filter block data 492 from the in-loop filtering module 488. As with the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 applies any DBF, ALF and SAO filtering operations. Generally, the motion vector is applied to both the luma and chroma channels, but the filtering process for sub-sample interpolated luma and chroma channels is different. The frame buffer 496 outputs the decoded video samples 135.

[0108] Figure 5is a schematic block diagram showing a set 500 of possible partitions or splits of a region into one or more sub-regions in a tree structure for general video coding. Figure 3 As described, the partitioning shown in set 500 may be utilized by block partitioner 310 of encoder 114 to partition each CTU into one or more CUs or CBs according to a coding tree as determined by Lagrangian optimization.

[0109] Although set 500 only shows the partitioning of a square region into other possible non-square sub-regions, it should be understood that diagram 500 is illustrating potential partitioning and does not require the containing region to be square. If the containing region is non-square, the size of the blocks resulting from the partitioning is scaled according to the aspect ratio of the containing block. Once a region is not split further, i.e., at a leaf node of the coding tree, a CU occupies the region. A specific sub-partition of a CTU into one or more CUs using block partitioner 310 is referred to as a "coding tree" of the CTU.

[0110] When the resulting sub-region reaches the minimum CU size, the process of sub-dividing the region into sub-regions must terminate. In addition to constraining the CU to prohibit block regions from being smaller than a predetermined minimum size (e.g., 16 samples), the CU is constrained to have a minimum width or height of 4. Other minimum values ​​are also possible in terms of width and height or both. The sub-division process can also be terminated before the deepest decomposition level, resulting in a CU larger than the minimum CU size. Splitting may not occur, resulting in a single CU occupying the entire CTU. A single CU occupying the entire CTU is the maximum available coding unit size. Due to the use of sub-sampling chroma formats such as 4:2:0, the arrangement of the video encoder 114 and the video decoder 134 can terminate the splitting of regions in the chroma channel earlier than in the luminance channel.

[0111] In the absence of further sub-division, there is a CU at the leaf node of the coding tree. For example, leaf node 510 contains a CU. At the non-leaf node of the coding tree, there is a split to two or more other nodes, where each node can be a leaf node forming a CU, or a leaf node containing further splits to smaller areas. At each leaf node of the coding tree, there is a coding block for each color channel. For both luminance and chrominance, the split that terminates at the same depth obtains three juxtaposed CBs. The split that terminates at a depth deeper than chrominance obtains multiple luminance CBs juxtaposed with the CBs of the chrominance channels.

[0112] like Figure 5As shown, quadtree split 512 divides the containing area into four equal-sized areas. Compared to HEVC, general video coding (VVC) achieves additional flexibility by adding horizontal binary split 514 and vertical binary split 516. Splits 514 and 516 each divide the containing area into two equal-sized areas. The splits are along horizontal boundaries (514) or vertical boundaries (516) within the containing block.

[0113] Further flexibility is achieved in general video coding by adding ternary horizontal split 518 and ternary vertical split 520. Ternary split 518 and 520 divide the block into three regions that form boundaries in the horizontal direction (518) or vertical direction (520) along 1 / 4 and 3 / 4 of the width or height of the containing area. The combination of quadtree, binary tree and ternary tree is called "QTBTTT". The root of the tree includes zero or more quadtree splits (the "QT" part of the tree). Once the QT part is terminated, zero or more binary or ternary splits (the "multi-tree" or "MT" part of the tree) may occur, eventually ending in a CB or CU at the leaf node of the tree. In the case where the tree describes all color channels, the leaf node is a CU. In the case where the tree describes a luminance channel or a chrominance channel, the leaf node is a CB.

[0114] Compared to HEVC, which only supports quadtrees and therefore only supports square blocks, QTBTTT obtains more possible CU sizes, especially considering the possible recursive application of binary and / or ternary tree splitting. The possibility of abnormal (non-square) block sizes can be reduced by constraining the splitting options to eliminate the block width or height that will be less than four samples or the split that will not be a multiple of four samples. Typically, the constraints will apply when considering luma samples. However, in the above arrangement, the constraints can also be applied separately to blocks of chroma channels. The splitting options that apply constraints to chroma channels can obtain different minimum block sizes for luma vs chroma (for example, when the frame data adopts a 4:2:0 chroma format or a 4:2:2 chroma format). Each split produces a sub-region with a side size that is unchanged, halved, or quartered relative to the containing area. Then, since the CTU size is a power of 2, the side sizes of all CUs are also powers of 2.

[0115] Figure 6 6 is a schematic flow diagram showing a data flow 600 of a QTBTTT (or "coding tree") structure used in general video coding. The QTBTTT structure is used for each CTU to define the partitioning of the CTU into one or more CUs. The QTBTTT structure for each CTU is determined by the block partitioner 310 in the video encoder 114 and encoded into the bitstream 115 or decoded from the bitstream 133 by the entropy decoder 420 in the video decoder 134. Figure 5As shown partitioned, data flow 600 further features permitted combinations that may be used by block partitioner 310 to partition a CTU into one or more CUs.

[0116] Starting from the top level of the hierarchy, i.e., at the CTU, zero or more quadtree partitions are first performed. Specifically, a quadtree (QT) split decision 610 is made by the block partitioner 310. The decision at 610 returns a "1" symbol, which indicates a decision to split the current node into four child nodes according to the quadtree split 512. The result is that four new nodes are generated, such as at 620, and for each new node, recursively return to the QT split decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates that no further splitting is to be performed (returns a "0" symbol), the quadtree partitioning stops, and multi-tree (MT) splitting is then considered.

[0117] First, an MT split decision 612 is made by the block partitioner 310. At 612, a decision to perform an MT split is indicated. A "0" symbol is returned at the decision 612, indicating that no further splitting of the node into child nodes will be performed. If no further splitting of the node will be performed, the node is a leaf node of the coding tree and corresponds to a CU. The leaf node is output at 622. Alternatively, if the MT split 612 indicates a decision to perform an MT split (a "1" symbol is returned), the block partitioner 310 proceeds to a direction decision 614.

[0118] Direction decision 614 indicates the direction of the MT split as horizontal ("H" or "0") or vertical ("V" or "1"). If decision 614 returns "0" indicating a horizontal direction, block partitioner 310 proceeds to decision 616. If decision 614 returns "1" indicating a vertical direction, block partitioner 310 proceeds to decision 618.

[0119] In each of decisions 616 and 618, the number of partitions into which the MT is split is indicated as two (binary split or "BT" nodes) or three (ternary split or "TT") in the case of a BT / TT split. That is, the BT / TT split decision 616 is made by the block partitioner 310 when the direction indicated from 614 is horizontal, and the BT / TT split decision 618 is made by the block partitioner 310 when the direction indicated from 614 is vertical.

[0120] The BT / TT split decision 616 indicates whether the horizontal split is a binary split 514 indicated by returning “0” or a ternary split 518 indicated by returning “1”. When the BT / TT split decision 616 indicates a binary split, at a generate HBT CTU node step 625, the block partitioner 310 generates two nodes according to the binary horizontal split 514. When the BT / TT split 616 indicates a ternary split, at a generate HTT CTU node step 626, the block partitioner 310 generates three nodes according to the ternary horizontal split 518.

[0121] The BT / TT split decision 618 indicates whether the vertical split is a binary split 516 indicated by returning "0" or a ternary split 520 indicated by returning "1". When the BT / TT split 618 indicates a binary split, at step 627 of generating VBT CTU nodes, the block partitioner 310 generates two nodes according to the vertical binary split 516. When the BT / TT split 618 indicates a ternary split, at step 628 of generating VTT CTU nodes, the block partitioner 310 generates three nodes according to the vertical ternary split 520. For each node obtained from steps 625-628, the recursion of the data flow 600 back to the MT split decision 612 is applied in a left-to-right or top-to-bottom order according to the direction 614. As a result, binary and ternary tree splits can be applied to generate CUs of various sizes.

[0122] Fig. 7A and 7B An example partition 700 of a CTU 710 into multiple CUs or CBs is provided. Fig. 7A An example CU 712 is shown in FIG. Fig. 7A 7 shows the spatial arrangement of CUs in a CTU 710. Example partitioning 700 in Figure 7B Also shown in FIG. 7 is a coding tree 720 .

[0123] exist Fig. 7A At each non-leaf node (e.g., nodes 714, 716, and 718) in the CTU 710 of the coding tree 720, the contained nodes (which may be further partitioned or may be CUs) are scanned or traversed in "Z order" to create a list of nodes represented as columns in the coding tree 720. For quadtree splits, the Z order scan results in an order from top left to right followed by a bottom left to right order. For horizontal and vertical splits, the Z order scan (traversal) is simplified to a scan from top to bottom and a scan from left to right, respectively. Figure 7B The coding tree 720 lists all nodes and CUs according to the scanning order applied. Each split generates a list of two, three or four new nodes at the next level of the tree until a leaf node (CU) is reached.

[0124] In reference Figure 3In the case where the image is decomposed into CTUs and further decomposed into CUs by the block partitioner 310 and each residual block (324) is generated using the CU, the residual block is forward transformed by the video encoder 114. An equivalent inverse transform process is performed in the video decoder 134 to obtain a TB from the bitstream 133.

[0125] In the video encoder 114, the quantized coefficients 336 may be rearranged into a one-dimensional list by performing a two-stage reverse diagonal scan. Similarly, in the video decoder 134, the quantized coefficients 424 may be rearranged from the one-dimensional list into a two-dimensional set of sub-blocks by the same two-stage reverse diagonal scan.

[0126] Fig. 8A A two-stage reverse diagonal scan 810 of an example 8×8 TB 800 is shown. The scan 810 is shown as proceeding from the lower right residual coefficient position of the TB 800 back to the upper left (DC) residual coefficient position of the TB 800. The path of the scan 810 proceeds in 4×4 regions (called subblocks) and proceeds from one subblock to the next. For TBs with a width or height of 2, subblock sizes of 2×2, 2×8, or 8×2 are available. Depending on the "encoded subblock flag", scanning within a particular subblock is performed or the subblock is skipped. When scanning of a subblock is skipped, it is inferred that all residual coefficients within the subblock have zero values. Although the scan 810 is shown as starting from the lower right residual coefficient position of the TB 800, for a given set of residual coefficients, the scan starts from the "last significant coefficient" position, and the coefficient is "last" when the order of coefficients is considered to proceed from the DC coefficient rather than the scan order.

[0127] Figure 8B An alternative two-stage forward diagonal scan 860 of an example 8×8 TB 850 is shown for use when TSRC processing is selected. When TSRC processing is used in the video encoder 114, the quantized coefficients 336 are rearranged into a one-dimensional list by the scan 860. Similarly, if TSRC processing is used for the current TB in the video decoder 134, the quantized coefficients 424 are rearranged from the one-dimensional list into a two-dimensional set of sub-blocks by the scan 860. The scan 860 is shown as proceeding from the upper left (DC) residual coefficient position of the TB 850 to the lower right residual coefficient position of the TB 850. Unlike the scan 810, the scan 860 does not terminate at the "last significant coefficient".

[0128] Fig. 8A and 8B The scanning patterns commonly used in VVC are shown. The examples described herein use scanning pattern 810 to encode residual coefficients that have been transformed by module 326, and scanning pattern 860 is used to skip transformed transform blocks. However, in some implementations, other scanning patterns may be used.

[0129] As described above, when the transform skip mode is used, the transform coefficient 332 is the same as the residual coefficient 324. Therefore, regardless of whether the transform skip mode is selected, the transform coefficient 332 may also sometimes be referred to as a residual coefficient. When lossless encoding is desired, the video encoder 114 selects transform skip for the current TB and signals the transform skip flag to the bitstream 133 with the value "true". The residual coefficient 332 associated with the current TB is encoded to the bitstream 133. Two residual encoding processes are available: "regular residual coding" (RRC) processing and "transform skip residual coding" (TSRC) processing. In normal operation of the video encoder 114, if transform skip is selected (the transform skip flag has the value "true"), the TSRC process is selected, otherwise (the transform skip flag has the value "false"), the RRC process is selected. However, in the case of lossless encoding, it is generally not expected that the encoding of the residual coefficient 332 is handled exclusively by TSRC.

[0130] In one arrangement of the video encoder 114, a TSRC disable flag is signaled in the bitstream 133. The TSRC disable flag can be signaled at a relatively high level (such as once per sequence or once per picture) so that the relative cost of signaling the TSRC disable flag is low. High-level syntax elements are usually grouped in parameter sets, such as "sequence parameter sets" (SPS) for sequence-level flags or "picture parameter sets" (PPS) for parameter-level flags. When the video data 113 belongs to a category that is unlikely to be suitable for (in terms of encoding loss and reproduction of features) to be encoded with TSRC processing, the TSRC disable flag can be set to "true". An example of video data that is not suitable for TSRC encoding is natural scene content. When the video data 113 belongs to a category that may be well encoded using TSRC processing, the TSRC disable flag can be set to "false". Video data suitable for encoding with TSRC processing includes artificial picture content.

[0131] If the video encoder 114 selects transform skip for the current TB and the TSRC disable flag is set to “true”, the residual coefficient 332 is encoded using RRC processing to the bitstream 133. Similarly, if the video decoder 134 determines that transform skip is for the current TB and the TSRC disable flag is set to “true”, the residual coefficient 432 is decoded from the bitstream 133 using RRC processing.

[0132] Fig. 9A method 900 for encoding a transform block of residual coefficients 332 using RRC processing is shown. The method 900 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. In addition, the method 900 may be performed by the video encoder 114 under execution of the processor 205. As such, the method 900 may be implemented as a module of the software 233 stored on a computer-readable storage medium and / or in the memory 206.

[0133] In some arrangements, the method 900 is implemented by the video encoder 114 at the quantizer 334 upon receiving the residual coefficients 332, and then by the entropy encoder 338. The method 900 begins at a step 910 of quantizing the coefficients.

[0134] At step 910 of quantizing the coefficients, step 910 calls the following Fig.11 The method 1100 described herein can be embodied by a device such as a configured FPGA, ASIC, or ASSP. In addition, the method 1100 can be performed by the video encoder 114 under the execution of the processor 205. As such, the method 1100 can be implemented as a module of the software 233 stored on a computer-readable storage medium and / or in the memory 206. The method 1100 quantizes the residual coefficients 332 to generate quantized coefficients 336. The method 900 proceeds from step 910 to the encoding last position step 920 under the control of the processor 205.

[0135] At an encode last position step 920, the video encoder 114 finds the position of the last significant coefficient in the quantized coefficients 336 used for the transformed block of residual coefficients 332. The last significant coefficient is determined relative to the forward direction of the appropriate scan pattern (e.g., in the direction of the two-level forward diagonal scan 860). A quantized coefficient is significant if the coefficient has any value other than zero. The position of the last significant coefficient is written to the bitstream 133. From step 920, the method 900 proceeds to an initialization state step 930 under the control of the processor 205.

[0136] At an initialize state step 930, the quantizer state Qstate is set to a value of zero. Additionally, the subblock containing the last significant coefficient is selected at step 930. From step 930, the method 900 proceeds to a determine encoded subblock flag step 940 under the control of the processor 205.

[0137] The descriptions in this article refer to some flags as being "true" or "false". Being set to "true" means that the flag value indicates that a mode is selected or a requirement is met. Being set to "false" means that the flag value indicates that a mode is not selected or a requirement is not met.

[0138] At a determine encoded sub-block flag step 940, the video encoder 114 determines and sets the encoded sub-block flag. If the currently selected sub-block is the first sub-block selected in the initialization state step 930, the encoded sub-block flag is set to "true" but is not encoded into the bitstream 133. If the currently selected sub-block is identified as the last sub-block, the encoded sub-block flag is set to "true" but is not encoded into the bitstream 133 as described below with respect to the last sub-block test 970.

[0139] Otherwise, if there is at least one significant coefficient in the 4×4 quantized coefficients belonging to the selected sub-block, the video encoder 114 sets the coded sub-block flag to (i) “true”, or if there is no significant coefficient, to (ii) “false”, and encodes the coded sub-block flag to the bitstream 133. The method 900 proceeds from step 940 to a coded sub-block flag test step 950 under the control of the processor 205.

[0140] At the coded sub-block flag test step 950, the method 900 determines the value of the coded sub-block flag. If the coded sub-block flag is set to "true", the method 900 proceeds to the coded sub-block step 960. Otherwise, if the coded sub-block flag is set to "false", the method 900 proceeds to the last sub-block test step 970.

[0141] At the encode subblock step 960, the entropy encoder 338 encodes the quantized coefficients in the selected subblock into the bitstream 133. Step 960 calls the following Fig.13 Described method 1300. Method 900 proceeds from step 960 to last sub-block test 970 under the control of the processor.

[0142] At the last sub-block test 970, the method 900 operates to determine whether the selected sub-block is the last sub-block in the current transform block. If the currently selected sub-block is the upper left sub-block of the transform block, step 900 returns "yes" and the method 900 terminates. Otherwise, if the currently selected sub-block is not the upper left sub-block of the transform block, step 970 returns "no" and the method 900 proceeds to the select next sub-block step 980.

[0143] At a select next subblock step 980, the next subblock in the transform block is selected. The next subblock is selected in the reverse diagonal scan order 810. From step 980, the method 900 proceeds to a determine encoded subblock flag step 940 for the selected subblock.

[0144] Fig.10A method 1000 is shown for decoding a transformed block of residual coefficients 432 by RRC processing. The method 1000 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. In addition, the method 1000 may be performed by the video decoder 134 under execution of the processor 205. As such, the method 1000 may be implemented as a module of the software 233 stored on a computer-readable storage medium and / or in the memory 206.

[0145] In some arrangements, the method 1000 is implemented by the video decoder 134 at the entropy decoder 420 upon receiving the bitstream 133, and at the dequantizer module 428. The method 1000 begins at a decode last position step 1010.

[0146] At a decode last position step 1010, the last significant coefficient position of the transformed block of residual coefficients 432 is decoded from the bitstream 133. From step 1010, the method 1000 proceeds to an initialization state step 1020 under the control of the processor 205.

[0147] At an initialize state step 1020, the video decoder 134 initializes the quantizer state Qstate to a value of zero. Additionally, the sub-block containing the last significant coefficient position is selected at step 1020. From step 1020, the method 1000 proceeds to a determine encoded sub-block flag step 1030 under the control of the processor 205.

[0148] At a determine encoded sub-block flag step 1030, the video decoder 134 determines the encoded sub-block flag. If the currently selected sub-block is the first sub-block selected in the initialization state step 1020, the encoded sub-block flag is set to "true" (i.e., the encoded sub-block flag is inferred to be "true"). If the currently selected sub-block is identified as the last sub-block, the encoded sub-block flag is inferred to be "true" as described below in the last sub-block test 1060. Otherwise, the video decoder 134 decodes the encoded sub-block flag from the bitstream 133. From step 1030, the method 1000 proceeds to the encoded sub-block flag test 1040 under the control of the processor 205.

[0149] At coded sub-block flag test 1040, method 1000 tests the value of the coded sub-block flag determined at step 1030. If it is determined at step 1040 that the coded sub-block flag has a value of "true", method 1000 proceeds to decode sub-block step 1050. Otherwise, if it is determined at step 1040 that the coded sub-block flag has a value of "false", all quantized coefficients in the currently selected sub-block are assigned a value of zero and method 1000 proceeds to last sub-block test 1060.

[0150] At decode subblock step 1050, entropy decoder 420 decodes the quantized coefficients of the selected subblock from bitstream 133. Step 1050 calls the following Fig.14 The method 1400 is described. The method 1000 proceeds to the last sub-block test 1060 under the control of the processor 205.

[0151] At last subblock test 1060, if the currently selected subblock is the top left subblock of the transform block, then step 1060 returns "yes" and method 1000 proceeds to scaling factor step 1080. Otherwise, step 1060 returns "no" and method 1000 proceeds to select next subblock step 1070.

[0152] At a select next sub-block step 1070, the next sub-block in the reverse diagonal scan order 810 is selected. From step 1070, the method 1000 proceeds to a determine encoded sub-block flag step 1030 under the control of the processor 205.

[0153] At a scale coefficients step 1080, the dequantizer module 428 applies scaling to the quantized coefficients 424, thereby producing reconstructed residual coefficients 432. The sub-block is decoded by reconstructing the residual coefficients of the sub-block using the decoded sign bits. Step 1080 calls the following regarding Fig.12 Method 1200 is described. Method 1000 terminates when step 1080 is executed.

[0154] Fig.11 A method 1100 is shown for quantizing the residual coefficients 332 of a transform block to produce quantized coefficients 336. At step 910 of method 900, method 1100 is implemented for a TB. Method 1100 begins with a DQ test 1110.

[0155] At DQ test 1110, the video encoder 114 determines whether to quantize the residual coefficients 332 using correlated quantization. The video encoder 114 checks the value of the enable correlated quantization flag, which is signaled in the bitstream 133 as a high-level syntax. The enable correlated quantization flag determines whether correlated quantization is allowed within the range of the flag. For example, the sequence level correlated quantization flag determines whether correlated quantization is allowed when encoding the entire video sequence. The picture level correlated quantization flag determines whether correlated quantization is allowed when encoding the current picture and will take precedence over the value of the sequence level correlated quantization flag. If the enable correlated quantization flag is "false", step 1110 returns "no" and the method 1100 proceeds to the scalar quantization step 1120.

[0156] In one arrangement of the DQ test 1110, if the enable correlated quantization flag is "true", the video encoder 114 also checks the value of the transform skip flag of the current TB. If the enable correlated quantization flag is "true" and the transform skip flag is "true", step 1110 returns "no" and the method 1100 proceeds to the scalar quantization step 1120. Otherwise, if the enable correlated quantization flag is "true" and the transform skip flag is "false", step 1110 returns "yes" and the method 1100 proceeds to the correlated quantization step 1130.

[0157] In another arrangement of the DQ test 1110, if the Enable Correlated Quantization flag is "true", the video encoder 114 also checks the value of the TSRC Disable flag. If the Enable Correlated Quantization flag is "true" and the TSRC Disable flag is "true", step 1110 returns "no" and the method 1100 proceeds to the scalar quantization step 1120. Otherwise, if the Enable Correlated Quantization flag is "true" and the TSRC Disable flag is "false", step 1110 returns "yes" and the method 1100 proceeds to the Correlated Quantization step 1130.

[0158] In yet another arrangement of the DQ test 1110, if the enable dependent quantization flag is "true", the video encoder 114 also checks the value of the quantization parameter (QP) of the current TB. The QP indicates the degree of quantization to be applied to the residual coefficients 332. The initial QP depends on the bit depth BD of the video encoder 114. i The QP is determined by QP = QP i +6*(BD-8). For example, if QP i is 4 and the bit depth is 8, then the QP is determined to be four. iis -8 and the bit depth is 10, then the QP is determined to be 4. In general, a QP of 4 indicates that the residual coefficients will not be quantized, and therefore lossless operation is possible. However, higher values ​​of QP may still enable lossless operation. For example, if the video data 113 was originally captured at a bit depth of 8, but was supplied to the video encoder 114 at a higher bit depth, then lossless operation is possible at a higher QP. For example, if the video data 113 was captured at a bit depth of 8 but was supplied to the video encoder 114 at a bit depth of 10, then lossless operation is possible at a QP of 4, 10, or 16. The QP at which lossless operation is possible may be indicated by the minimum QP for the transform skip block, which is signaled in the high-level syntax parameter set. If the enable correlated quantization flag is "true" and the QP is 4 (or any value indicating lossless operation), step 1110 returns "no" and the method 1100 proceeds to a scalar quantization step 1120. Otherwise, if the enable correlated quantization flag is “TRUE” and the QP is not 4 (or a similar value indicating lossless operation), step 1110 returns “YES” and the method 1100 proceeds to a correlated quantization step 1130 .

[0159] At the scalar quantization step 1120, the residual coefficient 332 is represented as r[n]. Then, the quantized coefficient q[n] is generated by quantizing the residual coefficient r[n] according to the following equation (1):

[0160] q[n]=(k*r[n]+offset)>>qbits (1)

[0161] In equation (1), k is a scaling factor, qbits is a coarse quantization factor, and offset controls the placement of the quantization threshold. k, qbits, sffset are determined based on the value of the quantization parameter of the current TB. For example, if the QP is 4, then k=1, qbits=0, and offset=0. Then when the QP is 4, q[n]=r[n] and no loss occurs in the scalar quantization step. The method 1100 proceeds from step 1120 to the SBH test 1140 under the control of the processor 205.

[0162] At the relevant quantization step 1130, each residual coefficient r[n] can be quantized by selecting one of multiple scalar quantizers. For the same QP, the scalar quantizers have the same quantization partition size, but the quantization thresholds are offset relative to each other. The scalar quantizer used for a specific residual coefficient r[n] depends on the current quantizer state Qstate updated for each coefficient, and on the parity (least significant bit) of the resulting q[n]. Due to the dependence on the previous state, the best quantization result is not determined based on each coefficient. An effective way to determine the best quantization result is by constructing a "grid" of possible quantization states at each coefficient position. The best quantization result can be found by equivalently finding the best path through the grid. The most suitable grid path can be determined by applying the Viterbi algorithm. Method 1100 terminates when step 1130 is executed.

[0163] At SBH test 1140, the video encoder 114 determines whether to use sign bit hiding to modify the quantized coefficients q[n] before encoding the coefficients of the TB. The video encoder 114 checks the value of the enable sign bit hiding flag. The enable sign bit hiding flag can be signaled as a high-level syntax in the bitstream 133. For example, the enable sign bit hiding flag can be signaled in the picture header. If the enable related quantization flag is "true", the enable sign bit hiding flag is implicitly "false". If the enable sign bit hiding flag is "false", step 1140 returns "no" and the method 1100 terminates.

[0164] In one arrangement of the SBH test 1140, the determination depends on the value of the enable sign bit hiding flag and the value of the transform skip flag of the current TB. If the enable sign bit hiding flag is "true", the video encoder 114 also checks the value of the transform skip flag of the current TB. If the enable sign bit hiding flag is "true" and the transform skip flag is "true", step 1140 returns "no" and the method 1100 terminates. Otherwise, if the enable sign bit hiding flag is "true" and the transform skip flag is "false", step 1140 returns "yes" and the method 1100 proceeds to the adjust parity step 1150.

[0165] In another arrangement of the SBH test 1140, the determination depends on the value of the enable sign bit hiding flag and the value of the TSRC disable flag. If the enable sign bit hiding flag is "true", the video encoder 114 also checks the value of the TSRC disable flag. If the enable sign bit hiding flag is "true" and the TSRC disable flag is "true", step 1140 returns "no" and the method 1100 terminates. Otherwise, if the enable sign bit hiding flag is "true" and the TSRC disable flag is "false", step 1140 returns "yes" and the method 1100 proceeds to the adjust parity step 1150.

[0166] In yet another arrangement of the SBH test 1140, the determination depends on the value of the enable sign bit hiding flag and the value of the quantization parameter (QP) for the current TB. If the enable sign bit hiding flag is "true", the video encoder 114 also checks the value of the QP for the current TB. If the enable sign bit hiding flag is "true" and the QP is 4 (or any value indicating lossless operation), step 1140 returns "no" and the method 1100 terminates. Otherwise, if the enable sign bit hiding flag is "true" and the QP is not 4 (or a similar value indicating lossless operation), step 1140 returns "yes" and the method 1100 proceeds to the adjust parity step 1150.

[0167] At the adjust parity step 1150, the video encoder 114 checks the positions of the first and last significant coefficients of each sub-block in the current TB. If the difference between the first significant position and the last significant position of the sub-block is greater than a threshold value (usually 3), then sign bit hiding will be used for the sub-block. For each sub-block for which sign bit hiding will be used, the video encoder 114 checks the sign of the first significant coefficient in the sub-block and adjusts the parity of the coefficients in the sub-block accordingly. If the coefficient is even, the parity of the coefficient is zero, and if the coefficient is odd, it is 1. If the number of odd coefficients is odd, the sum of the parities of the multiple coefficients is zero, and if the number of odd coefficients is even, the sum of the parities of the multiple coefficients is 1. If the sign of the first significant coefficient in the sub-block is positive, the coefficients in the sub-block are adjusted so that the sum of the parities is zero. If the sign of the first significant coefficient in the sub-block is negative, the coefficients in the sub-block are adjusted so that the sum of the parities is 1. The method 1100 terminates after performing step 1150.

[0168] Fig.12 A method 1200 is shown for applying scaling to the quantized coefficients 424 to produce reconstructed residual coefficients 432. The method 1200 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. Additionally, the method 1200 may be performed by the video decoder 134 under execution by the processor 205. As such, the method 1200 may be implemented as a module of software 233 stored on a computer-readable storage medium and / or in the memory 206. The method 1200 is implemented at step 1080 of the method 1000. The method 1200 begins at a DQ test 1210.

[0169] At DQ test 1210, the video decoder 134 determines whether to use correlated quantization to dequantize the quantized coefficients 424. The video decoder 134 checks the value of the enable correlated quantization flag, which can be decoded from the bitstream 133 or inferred based on the values ​​of other high-level syntax flags. If the enable correlated quantization flag is "false", step 1210 returns "no" and the method 1200 proceeds to the inverse scalar quantization step 1220.

[0170] In one arrangement of the DQ test 1210, if the enable correlated quantization flag is "true", the video decoder 134 also checks the value of the transform skip flag of the current TB. If the enable correlated quantization flag is "true" and the transform skip flag is "true", step 1210 returns "no" and the method 1100 proceeds to the inverse scalar quantization step 1220. Otherwise, if the enable correlated quantization flag is "true" and the transform skip flag is "false", step 1210 returns "yes" and the method 1100 proceeds to the inverse correlated quantization step 1230.

[0171] In another arrangement of the DQ test 1210, if the Enable Correlated Quantization flag is "true", the video decoder 134 also checks the value of the TSRC disable flag. If the Enable Correlated Quantization flag is "true" and the TSRC disable flag is "true", step 1210 returns "no" and the method 1100 proceeds to the inverse scalar quantization step 1220. Otherwise, if the Enable Correlated Quantization flag is "true" and the TSRC disable flag is "false", step 1210 returns "yes" and the method 1100 proceeds to the inverse correlated quantization step 1230.

[0172] In yet another arrangement of the DQ test 1210, if the Enable Dependent Quantization flag is "true", the video decoder 134 also checks the value of the quantization parameter (QP) of the current TB. If the Enable Dependent Quantization flag is "true" and the QP is 4 (or any value indicating lossless operation), step 1210 returns "no" and the method 1100 proceeds to the inverse scalar quantization step 1220. Otherwise, if the Enable Dependent Quantization flag is "true" and the QP is not 4 (or a similar value indicating lossless operation), step 1210 returns "yes" and the method 1100 proceeds to the inverse dependent quantization step 1230.

[0173] At the inverse scalar quantization step 1220, the video decoder 134 scales the quantized coefficients 424 to produce reconstructed residual coefficients 432. The quantized coefficients 424 are denoted as q[n]. The reconstructed residual coefficients r[n] are produced by scaling the quantized coefficients q[n] according to the following equation (2):

[0174]

[0175] In equation (2), s is a scaling factor determined based on the QP value of the current TB. For example, if the QP is 4, s=1 and r[n]=q[n]. The method 1200 terminates when executing step 1220.

[0176] At the inverse correlation quantization step 1230, the video decoder 134 applies inverse correlation quantization to the quantized coefficients 424, thereby producing reconstructed residual coefficients 432. The quantizer state Qstate is initially reset to zero. The quantized coefficients 424 are represented as q[n]. Each coefficient position n is accessed in a reverse diagonal scan order 810, and each reconstructed residual coefficient r[n] is calculated according to equation (3):

[0177]

[0178] In equation (3), s is a scaling factor determined based on the QP value of the current TB.

[0179] After calculating each reconstructed residual coefficient r[n], the quantizer state is updated based on the parity of q[n] according to Table 1:

[0180] Previous Qstate Updated Qstate (Oddity) Updated Qstate (singularity) 0 0 2 1 2 0 2 1 3 3 3 1

[0181] Table 1: Qstate transitions

[0182] Method 1200 terminates when executing step 1230.

[0183] In order to take advantage of the statistical properties of the quantized coefficients 336, the quantized coefficients are binarized into a plurality of syntax elements by the video encoder 114 (typically by the entropy encoder 338) prior to encoding. For example, since the quantized coefficients 336 typically have a value of zero, one syntax element is a significance flag, which is set to "false" for quantized coefficients having a value of zero. If the significance flag is set to "false", no other syntax elements for the associated quantized coefficient are signaled. The significance flag may be encoded into the bitstream 133 using a context adaptive binary arithmetic coding (CABAC) entropy encoder.

[0184] Although the CABAC encoder encodes context-coded syntactic elements relatively efficiently, it is generally desirable to limit the number of context-coded syntactic elements to minimize computational requirements and hardware implementation costs. Therefore, after the entropy encoder 338 binarizes the quantized coefficients 336 into multiple syntactic elements, some syntactic elements are context-coded to the bitstream 133, while other syntactic elements are bypass-coded to the bitstream 133. The total number of context-coded syntactic element bins is limited for each transform block. In the VVC standard, the limit is set to 1.75 bins per sample. For example, for an 8×8 transform block consisting of sixty-four samples, the context-coded bin budget is set to one hundred and twelve (112) bins. In the process of encoding a TB to the bitstream 133, each time a syntactic element is context-coded, the remaining context-coded bin budget is tracked and decremented. When the remaining context-coded bin budget is exhausted, any remaining quantized coefficients and associated syntactic elements must be bypass-coded.

[0185] Fig.13 A method 1300 is shown for encoding the quantized coefficients (336) of the currently selected sub-block into the bitstream 133. The method 1300 is implemented at step 960 of the method 900. The method 1300 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. Additionally, the method 1300 may be performed by the video encoder 114 under execution by the processor 205. As such, the method 1300 may be implemented as a module of the software 233 stored on a computer-readable storage medium and / or in the memory 206. The method 1300 begins at a select first coefficient step 1310.

[0186] At select first coefficient step 1310, method 1300 selects a quantized coefficient for the current subblock. If the current subblock contains the last significant coefficient position, the currently selected coefficient is set to the last significant coefficient. Otherwise, if the current subblock does not contain the last significant coefficient position, the currently selected coefficient is set to the lower right coefficient of the current subblock. Method 1300 proceeds to use context coding check 1320.

[0187] At the used context coding check 1320, the video encoder 114 checks whether the remaining context coded bin budget is greater than or equal to 4. If the remaining context coded bin budget is greater than or equal to 4, then step 1320 returns "yes" and the method 1300 proceeds to the encode context coded syntax element step 1330. Otherwise, if the current context coded bin budget is less than 4, then step 1320 returns "no" and the method 1300 proceeds to the encode remaining pass step 1370.

[0188] At the encode context coded syntax elements step 1330, the video encoder 114 may encode the plurality of syntax elements into the bitstream 133 using a CABAC encoder that may include a validity flag, a greater than 1 flag, a parity flag, and a greater than 3 flag. Each bin associated with a syntax element is encoded by the CABAC encoder using a "context model". The context model for each bin may be selected based on the current value of the quantizer state Qstate. Additionally, each time a context coded bin is encoded by the CABAC encoder into the bitstream 133, the remaining context coded bin budget is reduced by 1 at step 1330.

[0189] If the current coefficient is the last significant coefficient at step 1330, the significance flag is set to "true", but is not encoded to the bitstream 133. If the current selected subblock is not the first or last subblock in the reverse scan order 810, and the current selected coefficient is the final coefficient (as described below in the final coefficient check 1350), and all significance flags of the previous coefficients in the current selected subblock are "false", then the significance flag is set to "true". The significance flag is not encoded to the bitstream 133. If the magnitude of the current coefficient is zero, then at step 1330, the significance flag is set to "false" and is context encoded to the bitstream 133. Otherwise, at step 1330, the significance flag is set to "true" and is context encoded to the bitstream 133.

[0190] If the magnitude of the current coefficient is 1, then at step 1330, the greater than 1 flag is set to "false" and context encoded into the bitstream 133. Otherwise, the greater than 1 flag is set to "true" and context encoded into the bitstream 133.

[0191] If the current coefficient has a magnitude of at least 2, the parity flag is set to "false" if the current coefficient is even, or to "true" if the current coefficient is odd. At step 1330, the parity flag is context coded into the bitstream 133. If the current coefficient has a magnitude greater than 3, then at step 1330, the greater than 3 flag is set to "true" and context coded into the bitstream 133. Otherwise, if the current coefficient has a magnitude of 2 or 3, the greater than 3 flag is set to "false" and context coded into the bitstream 133.

[0192] The method 1300 proceeds from step 1330 to DQ test 1340 under the control of the processor 205. Depending on the coefficient selected at 1310, the method 1300 proceeds to step 1340 after setting (or in some cases encoding) the validity flag. Otherwise, the method 1300 proceeds to step 1340 after encoding the last appropriate flag of the greater than 1 flag, the parity flag, and the greater than 3 flag.

[0193] At DQ test 1340, the same conditions as checked in DQ test 1110 are used to determine whether step 1340 returns "yes" or "no." If step 1340 returns "yes," method 1300 proceeds to update Qstate step 1345. Otherwise, if step 1340 returns "no," method 1300 proceeds to final coefficient check 1350.

[0194] At an update Qstate step 1345 , the quantizer state Qstate is updated based on the parity of the current coefficient according to Table 1. From step 1345 , the method 1300 proceeds to a final coefficient check 1350 .

[0195] At the final coefficient check 1350, the video encoder 114 checks whether the currently selected coefficient is the upper left coefficient of the currently selected sub-block. If the currently selected coefficient is the upper left coefficient of the currently selected sub-block, step 1350 returns "yes" and the method 1300 proceeds to encode the remaining pass step 1370. Otherwise, if the current coefficient is not the upper left coefficient, step 1350 returns "no" and the method 1300 proceeds to select the next coefficient step 1360.

[0196] At select next coefficient step 1360, the next coefficient of the currently selected sub-block in reverse diagonal scan order 810 is selected. From step 1360, method 1300 proceeds to use context coding check 1320.

[0197] At the encoding residual pass step 1370, any residual magnitude of the quantized coefficients of the currently selected sub-block is binarized and bypass encoded into the bitstream 133, for example, by the entropy encoder 338. For example, the quantized coefficients are encoded in a reverse diagonal scan order 810. If the quantized coefficients are context coded by the CABAC encoder (i.e., the context coding check 1320 passes (returns "yes")), then if the greater than 3 flag is "true", the quantized coefficient at scan position n has a residual magnitude r[n]. The residual magnitude is determined using equation (4):

[0198] r[n]=(x[n]-4)>>1, (4)

[0199] Wherein equation (4), x[n] is the absolute magnitude of the quantized coefficient at scan position n. The magnitude r[n] is binarized and bypass-encoded to the bitstream 133. If the quantized coefficient is not context-encoded (use context encoding check 1320 fails / returns "no"), the absolute magnitude r[n] is binarized and bypass-encoded to the bitstream 133. From step 1370, method 1300 proceeds to SBH test 1380.

[0200] At SBH test 1380, the same conditions as checked in SBH test 1140 are used to determine whether step 1380 returns "yes" or "no". If SBH test 1140 would return "no", then step 1380 returns "no" and method 1300 proceeds to encode N symbols step 1390. Otherwise, video encoder 114 checks the positions of the first and last significant coefficients of the current sub-block. If the difference between the first significant position and the last significant position is greater than 3, then step 1380 returns "yes" and method 1300 proceeds to encode N-1 symbols step 1395. Otherwise, step 1380 returns "no" and method 1300 proceeds to encode N symbols step 1390.

[0201] As described with respect to step 1140, the sign bit hiding test may depend on a number of alternative flags or settings in different implementations. If the enable sign bit hiding flag is set (has a "true" value), different implementations may make a determination based on whether the transform skip flag of the TB, the TSRC disable flag, or the QP of the TB meets a threshold associated with lossless coding. Thus, step 1380 determines whether to enable sign bit hiding based on a flag or value associated with the transform block itself or a higher level value of the TSRC disable flag. Step 1380 provides some flexibility in implementing lossless coding. An implementation that uses a flag or value associated with a transform block to determine whether to enable sign bit hiding is particularly well suited to allow flexibility when implementing lossless coding using RRC.

[0202] At the encode N symbols step 1390, the sign bits of any significant coefficients of the currently selected subblock are bypass encoded into the bitstream 133. For example, the sign bits are bypass encoded into the bitstream 133 based on the reverse diagonal scan order 810. After performing step 1390, the method 1300 terminates.

[0203] At an encode N-1 signs step 1395, the sign bits of the significant coefficients of the currently selected sub-block are bypass encoded into the bitstream 133 based on the reverse diagonal scan order 810. The sign bit associated with the first significant coefficient (which is the last accessed in the reverse diagonal scan order 810) is not encoded into the bitstream 133. In other words, if there are N significant coefficients in the currently selected sub-block, then N-1 sign bits are bypass encoded into the bitstream 133. The method 1300 terminates when executing step 1395.

[0204] Fig.14A method 1400 is shown for decoding the quantized coefficients (424) of the currently selected sub-block from the bitstream 133. The method 1400 is implemented at step 1050 of the method 1000. The method 1400 may be embodied by a device such as a configured FPGA, ASIC, or ASSP. Additionally, the method 1400 may be performed by the video decoder 134 under execution by the processor 205. As such, the method 1400 may be implemented as a module of the software 233 stored on a computer-readable storage medium and / or in the memory 206. The method 1400 begins at a select first coefficient step 1410.

[0205] At select first coefficient step 1410, method 1400 selects the first quantized coefficient of the current sub-block. If the current sub-block contains the last significant coefficient position, the currently selected coefficient is set to the last significant coefficient. Otherwise, the currently selected coefficient is set to the lower right coefficient of the current sub-block. Method 1400 proceeds from step 1410 to check using context coding step 1420.

[0206] At the use context coding check 1420, the video decoder 134 checks whether the remaining context coded bin budget meets a threshold, typically whether the remaining context coded bin budget for the transform block is greater than or equal to 4 bins. If the remaining budget is greater than or equal to 4, step 1420 returns "yes" and the method 1400 proceeds to determine the context coded syntax elements step 1430. Otherwise, if the remaining CABAC budget is less than the threshold (4 bins), step 1420 returns "no" and the method 1400 proceeds to decode the remaining pass step 1470.

[0207] At the determine context coded syntax elements step 1430, the video decoder 134 may decode a plurality of context coded syntax elements from the bitstream 133 using a CABAC encoder. Each bin associated with a syntax element is decoded by the CABAC encoder using a "context model". The context model for each bin may be selected based on the current value of the quantizer state Qstate. Additionally, each time a context coded bin is decoded from the bitstream 133 by the CABAC encoder, the remaining context coded bin budget is reduced by 1.

[0208] If the current coefficient is the last significant coefficient, the significance flag is inferred to be "true" rather than decoded from the bitstream 133. If the currently selected sub-block is not the first or last sub-block in the reverse scan order 810, and the currently selected coefficient is the final coefficient (as described below in the final coefficient check 1450), and all significance flags of the previous coefficients in the currently selected sub-block are "false", then the significance flag is inferred to be "true". Otherwise, at step 1430, the significance flag is context-decoded from the bitstream 133. If the significance flag is set to "false", the currently selected coefficient is assigned a value of zero, and the method 1400 proceeds to the DQ test 1440.

[0209] If the validity flag is set to true, then the greater than 1 flag is context decoded from the bitstream 133 at step 1430. If the greater than 1 flag is set to false, then the currently selected coefficient is assigned a magnitude of 1 and the method 1400 proceeds to DQ test 1440.

[0210] If the greater than 1 flag is set to “true”, then the parity flag and the greater than 3 flag are context decoded from the bitstream 133. The method 1400 proceeds to DQ test 1440.

[0211] The number of flags determined at step 1430 depends on the location and value of the coefficient selected at step 1410. Advancement from step 1430 may occur after inferring or decoding the significance flag, or after decoding the appropriate flag among the greater than 1 flag, the parity flag, or the greater than 3 flag.

[0212] At DQ test 1440, the same conditions as checked in DQ test 1210 are used to determine whether step 1440 returns "yes" or "no." If step 1440 returns "yes," method 1400 proceeds to update Qstate step 1445. Otherwise, if step 1440 returns "no," method 1400 proceeds to final coefficient check 1450.

[0213] At an update Qstate step 1445, the quantizer state Qstate is updated based on the parity of the currently selected coefficient according to Table 1. If the currently selected coefficient has a value of zero, the parity is zero. If the magnitude of the currently selected coefficient is 1, the parity is 1. Otherwise, if the parity flag is set to "false", the parity is zero, or if the parity flag is set to "true", the parity is 1. From step 1445, the method 1400 proceeds to a final coefficient check 1450.

[0214] At the final coefficient check step 1450, the video decoder 134 checks whether the currently selected coefficient is the upper left coefficient of the currently selected sub-block. If the currently selected coefficient is the upper left coefficient of the currently selected sub-block, then step 1450 returns "yes" and the method 1400 proceeds to a decode remainder pass step 1470. Otherwise, if the currently selected coefficient is not the upper left coefficient, then step 1450 returns "no" and the method 1400 proceeds to a select next coefficient step 1460.

[0215] At select next coefficient step 1460, the next coefficient of the currently selected sub-block in reverse diagonal scan order 810 is selected. From step 1460, method 1400 proceeds to use context coding check 1420.

[0216] At decode residual pass step 1470, any residual magnitude of the quantized coefficients of the currently selected sub-block is bypass-decoded from the bitstream 133. The quantized coefficients are processed in reverse diagonal scan order 810. If the quantized coefficient is context decoded (using context coding check 1420 pass or return "yes"), and the greater than 3 flag is decoded with a value of "true", then the residual magnitude r[n] is bypass-decoded from the bitstream 133, where n is the scan position of the quantized coefficient. The absolute magnitude x[n] of the quantized coefficient is determined as x[n] = 4 + p[n] + 2*r[n], where p[n] has a value of zero if the parity flag is decoded as "false" and has a value of 1 if the parity flag is decoded as "true".

[0217] If the quantized coefficient is context decoded and the greater than 1 flag is decoded as "true", but the greater than 3 flag is not decoded or is decoded as "false", then the absolute magnitude is determined to be x[n]=2+p[n]. If the quantized coefficient is not context decoded (use context encoding check 1420 fails and returns "no"), then the absolute magnitude x[n] is bypassed from the bitstream 133. From step 1470, the method 1400 proceeds to the SBH test 1480.

[0218] At SBH test 1480, the video decoder 134 determines whether to use sign bit hiding, i.e., whether to infer a sign bit of the currently selected sub-block. The test used at step 1480 is related to the test used at the encoder side at step 1140. The video decoder 134 checks the value of the enable sign bit hiding flag, which can be signaled as a high-level syntax in the bitstream 133. If the enable dependent quantization flag is "true", the enable sign bit hiding flag is inferred to be "false". If the enable sign bit hiding flag is "false", step 1480 returns "no" and the method 1400 proceeds to the decode sign step 1490.

[0219] The video decoder 134 checks the positions of the first and last significant coefficients of the currently selected subblock. If the difference between the first significant position and the last significant position is less than or equal to 3, step 1480 returns "no" and the method 1400 proceeds to a decode symbol step 1490.

[0220] In one arrangement of the SBH test 1480, the determination depends on the value of the enable sign bit hiding flag and the value of the transform skip flag for the current TB. If the enable sign bit hiding flag is "true" and the difference between the first significant position and the last significant position is greater than 3, the video decoder 134 also checks the value of the transform skip flag for the current TB. If the transform skip flag is "true", step 1480 returns "no" and the method 1400 proceeds to a decode symbols step 1490. Otherwise, if the transform skip flag is "false", step 1480 returns "yes" and the method 1400 proceeds to a decode and infer symbols step 1495.

[0221] In another arrangement of the SBH test 1480, the determination depends on the value of the enable sign bit hiding flag and the value of the TSRC disable flag. If the enable sign bit hiding flag is "true" and the difference between the first significant position and the last significant position is greater than 3, the video decoder 134 also checks the value of the TSRC disable flag. If the TSRC disable flag is "true", step 1480 returns "no" and the method 1400 proceeds to the decode symbol step 1490. Otherwise, if the TSRC disable flag is "false", step 1480 returns "yes" and the method 1400 proceeds to the decode and infer symbol step 1495.

[0222] In yet another arrangement of the SBH test 1480, the determination depends on the value of the enable sign bit hiding flag and the value of the quantization parameter (QP) of the current TB, the difference between the first significant position and the last significant position of the quantization parameter QP of the transform block. If the enable sign bit hiding flag is "true" and the difference between the first significant position and the last significant position is greater than 3, the video decoder 134 also checks the value of the QP of the current TB. If the QP is 4 (or any value indicating lossless operation), step 1480 returns "no" and the method 1400 proceeds to a decode sign step 1490. Otherwise, if the QP is not 4 (or a similar value indicating lossless operation), step 1480 returns "yes" and the method 1400 proceeds to a decode and infer sign step 1495.

[0223] At decode sign pass step 1490, the sign bits of any significant coefficients of the currently selected sub-block are bypass decoded from the bitstream 133. The sign bits are bypass decoded from the bitstream 133 in reverse diagonal scan order 810. If the value of the associated sign bit is 1, the value of the quantized coefficient is set to -x[n]. If the value of the associated sign bit is zero, the value of the quantized coefficient is set to x[n]. Method 1400 terminates when step 1490 is performed.

[0224] At the decode and infer sign step 1495, the sign bits of the significant coefficients of the currently selected sub-block are bypass decoded from the bitstream 133 in the reverse diagonal scan order 810. The sign bit associated with the first significant coefficient (which is the last accessed in the reverse diagonal scan order 810) is not decoded from the bitstream 133. In other words, if there are N significant coefficients in the currently selected sub-block, N-1 sign bits are bypass decoded from the bitstream 133. The sign bit associated with the first significant coefficient is inferred based on the sum of the parities of the significant coefficients. If the sum of the parities is zero, the sign bit associated with the first significant coefficient is inferred to be zero. If the sum of the parities is 1, the sign bit associated with the first significant coefficient is inferred to be 1. If the value of the associated sign bit is 1, the value of the quantized coefficient is set to -x[n]. If the value of the associated sign bit is zero, the value of the quantized coefficient is set to x[n]. Then, the method 1400 terminates.

[0225] The arrangements described in methods 900 and 1000 allow lossless compression of video data while using a conventional residual coding process. Dependent quantization and sign bit hiding are lossy coding tools that are flexibly disabled when lossless operation is desired, but can still be used to achieve improved coding performance in lossy coded blocks.

[0226] Industrial Applicability

[0227] The described arrangement is suitable for use in the computer and data processing industry, and in particular in digital signal processing for encoding and decoding signals such as video and image signals, thereby achieving high compression efficiency.

[0228] The foregoing describes only some embodiments of the present invention, and modifications and / or changes may be made thereto without departing from the scope and spirit of the present invention, the embodiments being illustrative rather than restrictive.

[0229] CROSS-REFERENCE TO RELATED APPLICATIONS

[0230] This application claims the benefit under 35 U.S.C. §119 of the filing date of Australian patent application 2020201753, filed on March 10, 2020, the entire contents of which are incorporated herein by reference as if fully set forth herein.

Claims

1. A method for decoding a transform block from a bitstream, the method comprising: decoding a first flag for determining whether correlated quantization is used in the transform block; determining whether to use sign bit hiding in the transform block, wherein in the sign bit hiding, data indicating the sign of a significant coefficient in a given position of the transform block is not decoded from the bitstream, Wherein, when the first flag is true, the sign bit hiding is not used in the transform block; In a case where it is determined based on the first flag that the correlated quantization is used in the transform block, decoding the transform block by using the correlated quantization; and In a case where it is determined that the sign bit hiding is used in the transform block, decoding the transform block by using the sign bit hiding, wherein, when (a) a disable flag for skipping residual coding for transform is checked after checking the first flag, and (b) the disable flag is true, the correlated quantization is not used in the transform block, wherein, when the disable flag is true, the sign bit hiding is not used in the transform block even if the enable flag for the sign bit hiding is true, and The disable flag indicates whether to apply the first residual coding instead of the second residual coding even if the transform process is skipped, the first residual coding is used for the block that does not skip the transform process, and the second residual coding is used for the block that skips the transform process.

2. The method according to claim 1, in, The first flag is not a flag of a sequence parameter set.

3. The method according to claim 1, wherein: When the sign bit hiding is not used, the same number of signs as the number of significant coefficients in the sub-block are decoded.

4. The method according to claim 1, in, In case the sign bit hiding is to be used, at least the enable flag is true.

5. A method of encoding a transform block in a bitstream, the method comprising: encoding a first flag for determining whether correlated quantization is used in the transform block; determining whether to use sign bit hiding in the transform block, wherein data indicating the sign of a significant coefficient in a given position of the transform block is not encoded, Wherein, when the first flag is true, the sign bit hiding is not used in the transform block; In a case where it is determined based on the first flag that the correlated quantization is used in the transform block, encoding the transform block by using the correlated quantization; and In a case where it is determined to use the sign bit hiding in the transform block, encoding the transform block by using the sign bit hiding, wherein, when (a) a disable flag for skipping residual coding for transform is checked after checking the first flag, and (b) the disable flag is true, the correlated quantization is not used in the transform block, wherein, when the disable flag is true, the sign bit hiding is not used in the transform block even if the enable flag for the sign bit hiding is true, and The disable flag indicates whether to apply the first residual coding instead of the second residual coding even if the transform process is skipped, the first residual coding is used for the block that does not skip the transform process, and the second residual coding is used for the block that skips the transform process.

6. The method according to claim 5, in, The first flag is not a flag of a sequence parameter set.

7. The method according to claim 5, in, Without using the sign bit hiding, for a sub-block, a number of signs equal to the number of significant coefficients in the sub-block are encoded.

8. The method according to claim 5, in, In case the sign bit hiding is to be used, at least the enable flag is true.

9. A decoding device for decoding a transform block from a bit stream, the decoding device comprising: A processor for: decoding a first flag for determining whether correlated quantization is used in the transform block; determining whether to use sign bit hiding in the transform block, wherein in the sign bit hiding, data indicating the sign of a significant coefficient in a given position of the transform block is not decoded from the bitstream, Wherein, when the first flag is true, the sign bit hiding is not used in the transform block; In a case where it is determined based on the first flag that the correlated quantization is used in the transform block, decoding the transform block by using the correlated quantization; and In a case where it is determined that the sign bit hiding is used in the transform block, decoding the transform block by using the sign bit hiding, wherein, when (a) a disable flag for skipping residual coding for transform is checked after checking the first flag, and (b) the disable flag is true, the correlated quantization is not used in the transform block, wherein, when the disable flag is true, the sign bit hiding is not used in the transform block even if the enable flag for the sign bit hiding is true, and The disable flag indicates whether to apply the first residual coding instead of the second residual coding even if the transform process is skipped, the first residual coding is used for the block that does not skip the transform process, and the second residual coding is used for the block that skips the transform process.

10. An encoding device for encoding a transform block in a bit stream, the encoding device comprising: A processor for: encoding a first flag for determining whether correlated quantization is used in the transform block; determining whether to use sign bit hiding in the transform block, wherein in the sign bit hiding, data indicating a sign of a significant coefficient in a given position of the transform block is not encoded, wherein if the first flag is true, the sign bit hiding is not used in the transform block; In a case where it is determined based on the first flag that the correlated quantization is used in the transform block, encoding the transform block by using the correlated quantization; and In a case where it is determined to use the sign bit hiding in the transform block, encoding the transform block by using the sign bit hiding, wherein, when (a) a disable flag for skipping residual coding for transform is checked after checking the first flag, and (b) the disable flag is true, the correlated quantization is not used in the transform block, wherein, when the disable flag is true, the sign bit hiding is not used in the transform block even if the enable flag for the sign bit hiding is true, and The disable flag indicates whether to apply the first residual coding instead of the second residual coding even if the transform process is skipped, the first residual coding is used for the block that does not skip the transform process, and the second residual coding is used for the block that skips the transform process.

11. A non-transitory computer-readable storage medium storing instructions that cause a computer to perform a method of decoding a transform block from a bitstream, the method comprising: decoding a first flag for determining whether correlated quantization is used in the transform block; determining whether to use sign bit hiding in the transform block, wherein in the sign bit hiding, data indicating the sign of a significant coefficient in a given position of the transform block is not decoded from the bitstream; and In a case where it is determined that the sign bit hiding is used in the transform block, decoding the transform block by using the sign bit hiding, wherein, when (a) a disable flag for transform skipping residual coding is checked after checking the first flag, and (b) the disable flag is true, the correlated quantization is not used in the transform block, wherein when the disable flag is true, the sign bit hiding is not used in the transform block even if the enable flag for the sign bit hiding is true, and The disable flag indicates whether to apply the first residual coding instead of the second residual coding even if the transform process is skipped, the first residual coding is used for the block that does not skip the transform process, and the second residual coding is used for the block that skips the transform process.

12. A non-transitory computer-readable storage medium storing instructions that cause a computer to perform a method of encoding a transform block in a bitstream, the method comprising: encoding a first flag for determining whether correlated quantization is used in the transform block; determining whether to use sign bit hiding in the transform block, wherein data indicating the sign of a significant coefficient in a given position of the transform block is not encoded, Wherein, when the first flag is true, the sign bit hiding is not used in the transform block; In a case where it is determined based on the first flag that the correlated quantization is used in the transform block, encoding the transform block by using the correlated quantization; and In a case where it is determined to use the sign bit hiding in the transform block, encoding the transform block by using the sign bit hiding, wherein, when (a) a disable flag for skipping residual coding for transform is checked after checking the first flag, and (b) the disable flag is true, the correlated quantization is not used in the transform block, wherein, when the disable flag is true, the sign bit hiding is not used in the transform block even if the enable flag for the sign bit hiding is true, and The disable flag indicates whether to apply the first residual coding instead of the second residual coding even if the transform process is skipped, the first residual coding is used for the block that does not skip the transform process, and the second residual coding is used for the block that skips the transform process.

Images