Method and system for sub-block transformation for video coding and decoding

Abstract

A computing device performs a method for decoding video data in the following manner: receiving a first syntax element associated with an SBT codec unit from a bitstream, the first syntax element indicating that there is at least one non-zero transform coefficient in the SBT codec unit; determining a first transform unit including a non-zero transform coefficient; receiving a second syntax element associated with a first chroma component of the first transform unit, a third syntax element associated with a second chroma component, and a fourth syntax element associated with a luminance component of the first transform unit; decoding transform coefficients of chroma components and luminance components of the first transform unit based on the second, third and fourth syntax elements; and setting transform coefficients of luminance components and chroma components of the rest of the plurality of transform units to zero.

Description

[0001] This application is a divisional application of Chinese patent application 202080043251.1. Technical Field

[0002] The present application relates generally to video data encoding and decoding, and in particular, to methods and systems for sub-block transform (SBT) for video encoding and decoding. Background Art

[0003] Digital video is supported by a wide variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc. Electronic devices transmit, receive, encode, decode and / or store digital video data by implementing video compression / decompression standards as defined by MPEG-4, ITU-T H.263, ITU-TH.264 / MPEG-4 Part 10, Advanced Video Codec (AVC), High Efficiency Video Codec (HEVC) and Versatile Video Codec (VVC) standards. Video compression typically includes performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove redundancy inherent in video data. For block-based video codecs, a video frame is divided into one or more slices, each with multiple video blocks, which may also be referred to as codec tree units (CTUs). Each CTU may contain a codec unit (CU), or may be recursively split into smaller CUs until a predefined minimum CU size is reached. Each CU (also called a leaf-CU) contains one or more transform units (TUs), and each CU also contains one or more prediction units (PUs). Each CU can be coded and decoded in intra mode, inter mode, or IBC mode. Video blocks in an intra-coded (I) slice of a video frame are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same video frame. Video blocks in an inter-coded (P or B) slice of a video frame can use spatial prediction with respect to reference samples in neighboring blocks in the same video frame, or use temporal prediction with respect to reference samples in other previous and / or future reference video frames.

[0004] Based on spatial or temporal prediction of previously encoded reference blocks (e.g., neighboring blocks), a prediction block for the current video block to be encoded is generated. The process of finding the reference block can be accomplished by a block matching algorithm. The residual data representing the pixel differences between the current block to be encoded and the prediction block is called a residual block or prediction error. Inter-frame encoded blocks are encoded according to motion vectors and residual blocks, and the motion vectors point to reference blocks in the reference frame that form the prediction block. The process of determining motion vectors is typically called motion estimation. Intra-frame encoded blocks are encoded according to intra-frame prediction modes and residual blocks. For further compression, the residual block is transformed from the pixel domain to the transform domain (e.g., the frequency domain) to produce residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to produce a one-dimensional vector of transform coefficients and then entropy encoded into the video bitstream to achieve even greater compression.

[0005] The encoded video bitstream is then stored in a computer-readable storage medium (e.g., flash memory) for access by another electronic device with digital video capabilities, or transmitted directly to the electronic device by wire or wirelessly. The electronic device then performs video decompression (which is the reverse process of the video compression described above) by, for example, parsing the encoded video bitstream to obtain syntax elements from the bitstream, and reconstructing digital video data from the encoded video bitstream into its original format based at least in part on the syntax elements obtained from the bitstream; and the electronic device presents the reconstructed digital video data on a display of the electronic device. Summary of the invention

[0006] The present application describes implementations related to video data encoding and decoding, and more particularly, the implementations relate to improved systems and methods for sub-block transforms in video encoding and decoding.

[0007] According to a first aspect of the present application, a method for decoding a sub-block transform (SBT) codec unit, wherein the SBT codec unit is divided into a plurality of transform units, the method comprising: receiving a first syntax element associated with the SBT codec unit from a bitstream, wherein the first syntax element indicates that there is at least one non-zero transform coefficient in the SBT codec unit; in response to receiving the first syntax element indicating that there is at least one non-zero transform coefficient in the SBT codec unit, determining a first transform unit including the non-zero transform coefficient in the plurality of transform units; receiving a second syntax element, a third syntax element, and a fourth syntax element from the bitstream , a second syntax element is associated with a first chroma component of a first transform unit to indicate whether all transform coefficients of the first chroma component are zero, a third syntax element is associated with a second chroma component of the first transform unit to indicate whether all transform coefficients of the second chroma component are zero, and a fourth syntax element is associated with a luma component of the first transform unit to indicate whether all transform coefficients of the luma component are zero; decoding transform coefficients of the chroma components and the luma components of the first transform unit based on the second, third, and fourth syntax elements from the bitstream; and setting transform coefficients of the luma components and the chroma components of the rest of the plurality of transform units to zero.

[0008] According to a second aspect of the present application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory. When the programs are executed by the one or more processing units, the electronic device performs the operations described above.

[0009] According to a third aspect of the present application, a non-transitory computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units. When the programs are executed by the one or more processing units, the electronic device performs the operations described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are included to provide a further understanding of the implementations and are incorporated herein and constitute a part of the specification, illustrate the described implementations and together with the description serve to explain the underlying principles. Like reference numerals designate corresponding parts.

[0011] Figure 1 A block diagram is shown to illustrate an exemplary video encoding and decoding system in accordance with some implementations of the present disclosure.

[0012] Figure 2 A block diagram is shown to illustrate an exemplary video encoder in accordance with some implementations of the present disclosure.

[0013] Figure 3 A block diagram is shown to illustrate an exemplary video decoder according to some implementations of the present disclosure.

[0014] FIG. 4A to FIG. 4F A block diagram illustrating how a frame is recursively quadtree partitioned into a plurality of video blocks of different sizes and shapes according to some implementations of the present disclosure.

[0015] Figure 5A-Figure 5B A block diagram illustrating different sub-block transform partitions of a leaf codec block in accordance with some implementations of the present disclosure.

[0016] Figure 6 The present invention is a flowchart illustrating a process of decoding a sub-block transform codec unit according to some implementations of the present disclosure. DETAILED DESCRIPTION

[0017] Reference will now be made in detail to specific implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, many non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be implemented without these specific details. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0018] Figure 1 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some implementations of the present disclosure. Figure 1 As shown, system 10 includes a source device 12 that generates and encodes video data to be decoded at a later time by a target device 14. Source device 12 and target device 14 may include any of a wide variety of electronic devices, including desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and target device 14 are equipped with wireless communication capabilities.

[0019] In some implementations, the target device 14 may receive the encoded video data to be decoded via the link 16. The link 16 may include any type of communication medium or device capable of moving the encoded video data from the source device 12 to the target device 14. In one example, the link 16 may include a communication medium that enables the source device 12 to directly transmit the encoded video data to the target device 14 in real time. The encoded video data may be modulated according to a communication standard such as a wireless communication protocol and transmitted to the target device 14. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form a portion of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other equipment that may be useful for facilitating communication from the source device 12 to the target device 14.

[0020] In some other implementations, the encoded video data may be transferred from the output interface 22 to the storage device 32. Subsequently, the encoded video data in the storage device 32 may be accessed by the target device 14 via the input interface 28. The storage device 32 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a DVD, a CD-ROM, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device 32 may correspond to a file server or another intermediate storage device that may hold the encoded video data generated by the source device 12. The target device 14 may access the stored video data from the storage device 32 via streaming or downloading. The file server may be any type of computer capable of storing the encoded video data and transmitting the encoded video data to the target device 14. Exemplary file servers include a web server (e.g., for a website), an FTP server, a network attached storage (NAS) device, or a local disk drive. The target device 14 can access the encoded video data through any standard data connection suitable for accessing the encoded video data stored on the file server, including a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of the two. The transmission of the encoded video data from the storage device 32 can be a streaming transmission, a download transmission, or a combination of the two.

[0021] like Figure 1As shown, source device 12 includes video source 18, video encoder 20 and output interface 22. Video source 18 may include a source such as a video capture device, such as a video camera, a video archive containing previously captured video, a video feed interface receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video, or a combination of such sources. As an example, if video source 18 is a camera of a security monitoring system, source device 12 and target device 14 may form a camera phone or a video phone. However, the implementation described in the present application may be generally applicable to video encoding and decoding, and may be applied to wireless and / or wired applications.

[0022] The captured video, pre-captured video, or computer-generated video may be encoded by the video encoder 20. The encoded video data may be transmitted directly to the target device 14 via the output interface 22 of the source device 12. The encoded video data may also (or alternatively) be stored on a storage device 32 for later access by the target device 14 or other device for decoding and / or playback. The output interface 22 may further include a modem and / or a transmitter.

[0023] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included in the encoded video data transmitted over a communication medium, stored on a storage medium, or stored on a file server.

[0024] In some implementations, the target device 14 may include a display device 34, which may be an integrated display device as well as an external display device configured to communicate with the target device 14. The display device 34 displays the decoded video data to a user, and may include any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0025] The video encoder 20 and the video decoder 30 can operate in accordance with a proprietary standard or an industry standard, such as VVC, HEVC, MPEG-4 Part 10, Advanced Video Codec (AVC), or an extension of such a standard. It should be understood that the present application is not limited to a specific video codec / decoding standard and can be applied to other video codec / decoding standards. It is generally believed that the video encoder 20 of the source device 12 can be configured to encode the video data in accordance with any of these current or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 can be configured to decode the video data in accordance with any of these current or future standards.

[0026] The video encoder 20 and the video decoder 30 can each be implemented as any of a variety of suitable codec circuit systems, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device can store instructions for the software in a suitable non-transitory computer-readable medium and execute these instructions in hardware using one or more processors to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 can be included in one or more encoders or decoders, either of which can be integrated in the corresponding device as part of a combined encoder / decoder (CODEC).

[0027] Figure 2 1 is a block diagram illustrating an exemplary video encoder 20 in accordance with some implementations described herein. The video encoder 20 may perform intra- and inter-prediction codecs of video blocks within video frames. Intra-prediction codecs rely on spatial predictions to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-prediction codecs rely on temporal predictions to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence.

[0028] like Figure 2As shown, the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copy (BC) unit 48. In some implementations, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and a summer 62 for video block reconstruction. A deblocking filter (not shown) can be placed between the summer 62 and the DPB 64 to filter the block boundaries so as to remove blocking artifacts from the reconstructed video. In addition to the deblocking filter, a loop filter (not shown) can also be used to filter the output of the summer 62. The video encoder 20 can take the form of a fixed or programmable hardware unit, or can be divided into one or more hardware units in the fixed or programmable hardware units shown.

[0029] The video data memory 40 can store video data to be encoded by the components of the video encoder 20. The video data in the video data memory 40 can be obtained, for example, from the video source 18. The DPB 64 is a buffer that stores reference video data used by the video encoder 20 when encoding the video data (e.g., in an intra-frame or inter-frame prediction codec mode). The video data memory 40 and the DPB 64 can be formed by any of a variety of memory devices. In various different examples, the video data memory 40 can be on-chip with other components of the video encoder 20, or off-chip relative to those components.

[0030] like Figure 2 As shown, after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into strips, tiles, or other larger codec units (CUs) according to a predefined splitting structure associated with the video data (such as a quadtree structure). The video frame may be divided into a plurality of video blocks (or a set of video blocks referred to as tiles). The prediction processing unit 41 may select one of a plurality of possible prediction codec modes for the current video block based on error results (e.g., codec rate and distortion level), such as one of a plurality of intra-frame prediction codec modes or one of a plurality of inter-frame prediction codec modes. The prediction processing unit 41 may provide the resulting intra-frame or inter-frame prediction codec block to the summer 50 to generate a residual block, and to the summer 62 to reconstruct the coded block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements such as motion vectors, intra-frame mode indicators, segmentation information, and other such syntax information to the entropy coding unit 56.

[0031] In order to select an appropriate intra-prediction codec mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction codec of the current video block relative to one or more neighboring blocks in the same frame as the current block to be coded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 perform inter-prediction codec of the current video block relative to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple codec passes, for example, to select an appropriate codec mode for each block of video data.

[0032] In some implementations, motion estimation unit 42 determines the inter-prediction mode for a current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames, the motion vector indicating the displacement of a prediction unit (PU) of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors that estimate motion for video blocks. The motion vector may, for example, indicate the displacement of a PU of a video block within a current video frame or picture relative to a prediction block within a reference frame (or other unit of codec), the prediction block being relative to a current block being coded within the current frame (or other unit of codec). The predetermined pattern may designate video frames in a sequence as P-frames or B-frames. Intra BC unit 48 may determine vectors, such as block vectors, for intra BC coding in a manner similar to how motion estimation unit 42 determines motion vectors for inter prediction, or may utilize motion estimation unit 42 to determine block vectors.

[0033] A prediction block is a block of a reference frame that is considered to closely match a PU of a video block to be encoded in terms of pixel differences, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some implementations, video encoder 20 may calculate values ​​for sub-integer pixel positions of reference frames stored in DPB 64. For example, video encoder 20 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference frame. Thus, motion estimation unit 42 may perform motion searches relative to full pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision.

[0034] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-frame prediction codec frame by comparing the position of the PU with the position of a prediction block of a reference frame selected from a first reference frame list (list 0) or a second reference frame list (list 1), each of which identifies one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to motion compensation unit 44 and then to entropy encoding unit 56.

[0035] Motion compensation performed by motion compensation unit 44 may involve fetching or generating a prediction block based on the motion vector determined by motion estimation unit 42. Upon receiving the motion vector for the PU of the current video block, motion compensation unit 44 may locate the prediction block pointed to by the motion vector in one of the reference frame lists, retrieve the prediction block from DPB 64, and forward the prediction block to summer 50. Summer 50 then forms a residual video block of pixel difference values ​​by subtracting the pixel values ​​of the prediction block provided by motion compensation unit 44 from the pixel values ​​of the current video block being encoded and decoded. The pixel difference values ​​forming the residual video block may include a luma difference component or a chroma difference component or both. Motion compensation unit 44 may also generate syntax elements associated with the video block of the video frame for use by video decoder 30 when decoding the video block of the video frame. These syntax elements may include, for example, syntax elements defining a motion vector used to identify the prediction block, any flag indicating a prediction mode, or any other syntax information described herein. Note that motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes.

[0036] In some implementations, the intra BC unit 48 may generate vectors and fetch prediction blocks in a manner similar to that described above in conjunction with the motion estimation unit 42 and the motion compensation unit 44, but the prediction block is in the same frame as the current block being encoded and decoded, and the vector is referred to as a block vector rather than a motion vector. In particular, the intra BC unit 48 may determine the intra prediction mode used to encode the current block. In some examples, the intra BC unit 48 may encode the current block using various different intra prediction modes, for example, during multiple separate encoding passes, and test their performance through rate-distortion analysis. Next, the intra BC unit 48 may select a suitable intra prediction mode to use among the various tested intra prediction modes, and generate an intra mode indicator accordingly. For example, the intra BC unit 48 may calculate rate-distortion values ​​for the various tested intra prediction modes using rate-distortion analysis, and select the intra prediction mode with the best rate-distortion characteristics among the tested modes as the suitable intra prediction mode to be used. The rate-distortion analysis typically determines the amount of distortion (or error) between a coded block and the original uncoded block that was coded to produce the coded block, as well as the bit rate (i.e., the number of bits) used to produce the coded block. The intra BC unit 48 may calculate a ratio from the distortion and rate for the various different coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0037] In other examples, intra BC unit 48 may use, in whole or in part, motion estimation unit 42 and motion compensation unit 44 to perform such functions for intra BC prediction in accordance with implementations described herein. In either case, for intra block copying, the prediction block may be a block that is considered to closely match the block to be encoded in terms of pixel differences, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics, and identification of the prediction block may include calculating values ​​for sub-integer pixel positions.

[0038] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from a different frame according to inter-frame prediction, video encoder 20 can form a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded and decoded to form pixel difference values. The pixel difference values ​​forming the residual video block can include both luma component differences and chroma component differences.

[0039] As an alternative to the inter-frame prediction performed by the motion estimation unit 42 and the motion compensation unit 44, or the intra-frame block copy prediction performed by the intra BC unit 48 as described above, the intra-frame prediction processing unit 46 can perform intra-frame prediction on the current video block. In particular, the intra-frame prediction processing unit 46 can determine the intra-frame prediction mode used to encode the current block. To do so, the intra-frame prediction processing unit 46 can encode the current block using a variety of different intra-frame prediction modes, for example, during multiple separate encoding passes, and the intra-frame prediction processing unit 46 (or in some examples, the mode selection unit) can select the appropriate intra-frame prediction mode to be used from the tested intra-frame prediction modes. The intra-frame prediction processing unit 46 can provide information indicating the selected intra-frame prediction mode for the block to the entropy coding unit 56. The entropy coding unit 56 can encode the information indicating the selected intra-frame prediction mode in the bitstream.

[0040] After prediction processing unit 41 determines a prediction block for the current video block via inter-prediction or intra-prediction, summer 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more transform units (TUs) and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform.

[0041] The transform processing unit 52 may send the resulting transform coefficients to a quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan of the matrix comprising the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0042] After quantization, entropy coding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be transmitted to video decoder 30, or archived in storage device 32 for later transmission to or retrieval by video decoder 30. Entropy coding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded or decoded.

[0043] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain in order to generate a reference block for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0044] Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reference block for storage in DPB 64. The reference block may then be used as a prediction block by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 to inter-predict another video block in a subsequent video frame.

[0045] Figure 3 1 is a block diagram of an exemplary video decoder 30 according to some implementations of the present disclosure. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, a summer 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction processing unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform operations generally associated with the above-mentioned video encoder 20. Figure 2 For example, the motion compensation unit 82 may generate prediction data based on the motion vector received from the entropy decoding unit 80, and the intra-prediction unit 84 may generate prediction data based on the intra-prediction mode indicator received from the entropy decoding unit 80.

[0046] In some examples, units of the video decoder 30 may be tasked to perform implementations of the present application. Likewise, in some examples, implementations of the present disclosure may be divided among one or more of the units of the video decoder 30. For example, the intra BC unit 85 may perform implementations of the present application alone or in combination with other units of the video decoder 30 (such as the motion compensation unit 82, the intra prediction processing unit 84, and the entropy decoding unit 80). In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81 (such as the motion compensation unit 82).

[0047] The video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of the video decoder 30. The video data stored in the video data memory 79 may be obtained, for example, from the storage device 32, from a local video source (such as a camera), via a wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). The video data memory 79 may include a coded and decoded picture buffer (CPB) that stores encoded video data from an encoded video bitstream. A decoded picture buffer (DPB) 92 of the video decoder 30 stores reference video data for use by the video decoder 30 when decoding video data (e.g., in intra-frame or inter-frame prediction codec modes). The video data memory 79 and the DPB 92 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For purposes of illustration, the video data memory 79 and the DPB 92 are described in detail in detail in detail. Figure 3 92 are depicted as two different components of the video decoder 30. However, it will be apparent to those skilled in the art that the video data memory 79 and the DPB 92 may be provided by the same memory device or by separate memory devices. In some examples, the video data memory 79 may be on-chip with other components of the video decoder 30, or off-chip relative to those components.

[0048] During the decoding process, the video decoder 30 receives an encoded video bitstream, which represents video blocks and associated syntax elements of an encoded video frame. The video decoder 30 may receive syntax elements at the video frame level and / or at the video block level. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors and other syntax elements to the prediction processing unit 81.

[0049] When a video frame is encoded and decoded as an intra-frame prediction codec (I) frame or an intra-frame codec prediction block in other types of frames, the intra-frame prediction processing unit 84 of the prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra-frame prediction mode transmitted by the signal and the reference data from the previously decoded block of the current frame.

[0050] When the video frame is encoded and decoded as an inter-frame prediction codec (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vectors and other syntax elements received from the entropy decoding unit 80. Each of these prediction blocks can be generated from a reference frame in one of the reference frame lists. The video decoder 30 can construct the reference frame lists - List 0 and List 1 - based on the reference frames stored in the DPB 92 using a default construction technique.

[0051] In some examples, when a video block is encoded and decoded in accordance with the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates prediction blocks for the current video block based on the motion vectors and other syntax elements received from entropy decoding unit 80. These prediction blocks may be within a reconstructed region of the same picture as the current video block as defined by video encoder 20.

[0052] The motion compensation unit 82 and / or the intra BC unit 85 determine prediction information for the video block of the current video frame by parsing the motion vector and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded. For example, the motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode (e.g., intra or inter prediction) used to encode and decode the video block of the video frame, the inter prediction frame type (e.g., B or P), the construction information of one or more reference frame lists in the reference frame list for the frame, the motion vector for each inter prediction coded video block of the frame, the inter prediction state for each inter prediction coded video block of the frame, and other information for decoding the video block in the current video frame.

[0053] Similarly, the intra BC unit 85 may use some of the received syntax elements (e.g., flags) to determine whether the current video block is predicted using intra BC mode, construction information of which video blocks of the frame are within the reconstruction region and should be stored in the DPB 92, block vectors for each intra BC predicted video block of the frame, intra BC prediction status for each intra BC predicted video block of the frame, and other information for decoding video blocks in the current video frame.

[0054] Motion compensation unit 82 may also perform interpolation using interpolation filters as used by video encoder 20 during encoding of the video block to calculate interpolated values ​​for sub-integer pixels of a reference block. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 from the received syntax elements and use those interpolation filters to produce the prediction block.

[0055] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.

[0056] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the summer 90 reconstructs the decoded video block for the current video block by summing the residual block from the inverse transform processing unit 88 and the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter (not shown) can be placed between the summer 90 and the DPB 92 to further process the decoded video block. The decoded video block in a given frame is then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 can also store the decoded video for later display on a display device (such as Figure 1 is presented on a display device 34).

[0057] In a typical video encoding and decoding process, a video sequence typically includes an ordered set of frames or pictures. Each frame may include three sample arrays, denoted as SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In other cases, a frame may be monochrome and therefore include only a two-dimensional array of luma samples.

[0058] like Figure 4A As shown, the video encoder 20 (or more specifically, the segmentation unit 45) generates an encoded representation of the frame by first segmenting the frame into a set of codec tree units (CTUs). A video frame may include an integer number of CTUs sequentially ordered from left to right and from top to bottom in a raster scan order. Each CTU is the largest logical codec unit, and the width and height of the CTU are signaled by the video encoder 20 in a sequence parameter set so that all CTUs in a video sequence have the same size, which is one of 128×128, 64×64, 32×32, and 16×16. However, it should be noted that the present application is not necessarily limited to a specific size. Figure 4BAs shown, each CTU may include one codec tree block (CTB) of luma samples, two corresponding codec tree blocks of chroma samples, and syntax elements used to encode and decode samples of the codec tree blocks. These syntax elements describe the properties of different types of units of codec pixel blocks and how the video sequence can be reconstructed at the video decoder 30, including inter-frame or intra-frame prediction, intra-frame prediction mode, motion vectors and other parameters. In a monochrome picture or a picture with three separate color planes, a CTU may include a single codec tree block and syntax elements used to encode and decode samples of the codec tree block. The codec tree block may be an NxN sample block.

[0059] To achieve better performance, the video encoder 20 may recursively perform tree partitioning, such as binary tree partitioning, ternary tree partitioning, quadtree partitioning, or a combination of the two, on the codec tree block of the CTU and divide the CTU into smaller codec units (CUs). Figure 4C As depicted in , a 64x64 CTU 400 is first divided into four smaller CUs, each CU having a block size of 32x32. Among the four smaller CUs, CU 410 and CU 420 are each divided into four CUs with a block size of 16x16. Two 16x16 CUs 430 and CU 440 are each further divided into four CUs with a block size of 8x8. Figure 4D Depicts a quadtree data structure that illustrates how Figure 4C As the final result of the partitioning process of the CTU 400 depicted in FIG, each leaf node of the quadtree corresponds to a CU of various sizes ranging from 32x32 to 8x8. Figure 4B As in the CTU depicted in , each CU may include a codec block (CB) of luma samples and two corresponding codec blocks of chroma samples of the same size frame, and syntax elements used to encode and decode the samples of the codec block. In monochrome pictures or pictures with three separate color planes, a CU may include a single codec block and syntax structures used to encode and decode the samples of the codec block. It should be noted that Figure 4C and Figure 4D The quadtree partitioning depicted in FIG is for illustration purposes only, and a CTU can be split into CUs based on quadtree / ternary / binary tree partitioning to accommodate varying local characteristics. In a multi-type tree structure, a CTU is partitioned by a quadtree structure, and the leaf CU of each quadtree can be further partitioned by a binary and ternary tree structure. Figure 4E As shown, there are five possible partition types for a codec block with width W and height H, namely, quadruple partition, horizontal binary partition, vertical binary partition, horizontal ternary partition, and vertical ternary partition.

[0060] In some implementations, the video encoder 20 may further partition the codec block of the CU into one or more MxN prediction blocks (PBs). A prediction block is a rectangular (square or non-square) block of samples on which the same prediction (inter or intra) is applied. The prediction unit (PU) of a CU may include a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and syntax elements used to predict the prediction blocks. In a monochrome picture or a picture with three separate color planes, a PU may include a single prediction block and a syntax structure used to predict the prediction block. The video encoder 20 may generate a predicted luma block, a predicted Cb block, and a predicted Cr block for the luma prediction block, the Cb prediction block, and the Cr prediction block of each PU of the CU.

[0061] Video encoder 20 may generate a prediction block for a PU using intra prediction or inter prediction. If video encoder 20 generates a prediction block for a PU using intra prediction, video encoder 20 may generate the prediction block for the PU based on decoded samples of a frame associated with the PU. If video encoder 20 generates a prediction block for a PU using inter prediction, video encoder 20 may generate the prediction block for the PU based on decoded samples of one or more frames other than the frame associated with the PU.

[0062] After the video encoder 20 generates the predicted luma block, the predicted Cb block, and the predicted Cr block for one or more PUs of a CU, the video encoder 20 may generate a luma residual block for the CU by subtracting the predicted luma block of the CU from its original luma codec block, so that each sample in the luma residual block of the CU indicates the difference between a luma sample in one of the predicted luma blocks of the CU and a corresponding sample in the original luma codec block of the CU. Similarly, the video encoder 20 may generate a Cb residual block and a Cr residual block for the CU, respectively, so that each sample in the Cb residual block of the CU indicates the difference between a Cb sample in one of the predicted Cb blocks of the CU and a corresponding sample in the original Cb codec block of the CU, and each sample in the Cr residual block of the CU may indicate the difference between a Cr sample in one of the predicted Cr blocks of the CU and a corresponding sample in the original Cr codec block of the CU.

[0063] In addition, if Figure 4CAs shown, the video encoder 20 can use quadtree partitioning to decompose the luma residual block, Cb residual block and Cr residual block of the CU into one or more luma transform blocks, Cb transform blocks and Cr transform blocks. A transform block is a rectangular (square or non-square) block of samples on which the same transform is applied. The transform unit (TU) of the CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements used to transform the transform block samples. Therefore, each TU of the CU may be associated with a luma transform block, a Cb transform block and a Cr transform block. In some examples, the luma transform block associated with the TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In a monochrome picture or a picture with three separate color planes, a TU may include a single transform block and a syntax structure used to transform the samples of the transform block.

[0064] The video encoder 20 may apply one or more transforms to the luma transform block of the TU to generate a luma coefficient block for the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficient may be a scalar. The video encoder 20 may apply one or more transforms to the Cb transform block of the TU to generate a Cb coefficient block for the TU. The video encoder 20 may apply one or more transforms to the Cr transform block of the TU to generate a Cr coefficient block for the TU.

[0065] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), the video encoder 20 may quantize the coefficient block. Quantization generally refers to a process in which transform coefficients are quantized to possibly reduce the amount of data used to represent the transform coefficients, providing further compression. After the video encoder 20 quantizes the coefficient block, the video encoder 20 may entropy encode the syntax elements indicating the quantized transform coefficients. For example, the video encoder 20 may perform context-adaptive binary arithmetic coding (CABAC) on the syntax elements indicating the quantized transform coefficients. Finally, the video encoder 20 may output a bitstream including a sequence of bits that form a representation of a coded frame and associated data, and the bitstream is stored in a storage device 32 or transmitted to a target device 14.

[0066] After receiving the bitstream generated by the video encoder 20, the video decoder 30 can parse the bitstream to obtain syntax elements from the bitstream. The video decoder 30 can reconstruct a frame of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing video data is generally the inverse of the encoding process performed by the video encoder 20. For example, the video decoder 30 can perform an inverse transform on a coefficient block associated with a TU of the current CU to reconstruct a residual block associated with the TU of the current CU. The video decoder 30 also reconstructs the codec block of the current CU by adding samples of the prediction block for the PU of the current CU to corresponding samples of the transform block of the TU of the current CU. After reconstructing the codec block of each CU of the frame, the video decoder 30 can reconstruct the frame.

[0067] As digital video quality moves from high definition to 4Kx2K or even 8Kx4K, the amount of video data to be encoded / decoded grows exponentially. It is an eternal question how to encode / decode video data more efficiently while maintaining the image quality of the decoded video data. Different approaches have been adopted by various video codec standards to solve this problem. For example, it has been found that translational motion models perform poorly for motion compensated prediction in the presence of many kinds of motion in the real world (e.g., zoom-in / zoom-out, rotation, perspective motion, and other irregular motions). Therefore, a variety of block-based affine motion models have been proposed to improve the accuracy of motion compensated prediction.

[0068] Figure 4F An example of a multi-type tree partition structure is shown, which divides leaf CUs (e.g., Figure 4C The leaf-CU 440 in the example is split into one or more PUs. Each leaf-CU can be further split into one, two or four PUs according to the specified PU splitting type. The same motion information (e.g., MV and reference picture index) is shared by all samples of a PU, that is, only a single motion compensation process is required to generate the reference samples of the PU. After obtaining the residual signal of a CU by subtracting the prediction signal of the CU from the corresponding original signal, the CU can be split into multiple TUs according to another quadtree structure similar to the quadtree structure applied to split the CU.

[0069] In some embodiments (e.g., in VVC), a multi-type tree partitioning structure based on quadtree partitioning, binary partitioning, and ternary partitioning is applied to replace the concept of multiple partitioning types, that is, it eliminates the separation of the concepts of CU, PU, ​​and TU. In such a partitioning structure, a CU may have a square or rectangular shape. Specifically, a CTU is first partitioned based on a quadtree structure (e.g., referring to Figure 4C and Figure 4Dand related descriptions), each quad leaf node can then be further split by a multi-type tree split structure (e.g., Figure 4C 440 of the leaf-CU). In some embodiments, there are four types of splits in the multi-type tree structure: vertical binary split (SPLIT_BT_VER), horizontal binary split (SPLIT_BT_HOR), vertical ternary split (SPLIT_TT_VER), and horizontal ternary split (SPLIT_TT_HOR). For example, leaf-CU 450 (of size 32x32) is split according to horizontal binary split, which produces two regions, each of which has a size of 16x32. Leaf-CU 460 (of size 32x32) is split according to vertical binary split, which produces two regions, each of which has a size of 32x16. Leaf-CU 470 is split according to vertical ternary split and produces three regions of size 32x8, 32x16, and 32x8, respectively. Note that the middle region is further split according to horizontal binary split, and the right region is further split according to horizontal ternary split. The leaf-CU 480 (of size 32x32) is partitioned according to a horizontal ternary split and produces three regions with sizes of 8x32, 16x32, and 8x32.

[0070] In some embodiments, the leaf nodes of each multi-type tree are treated as a CU and used as the basic unit for both the prediction process and the transform process without further partitioning, except (1) when several sub-block coding modes are used to apply transforms on a sub-block basis, and (2) when the width or height of a CU is larger than the maximum allowed transform size (e.g., 64 samples). In the latter case, the CU will be split into multiple TUs in the horizontal direction, vertical direction, or both to ensure that the applied transform does not exceed the maximum supported transform size. As can be seen above, in most cases, CUs, PUs, and TUs share the same block size in the partitioning structure.

[0071] Figure 5A-Figure 5BThe figure illustrates different sub-block transform (SBT) partitions according to some implementations of the present disclosure. SBT mode is supported for inter-predicted CUs (also referred to as "inter CUs") to enhance the efficiency of transform coding. For each inter CU having at least one non-zero transform coefficient (e.g., indicated by the syntax element cu_root_cbf set to 1), a CU level flag cu_sbt_flag is signaled to indicate whether the residual of the entire CU is coded or only a portion of the CU is coded. If the residual of the entire CU is coded, the CU is viewed by the encoder as a single TU, that is, the residual of the entire CU is transformed, quantized, and coded into the bitstream based on a single transform (e.g., the size of the transform block is equal to the size of the CU). If only the residuals of a part of a CU (e.g., half, a quarter, or a third of the CU) are encoded and decoded, then the residuals of the entire CU are implicitly split into two groups: the first group of residuals are encoded and decoded using some pre-assigned transform (e.g., DCT-7 or DCT-8) and transmitted from the encoder to the decoder, while the second group of residuals are set to zero values ​​without signaling.

[0072] In some embodiments, the chroma residuals (Cr residual and Cb residual) of the SBT CU are always coded using DCT-2, while the luma residuals of the SBT CU are coded using a position-dependent transform method. Figure 5A As shown, depending on the different SBT partitioning patterns, two SBT positions are assigned DCT-8 or DST-7 transforms. For example, the horizontal transform and vertical transform for SBT-Hal-V and SBT-Quart-V position 0 are DCT-8 and DST-7, respectively. In order to avoid introducing new transform sizes, when one side of the residual TU is greater than 32, the corresponding transform is set to DCT-2. Therefore, the SBT mode jointly specifies the TU partitioning, whether each TU contains non-zero residual, and the horizontal transform and vertical transform applied, which can be considered as a syntactic shortcut for the case where the main residual of a CU is located on one side of the CU.

[0073] In conventional intra mode, reconstructed samples adjacent to the CU are used to generate intra prediction samples for the CU. Based on such an approach, the spatial correlation between the predicted samples and the reference samples is roughly proportional to the distance between the predicted samples and the reference samples. Therefore, samples in the internal part (especially samples located in the lower right corner of the block) typically have worse prediction quality than samples close to the top and / or left block boundaries. In order to improve intra prediction efficiency, short-distance intra prediction (SDIP) was proposed and fully studied during the development cycle of the HEVC standard. This method divides the intra-frame codec block horizontally or vertically into multiple sub-blocks for prediction. An extreme case of such sub-block-based intra prediction is the so-called row-based prediction, in which blocks are divided into 1-D rows / columns for prediction.

[0074] In some embodiments, an intra-frame coding tool called intra sub-partitioning (ISP) mode (designed in the same style of SDIP) is used to improve intra-frame coding efficiency. ISP divides the luminance samples of an intra CU into 2 or 4 sub-partitions vertically or horizontally according to the CU size. For example, the minimum CU size for ISP is 4x8 (or 8x4). If the CU size is equal to 4x8 (or 8x4), then the CU is divided into 2 sub-partitions in ISP mode. If the CU size is greater than 4x8 (or 8x4), then the CU is divided into 4 sub-partitions in ISP mode. According to the current ISP design, the sub-partition must contain at least 16 samples. In addition, in the existing ISP design, when the ISP mode is enabled for a CU, only the luminance residual of the CU is divided into multiple TUs for transform coding, while the residuals of the two chrominance components (i.e., Cb and Cr) are coded and decoded as a whole without further TU partitioning.

[0075] In the VVC standard, a syntax element called Coded Block Flag (CBF) is used to signal the importance of the residual for each CU, that is, it signals whether the CU contains non-zero residuals. According to the current design in VVC, there are separate CBFs for one luminance component and two chrominance components for a given CU. If the CBF of the corresponding component is equal to zero, no transform coefficients are signaled for the component and all transform coefficients are set to zero; otherwise, the transform coefficients are encoded and decoded into the bitstream. In addition, as mentioned earlier, there are three different scenarios in VVC in which a CU is split into multiple TUs and the residuals of each TU are encoded and decoded separately. These three scenarios include SBT TU partitioning, ISP TU partitioning, and TU partitioning caused by the maximum transform size limit when the CU width and / or height is greater than the maximum allowed transform size. According to the current VVC draft specification, in all three scenarios, the same signaling method is applied to indicate whether there are non-zero transform coefficients for the luminance component, that is, a tu_luma_cbf is sent separately for each TU. However, as described below, in these three scenarios, different methods are applied to signal the CBF of the chrominance components:

[0076] TU partitioning based on maximum transform size: Two additional CBF flags tu_cb_cbf and tu_cr_cbf are signaled for each TU to indicate whether there are non-zero coefficients in the corresponding chroma components of the TU. If a chroma CBF flag is equal to 0, it indicates that all residuals of the chroma component are zero and no residual information needs to be transmitted. Otherwise (i.e., CBF flag is equal to 1), the residual of the corresponding chroma component is transmitted.

[0077] ISP TU partitioning: As mentioned earlier, in ISP mode, TU partitioning is only applied to encode and decode luma residual, while residuals of chroma components are encoded and decoded without TU partitioning. Therefore, for the entire CU, two chroma CBF flags, namely tu_cb_cbf and tu_cr_cbf, are signaled to indicate whether one chroma component has non-zero transform coefficients. Note that since ISP TU partitioning is not applied to chroma components, the CU level is equal to the TU level for chroma CBF signaling in ISP mode.

[0078] SBT TU segmentation (such as Figure 5A(as shown): Unlike ISP mode, TU partitioning of SBT mode is applied to both luma and chroma components. In addition, the residuals of one of the TU partitions (both luma and chroma components) are forced to be zero. Based on the existing SBT design, two CBFs are signaled separately for the two chroma components at the CU level. When the CBF of a chroma component is equal to 0, all residuals of the chroma component are set to zero without signaling. Otherwise (i.e., CBF is equal to 1), only non-zero SBT partitions (e.g., Figure 5A The residuals of the left split of SBT-Hal-V position 0 and SBT-Quart-V position 0, and the right split of SBT-Hal-V position 1 and SBT-Quart-V position 1) are transmitted in the bitstream. In the current VVC reference software (VTM), the chroma residuals of other SBT splits are always set to zero by sending the corresponding CBF as zero.

[0079] Based on the above comparison, the following problems are found for the chrominance CBF signaling in SBT mode:

[0080] 1. In different scenarios, different signaling methods are used for TU partitioning schemes to signal the chroma CBF. In particular, for both TU partitioning based on maximum transform size and ISP TU partitioning, the chroma CBF is signaled at the TU level to indicate whether a TU contains a non-zero residual for a corresponding chroma component. However, for SBT TU partitioning, two chroma CBFs are signaled at the CU level. Such a non-uniform design may not be optimal from a standardization point of view.

[0081] 2. As pointed out earlier, in the current VTM, for zero SBT splits ( Figure 5A The right split of SBT-Hal-V position 0 and SBT-Quart-V position 0, and the left split of SBT-Hal-V position 1 and SBT-Quart-V position 1) always signal two chroma CBFs. However, given that all chroma residuals for a zero SBT split are by definition zero, i.e., the corresponding chroma CBF is always equal to 0, these two flags are redundant.

[0082] 3. Different context models are used to encode and decode the chroma CBF for the three TU partitioning modes. In particular, for TU partitioning based on maximum transform size and ISP TU partitioning, the context model used to encode and decode the chroma CBF (i.e., tu_cb_cbf and tu_cr_cbf) is determined by the TU depth (i.e., the corresponding TU partitioning level of the CU). This is different from the SBT mode, in which a fixed context model is used regardless of the depth of the TU partition.

[0083] In some embodiments, for SBT mode, the chroma CBF signaling methods of different TU partitioning schemes are unified by moving the chroma CBF signaling from the CU level to the TU level. Table 1 illustrates an example pseudo code of the corresponding transform unit syntax table when the proposed method is applied, where the SBT related syntax is bolded. Figure 6 and related description of the coordination details of chroma CBF signaling for different TU partitioning schemes.

[0084]

[0085] Table 1 is the modified codec unit syntax table after applying solution 1.

[0086] As shown in Table 1, with the proposed method, the luma CBF tu_cbf_luma and two chroma CBFs tu_cbf_cb and tu_cbf_cr are signaled separately for each SBT TU. In addition, the presence of the luma CBF and chroma CBF of one SBT TU is conditioned on the position information of the SBT partition. More specifically, for the corresponding zero SBT partition ( Figure 5A The signaling of both luma CBF and chroma CBF is omitted for the right partition of SBT-Hal-V position 0 and SBT-Quart-V position 0, and the left partition of SBT-Hal-V position 1 and SBT-Quart-V position 1). On the other hand, based on the existing SBT design in VVC, SBT mode can be enabled only for inter CUs whose root CBF (i.e., cu_root_cbf) is equal to 1. In other words, SBT mode can only be enabled when there is at least one non-zero transform coefficient (luminance or chroma) for the entire CU. In addition, as shown in Table 1, for a non-zero SBT TU partition, the chroma CBF is signaled before the luma CBF, and the luma CBF is signaled only when at least one of the two chroma CBFs is 1.

[0087] In some embodiments, such as by Figure 5B As shown, ternary tree (TT) splitting for sub-block partitioning of SBT mode is enabled. In particular, in addition to the current binary tree (BT) and adaptive binary tree (ABT) splitting types, two additional SBT partitioning types, namely SBT-TT-V and SBT-TT-H, are introduced. In SBT-TT-V / SBT-TT-H mode, the current CU is vertically / horizontally split into three TUs with a 1:2:1 partition ratio, and only the middle TU partition contains non-zero residuals, while the residuals of the two boundary TU partitions are always set to zero without signaling.

[0088] In some embodiments, when TT splitting is used for sub-block partitioning in SBT mode, in order to better compress local residual energy in frequency domain, a position-dependent transform selection method is applied to adaptively select core transforms of different dimensions for non-zero TT partitioning. Figure 5B The figure in the middle shows an example, in which the horizontal transform and vertical transform of the middle TU of SBT-TT-V are DCT-2 and DST-7, respectively. In another example, in order to avoid introducing different DST-7 transform sizes, the DST-7 transform is enabled only when the non-split CU side (i.e., the side without TU splitting) is no larger than 32 samples; otherwise, the corresponding transform along that direction of the TU will be replaced by DCT-2. In a third example, other core transforms (in addition to DCT-2 and DST-7) are applied in the proposed TT-based SBT partitioning scheme to adapt to the varying characteristics of the residuals in different blocks.

[0089] In some embodiments, when the proposed TT partitioning pattern is used, other different SBT partitioning patterns can be enabled in whole and / or in part in SBT mode. In one example, it is proposed to use the TT partitioning pattern instead of the ABT partitioning pattern to split the TU of the SBT mode. In particular, by this method, there are a total of 6 SBT partitions, namely SBT-BT-H position 0, SBT-BT-H position 1, SBT-BT-V position 0, SBT-BT-V position 1, SBT-TT-H and SBT-TT-V. In another example, it is proposed to enable all BT partitioning patterns, ABT partitioning patterns and TT partitioning patterns for SBT mode, namely SBT-BT-H position 0, SBT-BT-H position 1, SBT-BT-V position 0, SBT-BT-V position 1, SBT-Quart-H position 0, SBT-Quart-H position 1, SBT-Quart-V position 0, SBT-Quart-V position 1, SBT-TT-H and SBT-TT-V are all included.

[0090] Figure 6 600 is a flowchart illustrating a process 600 for decoding a sub-block transform (SBT) codec unit according to some implementations of the present disclosure. For convenience, the process 600 is described as being performed by a decoder (such as Figure 3 The video decoder 30) is executed.

[0091] As mentioned above Figure 5A-Figure 5BAs described, the SBT codec unit is divided into multiple transform units (e.g., SBT-Hal-V position 0, SBT-Hal-V position 1, etc.). Among the multiple transform units, only one includes a non-zero coefficient. For example, in the SBT-Hal-V position 0 partitioning scheme, only the left transform unit includes a non-zero coefficient, and the right transform unit includes all zero coefficients.

[0092] As a first step in decoding an SBT transform unit, the decoder receives a first syntax element associated with the SBT codec unit from a bitstream (e.g., a cu_root_cbf flag as described in Table 1), wherein the first syntax element indicates that there is at least one non-zero transform coefficient in the SBT codec unit (e.g., the first syntax element is "1") (610).

[0093] Next, in response to determining that the first syntax element indicates that there is at least one non-zero transform coefficient in the SBT codec unit (e.g., the first syntax element is 1), the decoder determines a first transform unit in the plurality of transform units that includes the non-zero transform coefficient. In some embodiments, the first transform unit is the only transform unit in the plurality of transform units that includes the non-zero transform coefficient. For example, in Figure 5A In the SBT-Hal-V position 0 partitioning scheme described in , the left transform unit is a transform unit that may include non-zero transform coefficients. Figure 5B In the SBT-TT-V partitioning scheme described in , an intermediate transform unit is a transform unit that may include non-zero transform coefficients. In some embodiments, determining the transform unit including non-zero coefficients depends on position information of a specific partitioning scheme, such as indicated by the syntax element "cu_sbt_pos_flag" in Table 1 (620).

[0094] The decoder then receives from the bitstream a second syntax element (e.g., tu_cbf_cb in Table 1) and a third syntax element (e.g., tu_cbf_cr in Table 1), the second syntax element being associated with a first chroma component (e.g., cb) of a first transform unit to indicate whether all transform coefficients of the first chroma component are zero, and the third syntax element being associated with a second chroma component (e.g., cr) of the first transform unit to indicate whether all transform coefficients of the second chroma component are zero (630).

[0095] The decoder then receives a fourth syntax element (eg, tu_cbf_luma in Table 1) associated with the luma component of the first transform unit from the bitstream, which indicates whether all transform coefficients of the luma component are zero (640).

[0096] Based on the second, third, and fourth syntax elements, the decoder infers whether the chroma components and the luma components of the first transform unit include all-zero transform coefficients, and decodes the transform coefficients of the chroma components and the luma components (650).

[0097] Finally, since other transform units (e.g., the transform unit to the right of SBT-Hal-V position 0) include only zero coefficients, the decoder sets the luma and chroma components of other transform units in the SBT codec unit to zero (660).

[0098] In some embodiments, the decoder receives the fourth syntax element from the bitstream only when at least one of the second and third syntax elements indicates that the corresponding chroma component includes non-zero coefficients (e.g., the second and / or third syntax elements are equal to 1). On the other hand, when the second and third syntax elements indicate that the chroma component includes only zero coefficients (e.g., the second and third syntax elements are equal to 0), the decoder sets the fourth syntax element to 1.

[0099] In some embodiments, the SBT codec unit is partitioned into a ternary tree partitioning scheme (e.g., Figure 5B As shown), the coding unit is divided into three transform units, and the transform coefficients of two of the three transform units are set to zero.

[0100] In some embodiments, the three transform units are adjacent to each other in a vertical direction and include a top transform unit, a center transform unit, and a bottom transform unit, and wherein the top transform unit and the bottom transform unit have all-zero coefficients.

[0101] In some embodiments, a DST-VII transform is applied in a horizontal direction of a center transform unit, and a DCT-II transform is applied in a vertical direction of the center transform unit.

[0102] In some embodiments, the three transform units are adjacent to each other in a horizontal direction and include a leftmost transform unit, a center transform unit, and a rightmost transform unit, and the leftmost transform unit and the rightmost transform unit have all-zero transform coefficients.

[0103] In some embodiments, a DST-VII transform is applied in a vertical direction of a center transform unit, and a DCT-II transform is applied in a horizontal direction of the center transform unit.

[0104] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium or a communication medium, wherein the computer-readable storage medium corresponds to a tangible medium such as a data storage medium, and the communication medium includes, for example, any medium that facilitates the transfer of a computer program from one place to another in accordance with a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-temporary tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the implementation described in the present application. A computer program product may include a computer-readable medium.

[0105] The terms used in the description of the implementation herein are only for the purpose of describing the specific implementation, and are not intended to limit the scope of the claims. As used in the description of the implementation and the appended claims, the singular forms "a", "an" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will be further understood that when used in this specification, the terms "include" and / or "comprising" clearly stipulate the presence of stated features, elements and / or components, but do not exclude the presence or addition of one or more other features, elements, components and / or groups thereof.

[0106] It will also be understood that although the terms first, second, etc. may be used herein to describe various different elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first electrode may be referred to as a second electrode, and similarly, a second electrode may be referred to as a first electrode without departing from the scope of the implementation. Both the first electrode and the second electrode are electrodes, but they are not the same electrode.

[0107] The description of the present application has been given for the purpose of illustration and description, and is not intended to be exhaustive or limited to the invention in the disclosed form. Many modifications, variations and alternative implementations will be apparent to those of ordinary skill in the art who benefit from the teachings given in the foregoing description and the associated drawings. The embodiments are selected and described to best explain the principles of the present invention, practical applications, and to enable those skilled in the art to understand the present invention with respect to various different implementations, and to best utilize the underlying principles and various implementations with various modifications suitable for the specific purposes envisioned. Therefore, it will be understood that the scope of the claims will not be limited to the specific examples of the disclosed implementations, and modifications and other implementations are intended to be included within the scope of the appended claims.

Claims

1. A video encoding method, comprising: Determining a first syntax element associated with a current codec unit, wherein the first syntax element indicates whether there is at least one non-zero transform coefficient in the current codec unit; In response to determining the first syntax element indicating that at least one non-zero transform coefficient is present in the current codec unit, determining a second syntax element indicating whether the current codec unit is encoded in a sub-block transform mode; In response to determining the second syntax element indicating that the current codec unit is encoded in the sub-block transform mode, determining, for a first transform unit among a plurality of transform units partitioned from the current codec unit, a third syntax element and a fourth syntax element, the third syntax element being associated with a first chroma component of the first transform unit to indicate whether all transform coefficients of the first chroma component are zero, the fourth syntax element being associated with a second chroma component of the first transform unit to indicate whether all transform coefficients of the second chroma component are zero, and when the third syntax element indicates that one or more transform coefficients of the first chroma component are not zero and / or the fourth syntax element indicates that one or more transform coefficients of the second chroma component are not zero, determining a fifth syntax element, the fifth syntax element being associated with a luma component of the first transform unit to indicate whether all transform coefficients of the luma component are zero, wherein the first transform unit includes non-zero transform coefficients, and when the third syntax element indicates that all transform coefficients of the first chroma component are zero and the fourth syntax element indicates that all transform coefficients of the second chroma component are zero, the fifth syntax element is set to 1 to indicate that one or more transform coefficients of the luma component are not zero; as well as At least one of the first syntax element, the second syntax element, the third syntax element, the fourth syntax element, and the fifth syntax element is encoded into a video bitstream. 2 . The method of claim 1 , wherein the first transform unit is the only one of the plurality of transform units that includes a non-zero transform coefficient.

3. The method of claim 1 , wherein encoding at least one of the first syntax element, the second syntax element, the third syntax element, the fourth syntax element, and the fifth syntax element into a video bitstream comprises: The fifth syntax element is encoded into the video bitstream when the third syntax element indicates that one or more transform coefficients of the first chroma component are non-zero and / or the fourth syntax element indicates that one or more transform coefficients of the second chroma component are non-zero.

4. The method according to claim 1, wherein the current codec unit is partitioned into a ternary tree partitioning scheme, the ternary tree partitioning scheme divides the current codec unit into three transform units, and wherein transform coefficients of two of the three transform units are set to zero. 5 . The method of claim 4 , wherein the three transform units are adjacent to each other in a vertical direction and include a top transform unit, a center transform unit, and a bottom transform unit, and wherein the top transform unit and the bottom transform unit have all-zero coefficients. 6 . The method of claim 5 , wherein a DST-VII transform is applied in a horizontal direction of the central transform unit, and a DCT-II transform is applied in a vertical direction of the central transform unit. 7 . The method according to claim 4 , wherein the three transform units are adjacent to each other in a horizontal direction and include a leftmost transform unit, a center transform unit, and a rightmost transform unit, and the leftmost transform unit and the rightmost transform unit have all-zero transform coefficients. 8 . The method of claim 7 , wherein a DST-VII transform is applied in a vertical direction of the central transform unit, and a DCT-II transform is applied in a horizontal direction of the central transform unit.

9. A computing device comprising: one or more processors; a memory coupled to the one or more processors; as well as A plurality of programs stored in the memory, which, when executed by the one or more processors, cause the computing device to perform the method of any one of claims 1-8.

10. A non-transitory computer-readable storage medium storing a plurality of programs for execution by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, causes the computing device to perform the steps of the method of any one of claims 1 to 8 to generate a video bitstream and send the video bitstream.

11. A computer program product comprising instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.

Images