Method, apparatus and system for encoding and decoding blocks of video samples

Abstract

To provide a video encoding and decoding method that improves the coding efficiency of information related to block size constraints.SOLUTION: The video decoding method includes decoding a maximum conversion block size constraint and / or a maximum coding tree unit (CTU) size constraint from a bitstream 801 and decoding a maximum effective conversion block size and / or a maximum effective CTU size from the bitstream. The decoded maximum effective conversion block size is less than or equal to the decoded maximum conversion block size constraint. The decoded maximum effective CTU size is less than or equal to the decoded maximum CTU size constraint.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] Reference to related applications This application claims the benefit under 35 U.S.C. § 119 of the filing date of Australian Patent Application No. 2020203330, filed May 21, 2020, the entirety of which is incorporated herein by reference as if fully set forth herein.

[0002] The present invention relates generally to digital video signal processing, and more particularly to a method, apparatus and system for encoding and decoding blocks of video samples. The present invention also relates to a computer program product including a computer-readable medium having recorded thereon a computer program for encoding and decoding blocks of video samples. [Background technology]

[0003] Many applications for video coding currently exist, including applications for the transmission and storage of video data. Many video coding standards have been developed, and others are currently under development. Recent developments in video coding standardization have led to the formation of a group called the "Joint Video Experts Team" (JVET). The Joint Video Experts Team (JVET) includes members from two Standards Setting Organizations (SSOs): Study Group 16, Question 6 (SG16 / Q6) of the International Telecommunication Union's (ITU) Telecommunication Standardization Sector (ITU-T), also known as the "Video Coding Experts Group" (VCEG), and International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the "Moving Picture Experts Group" (MPEG).

[0004] The Joint Video Experts Team (JVET) analyzed the responses at its 10th meeting in San Diego, USA, and issued a Call for Proposals (CfP). The submitted responses demonstrated video compression capabilities that significantly exceeded those of the current state-of-the-art video compression standard, "high efficiency video coding" (HEVC). Based on this outperformance, it was decided to initiate a project to develop a new video compression standard called "versatile video coding" (VVC). VVC is expected to address the ongoing demand for ever-higher compression performance, particularly as video formats increase in capacity (e.g., at higher resolutions and higher frame rates) and address increasing market demand for service delivery over wide area networks (WANs), where bandwidth costs are relatively high. VVC must be implementable in modern silicon processes and offer an acceptable trade-off between achieved performance and implementation cost. Implementation cost may be considered, for example, in terms of one or more of silicon area, CPU processor load, memory utilization, and bandwidth. The versatility of the VVC standard lies in part in the wide selection of tools available for compressing video data and the wide range of applications for which VVC is suitable.

[0005] Video data includes a sequence of frames of image data, each of which contains one or more color channels. Generally, one primary color channel and two secondary color channels are required. The primary color channel is commonly referred to as the "luma" channel, and the secondary color channels are commonly referred to as the "chroma" channels. Video data is typically displayed in the RGB (red-green-blue) color space, which has a high degree of correlation between each of its three components. Video data representations viewed by encoders or decoders often use color spaces such as YCbCr. YCbCr concentrates luminance, mapped to "luma" according to a transfer function, in the Y (primary) channel and chroma in the Cb and Cr (secondary) channels. Due to the use of uncorrelated YCbCr signals, the statistics of the luma channel differ significantly from those of the chroma channels. The key difference is that, after quantization, the chroma channels contain relatively fewer significant coefficients for a given block compared to the coefficients of the corresponding luma channel block. Additionally, the Cb and Cr channels may be spatially sampled (subsampled) at a lower rate compared to the luma channel, e.g., half horizontally and half vertically, known as the "4:2:0 chroma format." The 4:2:0 chroma format is commonly used for Internet video streaming, broadcast television, Blu-Ray, and other applications. TMIt is commonly used in "consumer" applications, such as storage to disk. Subsampling the Cb and Cr channels at half the rate horizontally and not subsampling vertically is known as a "4:2:2 chroma format." The 4:2:2 chroma format is typically used in professional applications, including capturing video for filmmaking and other applications. The higher sampling rate of the 4:2:2 chroma format makes the resulting video more resilient to editing operations such as color grading. Before distribution to consumers, 4:2:2 chroma format material is often converted to a 4:2:0 chroma format and then encoded for distribution to consumers. In addition to the chroma format, video is also characterized by its resolution and frame rate. Example resolutions are ultra-high definition (UHD) with a resolution of 3840x2160, or "8K" with a resolution of 7680x4320, with example frame rates of 60 or 120 Hz. The luma sample rate may range from approximately 500 megasamples / second to several gigasamples / second. For a 4:2:0 chroma format, the sample rate of each chroma channel is one-quarter of the luma sample rate, and for a 4:2:2 chroma format, the sample rate of each chroma channel is one-half of the luma sample rate.

[0006] The VVC standard is a "block-based" codec, where a frame is first divided into a square array of regions known as "coding tree units" (CTUs). CTUs typically occupy a relatively large area, such as 128 x 128 luma samples. However, the CTUs at the right and bottom of each frame may be smaller in area. Associated with each CTU is a "coding tree" for both the luma and chroma channels (a "shared tree"), or separate trees for each of the luma and chroma channels. The coding tree defines the decomposition of the CTU's area into a set of blocks, also called "coding blocks" (CBs). When a shared tree is in use, a single coding tree specifies the blocks for both the luma and chroma channels. In that case, the collection of co-located coding blocks is called a "coding unit" (CU), i.e., each CU has a coding block for each color channel. The CBs are processed for encoding or decoding in a specific order. As a result of the use of the 4:2:0 chroma format, a CTU having a luma coding tree for a 128x128 luma sample area has a corresponding chroma coding tree for a 64x64 chroma sample area collocated with the 128x128 luma sample area. When a single coding tree is used for the luma and chroma channels, the collection of collocated blocks for a given area is generally referred to as a "unit," e.g., the CU mentioned above, as well as a "prediction unit" (PU) and a "transform unit" (TU). A single tree with CUs spanning the color channels of 4:2:0 chroma format video data results in chroma blocks that are half the width and height of the corresponding luma blocks. When separate coding trees are used for a given area, the CB mentioned above, as well as a "prediction block" (PB) and a "transform block" (TB) are used.

[0007] Despite the above distinction between "units" and "blocks", the term "block" may be used as a generic term for an area or region of a frame where an operation is applied to all color channels.

[0008] For each CU, a prediction unit (PU) is generated for the contents (sample values) of the corresponding area of ​​frame data ("prediction unit"). Furthermore, a representation of the difference (or "spatial domain" residual) between the prediction and the contents of the region seen at the input to the encoder is formed. The differences in each color channel can be transformed and coded as a sequence of residual coefficients, forming one or more TUs for a given CU. The applied transform can be a discrete cosine transform (DCT) or other transform applied to each block of residual values. This transform is applied separably; i.e., a two-dimensional transform is performed in two passes. The block is first transformed by applying a one-dimensional transform to each row of samples in the block. The partial results are then transformed by applying a one-dimensional transform to each column of the partial results, producing a final block of transform coefficients that substantially decorrelates the residual samples. Transforms of various sizes are supported by the VVC standard, including transforms of rectangular-shaped blocks, where each side dimension is a power of two. The transform coefficients are quantized for entropy coding into the bitstream.

[0009] VVC features intra-frame prediction and inter-frame prediction. Intra-frame prediction involves the use of previously processed samples within a frame to generate a prediction of the current block of samples within the frame. Inter-frame prediction involves using a block of samples obtained from a previously decoded frame to generate a prediction of the current block of samples in a frame. The block of samples obtained from the previously decoded frame is often offset from the spatial location of the current block according to a motion vector to which filtering has been applied. An intra-frame predicted block can be (i) a uniform sample value ("DC intra-prediction"), (ii) a plane with an offset and horizontal and vertical gradients ("plane intra-prediction"), (iii) a population of blocks with neighboring samples applied in a specific direction ("angular intra-prediction"), or (iv) the result of matrix multiplication using neighboring samples and selected matrix coefficients. Further discrepancies between the predicted block and the corresponding input samples can be corrected to some extent by encoding a "residual" into the bitstream. The residual is typically transformed from the spatial domain to the frequency domain to form residual coefficients (in a "primary transform domain"), which may be further transformed by application of a "secondary transform" (to produce residual coefficients in a "secondary transform domain"). The residual coefficients are quantized according to a quantization parameter, resulting in a loss of precision in the reconstruction of the samples produced at the decoder, but with a concomitant bitrate reduction in the bitstream.

[0010] The intra block copy (IBC) mode allows for generating a prediction of a block using a block of samples from the same frame. An alternative "matrix intra prediction" (MIP) mode is available, whereby the predicted block is generated using a matrix multiplication of a predetermined vector with block-neighboring samples. The block may be palette coded instead of using a transform. The three color channels of the block may be passed through an adaptive color transform (ACT), typically decorrelating the RGB color space to the YCbCr color space. The Cb and Cr channel residuals may be jointly coded (JCbCr). A selection of primary transforms (MTS) is available, including DCT-2, DCT-8, and DST-7, as well as a transform skip (TS) mode. An optional secondary transform is also available, whereby one kernel from a set of kernels is selected based on the intra prediction mode and block size and can be applied to the low-frequency region of the transform block, known as the low-frequency non-separable transform (LFNST). The residual may be quantized and coded with a trellis scheme known as "dependent quantization" (DQ). Inter-predicted blocks may be predicted as a set of sub-blocks, each with a different motion vector derived according to an affine motion model. Blocks may be generated using combined (uniform) blended intra and inter prediction operations. Blocks may also be generated using geometrically oriented blends of two different inter-predicted blocks (CIIPs). A luma mapping with chroma scaling (LMCS) process may be applied as part of the decoding process, mapping luma samples to specific values ​​and applying scaling operations to the values ​​of chroma samples. An adaptive loop filter (ALF) may be applied, whereby luma and chroma samples are spatially filtered using one of multiple filters sent in the bitstream, primarily for smoothing purposes.Inverse quantization of the residual coefficients for a transform block may be performed in a non-uniform manner according to a (spatial) scaling matrix (or "scaling list") that is applied in combination with the quantization parameters for the transform block. Summary of the Invention

[0011] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0012] One aspect of the present disclosure provides a method for decoding an image frame from a bitstream, the image frame being split into a plurality of coding tree units, each of the plurality of coding tree units being split into one or more transform blocks, the method including: decoding a maximum transform block size constraint from the bitstream; decoding a maximum valid transform block size from the bitstream; the decoded maximum valid transform block size being less than or equal to the decoded maximum transform block size constraint; determining the one or more transform blocks for each of the plurality of coding tree units according to the decoded maximum valid transform block size and a split flag decoded from the bitstream; and decoding each of the determined one or more transform blocks from the bitstream to decode the image frame.

[0013] According to another aspect, the maximum transform block size constraint uses a 1-bit fixed length codeword.

[0014] According to another aspect, the maximum valid transform block size constraint is decoded from a general_constraint_info syntax structure in the bitstream.

[0015] According to another aspect, the maximum transform block size constraint is decoded from one of a video parameter set and a sequence parameter set of the bitstream.

[0016] According to another aspect, the maximum valid transform block size uses a 1-bit fixed length codeword.

[0017] According to another aspect, the maximum valid transform block size is decoded from a seq_parameter_set_rbsp() syntax structure in the bitstream.

[0018] Another aspect of the present disclosure provides a method for decoding an image frame from a bitstream, the image frame being split into a plurality of coding tree units, each of the plurality of coding tree units being split into one or more coding units, the method including: decoding a maximum coding tree unit size constraint from the bitstream; decoding a maximum valid coding tree unit size from the bitstream; the decoded maximum valid coding tree unit size being less than or equal to the decoded maximum coding unit size constraint; determining the one or more coding units for each of the plurality of coding tree units according to the decoded maximum valid coding tree size and a split flag decoded from the bitstream; and decoding the image frame by decoding each of the determined one or more coding units from the bitstream.

[0019] According to another aspect, the maximum coding tree unit size constraint uses a fixed length codeword of 1 bit.

[0020] According to another aspect, the maximum valid coding tree unit size constraint is decoded from a general_constraint_info syntax structure in the bitstream.

[0021] According to another aspect, the maximum coding tree unit size constraint is decoded from one of a video parameter set and a sequence parameter set of the bitstream.

[0022] According to another aspect, the maximum possible coding tree unit size uses a fixed length codeword of 1 bit.

[0023] According to another aspect, the maximum valid coding tree unit size is decoded from a seq_parameter_set_rbsp() syntax structure in the bitstream.

[0024] Another aspect of the present disclosure provides a non-transitory computer-readable medium having stored thereon a computer program for performing a method for decoding an image frame from a bitstream, the image frame being split into a plurality of coding tree units, each of the plurality of coding tree units being split into one or more transform blocks, the method including: decoding a maximum transform block size constraint from the bitstream; decoding a maximum valid transform block size from the bitstream; the decoded maximum valid transform block size being less than or equal to the decoded maximum transform block size constraint; determining the one or more transform blocks for each of the plurality of coding tree units according to the decoded maximum valid transform block size and a split flag decoded from the bitstream; and decoding each of the determined one or more transform blocks from the bitstream to decode the image frame.

[0025] Another aspect of the present disclosure provides a video decoder configured to implement a method for decoding an image frame from a bitstream, the image frame being split into a plurality of coding tree units, each of the plurality of coding tree units being split into one or more coding units, the method including: decoding a maximum coding tree unit size constraint from the bitstream; decoding a maximum valid coding tree unit size from the bitstream; the decoded maximum valid coding tree unit size being less than or equal to the decoded maximum coding unit size constraint; determining the one or more coding units for each of the plurality of coding tree units according to the decoded maximum valid coding tree size and a split flag decoded from the bitstream; and decoding each of the determined one or more coding units from the bitstream to decode the image frame.

[0026] Another aspect of the present disclosure provides a system comprising a memory and a processor, wherein the processor is configured to execute code stored in the memory to implement a method for decoding an image frame from a bitstream, the image frame being split into a plurality of coding tree units, each of the plurality of coding tree units being split into one or more transform blocks, the method including: decoding a maximum transform block size constraint from the bitstream; decoding a maximum valid transform block size from the bitstream; the decoded maximum valid transform block size being less than or equal to the decoded maximum transform block size constraint; determining the one or more transform blocks for each of the plurality of coding tree units according to the decoded maximum valid transform block size and a split flag decoded from the bitstream; and decoding the image frame by decoding each of the determined one or more transform blocks from the bitstream.

[0027] Another aspect of the present disclosure provides a method for decoding an image frame from a bitstream, the image frame being divided into a plurality of coding tree units, each of the plurality of coding tree units being divided into one or more coding blocks, the method including: decoding from the bitstream a first flag indicating a constraint on the use of a scaling list, the flag being included in a general constraint information syntax; decoding from the bitstream a second flag indicating enabling of the scaling list, the second flag being constrained by the first flag, decoding the one or more coding blocks from the bitstream using scaling performed according to the value of the second flag; and decoding the image frame using the decoded one or more coding units.

[0028] Another aspect of the present disclosure provides a video decoding device for decoding an image frame from a bitstream, the image frame being split into a plurality of coding tree units, each of the plurality of coding tree units being split into one or more transform blocks, the video decoding device comprising: a first decoding unit that decodes a maximum transform block size constraint from the bitstream; a second decoding unit that decodes a maximum valid transform block size from the bitstream; a determination unit that determines the one or more transform blocks for each of the plurality of coding tree units according to the decoded maximum valid transform block size and a split flag decoded from the bitstream, wherein the decoded maximum valid transform block size is less than or equal to the decoded maximum transform block size constraint; and a third decoding unit that decodes each of the determined one or more transform blocks from the bitstream to decode the image frame.

[0029] Another aspect of the present disclosure provides a video decoding device for decoding an image frame from a bitstream, the image frame being split into a plurality of coding tree units, each of the plurality of coding tree units being split into one or more coding units, the video decoding device comprising: a first decoding unit that decodes a maximum coding tree unit size constraint from the bitstream; a second decoding unit that decodes a maximum valid coding tree unit size from the bitstream; a determination unit that determines the one or more coding units for each of the plurality of coding tree units according to the decoded maximum valid coding tree size and a split flag decoded from the bitstream, the decoded maximum valid coding tree unit size being less than or equal to the decoded maximum coding unit size constraint; and a third decoding unit that decodes the image frame using the decoded one or more coding units.

[0030] Another aspect of the present disclosure provides a video decoding device for decoding an image frame from a bitstream, the image frame being divided into a plurality of coding tree units, each of the plurality of coding tree units being divided into one or more coding blocks, the video decoding device comprising: a first decoding unit that decodes from the bitstream a first flag indicating a constraint on the use of a scaling list, the flag being included in a general constraint information syntax; a second decoding unit that decodes from the bitstream a second flag indicating the enabling of the scaling list, the second flag being constrained by the first flag, a third decoding unit that decodes the one or more coding blocks from the bitstream using scaling that is performed according to the value of the second flag; and a fourth decoding unit that decodes the image frame using the decoded one or more coding units.

[0031] Another aspect of the present disclosure provides a non-transitory computer-readable medium having stored thereon a computer program for performing a method for decoding an image frame from a bitstream, the image frame being split into a plurality of coding tree units, each of the plurality of coding tree units being split into one or more coding units, the method including: decoding a maximum coding tree unit size constraint from the bitstream; decoding a maximum valid coding tree unit size from the bitstream; the decoded maximum valid coding tree unit size being less than or equal to the decoded maximum coding unit size constraint; determining the one or more coding units for each of the plurality of coding tree units according to the decoded maximum valid coding tree size and a split flag decoded from the bitstream; and decoding the image frame by decoding each of the determined one or more coding units from the bitstream.

[0032] Another aspect of the present disclosure provides a non-transitory computer-readable medium having stored thereon a computer program for executing a method for decoding an image frame from a bitstream, the image frame being divided into a plurality of coding tree units, each of the plurality of coding tree units being divided into one or more coding blocks, the method including: decoding from the bitstream a first flag indicating a constraint on use of a scaling list, the flag being included in a general constraint information syntax; decoding from the bitstream a second flag indicating enabling of the scaling list, the second flag being constrained by the first flag, decoding the one or more coding blocks from the bitstream using scaling performed according to the value of the second flag; and decoding the image frame using the decoded one or more coding units to decode the image frame.

[0033] Other aspects are also disclosed. [Brief explanation of the drawings]

[0034] At least one embodiment of the present invention will now be described with reference to the following drawings and appendices. [Figure 1] FIG. 1 is a schematic block diagram illustrating a video encoding and decoding system. [Figure 2A] , [Figure 2B] 2A and 2B form a schematic block diagram of a general-purpose computer system upon which one or both of the video encoding and decoding systems of FIG. 1 may be implemented. [Figure 3] FIG. 3 is a schematic block diagram illustrating the functional modules of a video encoder. [Figure 4] FIG. 4 is a schematic block diagram showing the functional modules of a video decoder. [Figure 5] FIG. 5 is a schematic block diagram illustrating possible divisions of a block into one or more blocks in a generic video coding tree structure. [Figure 6] FIG. 6 is a schematic diagram of a data flow for achieving permitted division of a block into one or more blocks in a tree structure of generic video coding. [Figure 7A] , [Figure 7B] 7A and 7B show an exemplary division of a coding tree unit (CTU) into several coding units (CUs). [Figure 8] FIG. 8 shows a syntax structure for a bitstream having multiple slices, each containing multiple coding units. [Figure 9] FIG. 9 shows the syntax structure of a bitstream having coding blocks of a coding tree unit. [Figure 10] FIG. 10 illustrates a method for encoding a frame into a bitstream that includes one or more slices as a sequence of coding units. [Figure 11] 11 illustrates a method for decoding a frame from a bitstream as a sequence of coding units arranged in slices. Appendix A provides an example of working draft text for the VVC standard that has been adapted to correspond to the methods disclosed herein. DETAILED DESCRIPTION OF THE INVENTION

[0035] When referring to steps and / or features having the same reference number in one or more of the accompanying drawings, those steps and / or features have the same function or operation for the purposes of this specification, unless a contrary intention appears.

[0036] VVC encoders and decoders include a capability signaling mechanism known as "constraints." Initially in a bitstream, there is a set of constraints that indicate which VVC standard capabilities will not be used in the bitstream. The constraints are signaled along with the bitstream's "profile" and "level." A profile broadly indicates the set of tools that must be available to decode the bitstream. Constraints also provide finer granularity of control over which tools are further constrained in a given profile. Further constraints are called "sub-profiling." Sub-profiling allows specific tools to be effectively removed from a profile even after it has been defined, and implementers agree on a common sub-profile to be deployed. For example, if a given tool proves problematic to implement despite SSO analysis efforts, the tool can be removed later. One use of sub-profiling is to reduce the "proliferation" of profiles, some of which may never be used. Defining fewer profiles helps avoid market fragmentation of VVC standard implementations. Profiles and sub-profiles also define what are called "interoperability points." An interoperability point is a set of tools that manufacturers agree to be supported by their implementations. Profiles are agreed upon when the standard is finalized, and subprofiles can be agreed upon at a later date, allowing unforeseen implementation complexities or other issues of a particular tool to be addressed without having to add additional profiles to the standard. A particular encoder may choose to further restrict the use of coding tools, for example, for the purposes of complexity reduction, without deviating from the agreed profile and subprofile definitions of the resulting bitstream. Any further restrictions on tool selection may be signaled early in the bitstream, thereby suppressing the signaling of unused tools.Alternatively, the encoder can leave the tool available but never signal its use (leaving the signaling to control the tool, but always choosing a disabled value).

[0037] 1 is a schematic block diagram illustrating the functional modules of a video encoding and decoding system 100. System 100 includes constraints on block structure in addition to constraints on specific coding tools, providing additional flexibility for sub-profiling.

[0038] System 100 includes source device 110 and destination device 130. Communication channel 120 is used to communicate encoded video information from source device 110 to destination device 130. In some configurations, source device 110 and destination device 130 may comprise either or both of a respective mobile phone handset or “smartphone,” in which case communication channel 120 is a wireless channel. In other configurations, source device 110 and destination device 130 may comprise video conferencing equipment, in which case communication channel 120 is typically a wired channel such as an Internet connection. Furthermore, source device 110 and destination device 130 may comprise any of a wide range of devices, including devices supporting over-the-air television broadcasts, cable television applications, Internet video applications (including streaming), and applications in which encoded video data is captured on some computer-readable storage medium, such as a hard disk drive in a file server.

[0039] 1, source device 110 includes video source 112, video encoder 114, and transmitter 116. Video source 112 typically comprises a source of captured video frame data (shown as 113), such as an imaging sensor, a previously captured video sequence stored on a non-transitory recording medium, or video sourced from a remote imaging sensor. Video source 112 may also be the output of a computer graphics card, e.g., displaying the video output of an operating system and various applications running on a computing device such as a tablet computer. Examples of source device 110 that may include an imaging sensor as video source 112 include smartphones, video cameras, professional video cameras, and network video cameras.

[0040] Video encoder 114 converts (or “encodes”) captured frame data (indicated by arrow 113) from video source 112 into a bitstream (indicated by arrow 115), as further described with reference to FIG. 3 . Video encoder 114 encodes captured frame data 113 using a specific set of VVC coding tools (or “profile”). An indication of which profile was used is coded into bitstream 115 using a “profile_tier_level” syntax structure embedded in either the “video parameter set” (VPS) or “sequence parameter set” (SPS) of bitstream 115. The SPS syntax structure is also referred to as “seq_parameter_set_rbsp(),” and the VPS syntax structure is also referred to as “video_parameter_set_rbsp().” Additionally, further constraints on the set of coding tools used may also be coded into bitstream 115 using a “general_constraint_info” syntax structure, which is part of the aforementioned profile_tier_level syntax structure. The scaling list, the filters for ALF, and the parameters for LMCS are transmitted in one or more "Adaptation Parameter Sets" (APS), each containing parameters for one of these three tools, identified by the "aps_params_type" syntax element in the respective APS.

[0041] Bitstream 115 is transmitted by transmitter 116 as coded video data (or "coded video information") over communication channel 120. Bitstream 115 may, in some implementations, be stored in a non-transitory storage device 122, such as "flash" memory or a hard disk drive, until later transmitted over communication channel 120, or in lieu of transmission over communication channel 120. For example, coded video data may be provided to customers on demand over a wide area network (WAN) for video streaming applications.

[0042] Destination device 130 includes a receiver 132, a video decoder 134, and a display device 136. Receiver 132 receives encoded video data from communication channel 120 and passes the received video data as a bitstream to video decoder 134 (indicated by arrow 133). Video decoder 134 then outputs decoded frame data (indicated by arrow 135) as video to display device 136 for display. Decoded frame data 135 has the same chroma format as frame data 113. Examples of display device 136 include a cathode ray tube, a liquid crystal display such as in a smartphone, a tablet computer, a computer monitor, or a standalone television set. The functionality of each of source device 110 and destination device 130 can also be implemented in a single device, examples of which include a mobile phone handset and a tablet computer. The decoded frame data may be further transformed before being presented to a user. For example, a "viewport" having a particular latitude and longitude can be rendered from the decoded frame data using a projection format to represent a 360° view of the scene.

[0043] Notwithstanding the above exemplary devices, each of source device 110 and destination device 130 may be configured within a general-purpose computing system, typically via a combination of hardware and software components. Figure 2A illustrates such a computer system 200, including a computer module 201, input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 that may be configured as a video source 112, and a microphone 280, and output devices including a printer 215, a display device 214 that may be configured as a display device 136, and speakers 217. An external modulator-demodulator (modem) transceiver device 216 may be used by computer module 201 to communicate with a communications network 220 via connection 221. Communications network 220, which may represent communications channel 120, may be a wide area network (WAN), such as the Internet, a cellular telecommunications network, or a private wide area network (WAN). If connection 221 is a telephone line, modem 216 may be a conventional "dial-up" modem. Alternatively, modem 216 may be a broadband modem if connection 221 is a high-capacity (e.g., cable or optical) connection. A wireless modem may also be used for wireless connection to communication network 220. Transceiver device 216 may provide the functionality of transmitter 116 and receiver 132, and communication channel 120 may be embodied within connection 221.

[0044] The computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, the memory unit 206 can include semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). The computer module 201 also includes several input / output (I / O) interfaces, including an audio / video interface 207 that couples to a video display 214, speakers 217, and a microphone 280; an I / O interface 213 that couples to a keyboard 202, a mouse 203, a scanner 226, a camera 227, and optionally a joystick or other human interface device (not shown); and an interface 208 for an external modem 216 and a printer 215. The signal from the audio / video interface 207 to the computer monitor 214 is typically an output of a computer graphics card. In some implementations, the modem 216 may be incorporated into the computer module 201, for example, in the interface 208. The computer module 201 also has a local network interface 211, which allows the computer system 200 to be coupled to a local area communications network 222, known as a local area network (LAN), via a connection 223. As shown in Figure 2A, the local communications network 222 can also be coupled to a wide network 220 via a connection 224, which typically includes a so-called "firewall" device or device of similar functionality. The local network interface 211 can be an Ethernet TM Circuit card, Bluetooth TM This may include a wireless or IEEE 802.11 wireless configuration, although many other types of interfaces may be implemented for interface 211. Local network interface 211 may also provide the functionality of transmitter 116, and receiver 132 and communication channel 120 may also be embodied in local communications network 222.

[0045] The I / O interfaces 208 and 213 can provide either or both serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided and typically includes a hard disk drive (HDD) 210. Other storage devices, such as floppy disk drives and magnetic tape drives (not shown), may also be used. An optical disk drive 212 is typically provided to serve as a non-volatile source of data. Optical disks (e.g., CD-ROMs, DVDs, Blu-ray Discs) may be used. TM ), USB-RAM, portable, external hard drives, and portable memory devices such as floppy disks, for example, can be used as suitable sources of data for computer system 200. Typically, any of HDD 210, optical drive 212, and networks 220 and 222 may be configured to operate as video source 112 or as a destination for decoded video data to be stored for playback via display 214. Source device 110 and destination device 130 of system 100 may be embodied in computer system 200.

[0046] The components 205 to 213 of the computer module 201 typically communicate via an interconnected bus 204 and in a manner which results in a conventional mode of operation of the computer system 200 known to those skilled in the art. For example, the processor 205 is coupled to the system bus 204 using a connection 218. Similarly, the memory 206 and optical disk drive 212 are coupled to the system bus 204 by a connection 219. Examples of computers on which the above arrangements can be executed include IBM-PCs and compatibles, Sun SPARC stations, Apple Macs, TM or a similar computer system.

[0047] Where appropriate or necessary, video encoder 114 and video decoder 134, as well as the methods described below, may be implemented using computer system 200. Specifically, video encoder 114, video decoder 134, and the methods described may be implemented as one or more software application programs 233 executable within computer system 200. Specifically, video encoder 114, video decoder 134, and the steps of the methods described are performed by instructions 231 (see FIG. 2B ) in software 233 executing within computer system 200. The software instructions 231 may be formed as one or more code modules, each for performing one or more specific tasks. The software may also be divided into two separate portions, where a first portion and corresponding code module performs the methods described and a second portion and corresponding code module manages a user interface between the first portion and a user.

[0048] The software may be stored on a computer-readable medium, including, for example, the storage devices described below. The software is loaded from the computer-readable medium into computer system 200 and then executed by computer system 200. Such software or a computer-readable medium having a computer program recorded thereon is a computer program product. Use of the computer program product in computer system 200 preferably results in an advantageous apparatus for implementing video encoder 114, video decoder 134, and the methods described.

[0049] The software 233 is typically stored on the HDD 210 or in the memory 206. The software is loaded into the computer system 200 from a computer-readable medium and executed by the computer system 200. Thus, for example, the software 233 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 225 that is read by the optical disk drive 212.

[0050] In some cases, application program 233 may be encoded on one or more CD-ROMs 225 and supplied to the user and read via corresponding drive 212, or alternatively, may be read by the user from network 220 or 222. Additionally, software may also be loaded into computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory, tangible storage medium that provides recorded instructions and / or data to computer system 200 for execution and / or processing. Examples of such storage media are floppy disks, magnetic tape, CD-ROMs, DVDs, Blu-ray Discs, and the like. TM , a hard disk drive, a ROM or integrated circuit, a USB memory, a magneto-optical disk, or a computer-readable card such as a PCMCIA card, etc., whether such devices are internal or external to the computer module 201. Examples of transitory or non-tangible computer-readable transmission media that may also participate in providing software, application programs, instructions and / or video data or encoded video data to the computer module 401 include wireless or infrared transmission channels, as well as network connections to another computer or networked device, and the Internet or intranets, including email transmissions and information recorded on websites and the like.

[0051] A second portion of application program 233 and the corresponding code modules described above may be executed to implement one or more graphical user interfaces (GUIs) that are rendered or otherwise represented on display 214. Typically through manipulation of keyboard 202 and mouse 203, a user of application and computer system 200 can manipulate the interface in a functionally adaptable manner to provide control commands and / or input to the application associated with the GUI. Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface that utilizes speech prompts output via speaker 217 and user voice commands input via microphone 280.

[0052] Figure 2B is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents a logical collection of all memory modules (including HDD 209 and semiconductor memory 206) accessible by computer module 201 of Figure 2A.

[0053] When the computer module 201 is first powered on, a power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 of FIG. 2A. Hardware devices, such as the ROM 249, that store software are sometimes referred to as firmware. The POST program 250 inspects the hardware within the computer module 201 to ensure proper functioning and typically checks the processor 205, memory 234 (209, 206), and a basic input / output system software (BIOS) module 251, which is typically also stored in the ROM 249, for correct operation. If the POST program 250 runs successfully, the BIOS 251 boots the hard disk drive 210 of FIG. 2A. Booting the hard disk drive 210 executes a bootstrap loader program 252, resident on the hard disk drive 210, via the processor 205. This loads the operating system 253 into the RAM memory 206, upon which the operating system 253 begins operation. Operating system 253 is a system-level application executable by processor 205 and performs a variety of high-level functions, including processor management, memory management, device management, storage management, software application interface, and general-purpose user interface.

[0054] Operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on computer module 201 has enough memory to execute without conflicting with memory allocated to another process. Furthermore, the different types of memory available in computer system 200 of FIG. 2A must be used appropriately so that each process can execute effectively. Thus, aggregate memory 234 is not intended to indicate how specific segments of memory are allocated (unless otherwise noted), but rather is intended to provide a general view of memory accessible by computer system 200 and how such segments are used.

[0055] As shown in FIG. 2B, processor 205 includes a number of functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and a local or internal memory 248, sometimes referred to as a cache memory. Cache memory 248 typically includes a number of storage registers 244-246 within a register section. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using connection 218. Memory 234 is coupled to bus 204 using connection 219.

[0056] Application program 233 includes a sequence of instructions 231, which may include conditional branch and loop instructions. Program 233 may also include data 232 used in the execution of program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative sizes of instruction 231 and memory locations 228-230, a particular instruction may be stored in a single memory location, as indicated by the instruction shown in memory location 230. Alternatively, the instruction may be segmented into several parts, each stored in a separate memory location, as indicated by the instruction segments shown in memory locations 228 and 229.

[0057] Generally, the processor 205 is given a set of instructions to execute within it. The processor 205 waits for a subsequent input, to which the processor 205 responds by executing another set of instructions. Each input may come from one or more of several sources, including data generated by one or more of the input devices 202, 203, data received from an external source via one of the networks 220, 202, data retrieved from one of the storage devices 206, 209, or data retrieved from a storage medium 225 inserted into a corresponding reader 212, all shown in FIG. 2A. Execution of the set of instructions may result in data output. Execution may also include storing data or variables to memory 234.

[0058] Video encoder 114, video decoder 134, and the described methods may use input variables 254 that are stored in corresponding memory locations 255, 256, 257 in memory 234. Video encoder 114, video decoder 134, and the described methods generate output variables 261, which are stored in corresponding memory locations 262, 263, 264 in memory 234. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

[0059] Referring to processor 205 of FIG. 2B, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 cooperate to perform the sequence of micro-operations required to perform a "fetch, decode, and execute" cycle for every instruction in the instruction set that makes up program 233. Each fetch, decode, and execute cycle is A fetch operation to fetch or read an instruction 231 from memory locations 228, 229, 230 A decode operation in which the control unit 239 determines which instructions have been fetched. The control unit 239 and / or the ALU 240 perform the operations to execute the instructions. It has.

[0060] Thereafter, a further fetch, decode, and execute cycle of the next instruction can be performed. Similarly, a store cycle can be performed in which control unit 239 stores or writes a value to memory location 232.

[0061] Each step or sub-process in the methods of Figures 10 and 11 described below is associated with one or more segments of program 233 and is typically performed by register sections 244, 245, 247, ALU 240, and control unit 239 within processor 205 working together to perform a fetch, decode, and execute cycle for all instructions in the instruction set for the noted segment of program 233.

[0062] Figure 3 is a schematic block diagram illustrating the functional modules of the video encoder 114. Figure 4 is a schematic block diagram illustrating the functional modules of the video decoder 134. Generally, data passes between the functional modules of the video decoder 134 and the video encoder 114 in groups of samples or coefficients, such as a division of a block into fixed-size sub-blocks, or as an array. The video encoder 114 and the video decoder 134 can be implemented using a general-purpose computer system 200, as shown in Figures 2A and 2B, with the various functional modules being realized by software executable within the computer system 200, such as one or more software code modules of a software application program 233 resident on a hard disk drive 205 and controlled during its execution by the processor 205, or by dedicated hardware within the computer system 200. Alternatively, the video encoder 114 and the video decoder 134 may be implemented by a combination of software executable within the computer system 200 and dedicated hardware. The video encoder 114, the video decoder 134, and the described methods may alternatively be implemented by dedicated hardware, such as one or more integrated circuits that perform the functions or sub-functions of the described methods. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), an application specific standard product (ASSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or one or more microprocessors and associated memory. In particular, the video encoder 114 includes modules 310-390, and the video decoder 134 includes modules 420-496, which may each be implemented as one or more software code modules of a software application program 233.

[0063] The video encoder 114 of FIG. 3 is an example of a generic video coding (VVC) video encoding pipeline, although other video codecs may be used to perform the processing stages described herein. The video encoder 114 receives captured frame data 113, such as a series of frames, each frame including one or more color channels. The frame data 113 may be in any chroma format, for example, 4:0:0, 4:2:0, 4:2:2, or 4:4:4 chroma format. The block partitioner 310 first divides the frame data 113 into CTUs that are approximately square in shape and configured to use a specific size for the CTUs. The maximum valid size of a CTU may be, for example, 32x32, 64x64, or 128x128 luma samples, as configured by the "sps_log2_ctu_size_minus5" syntax element present in the "sequence parameter set." The CTU size also provides the maximum CU size, since a CTU without further division contains one CU. The block partitioner 310 further divides each CTU into one or more CBs according to the luma coding tree and the chroma coding tree. The luma channel may also be referred to as a primary color channel. Each chroma channel may also be referred to as a secondary color channel. CBs have various sizes and may include both square and non-square aspect ratios. The operation of the block partitioner 310 is further described with reference to FIG. 10. However, in the VVC standard, CBs, CUs, PUs, and TUs always have side lengths that are powers of two. Thus, a current CB, denoted as 312, is output from the block partitioner 310 after iterating over one or more blocks of the CTU according to the luma coding tree and the chroma coding tree of the CTU. Options for dividing a CTU into CBs are further described below with reference to FIGS. 5 and 6. While the operation is generally described per CTU, the video encoder 114 and the video decoder 134 may operate on smaller-sized regions to reduce memory consumption.For example, each CTU may be divided into smaller regions known as "virtual pipeline data units" (VPDUs) of size 64x64. VPDUs form a data granularity more suitable for pipeline processing in hardware architectures, and the reduced memory footprint reduces silicon area and therefore cost compared to operations on full CTUs. When the CTU size is 128x128, restrictions are placed on the allowed coding trees to ensure that processing of one VPDU is fully completed before proceeding to the next VPDU. For example, at the root node of a coding tree for a 128x128 CTU, ternary splitting is prohibited because the resulting CUs (such as 32x128 / 128x32 or further decompositions) cannot be processed in the required progression from one 64x64 region to a subsequent 64x64 region. When the CTU size is 64x64, regardless of the coding tree selected by the encoder, processing always completes one 64x64 region before proceeding to the next 64x64 region, i.e., before proceeding from one CTU to the next.

[0064] The CTUs resulting from the initial partitioning of frame data 113 may be scanned in raster scan order and grouped into one or more "slices." A slice may be an "intra" (or "I") slice. An intra slice (I slice) indicates that all CUs in the slice are intra-predicted. Alternatively, a slice may be uni- or bi-predictive ("P" or "B" slice, respectively), indicating the further availability of uni- and bi-prediction in the slice, respectively.

[0065] In an I slice, the coding tree of each CTU may branch below the 64x64 level into two separate coding trees: one for luma and one for chroma. The use of separate trees allows for different block structures between luma and chroma within the luma 64x64 area of ​​a CTU. For example, a large chroma CB may be co-located with multiple smaller luma CBs, and vice versa. In a P or B slice, a single coding tree for a CTU defines a common block structure for luma and chroma. The resulting blocks of the single tree may be intra-predicted or inter-predicted.

[0066] For each CTU, the video encoder 114 operates in two stages. In the first stage (called the "search" stage), the block partitioner 310 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated "candidate" CB. The first stage involves testing various candidate CBs to select the CB that provides relatively high compression efficiency with relatively low distortion. This testing typically involves Lagrangian optimization, whereby the candidate CBs are evaluated based on a weighted combination of rate (coding cost) and distortion (error with respect to the input frame data 113). The "best" candidate CB (the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into the bitstream 115. Evaluation of the candidate CBs includes the options of using the CB for a given area, or further dividing the area according to various partitioning options and encoding each of the resulting smaller areas with additional CBs, or further dividing the area. Consequently, both the coding tree and the CBs themselves are selected in the search stage.

[0067] For each CB, e.g., CB 312, the video encoder 114 generates a predictive block (PB), indicated by arrow 320. The PB 320 is a prediction of the content of the associated CB 312. A subtractor module 322 generates a difference (or a "residual," which refers to the difference in the spatial domain) between the PB 320 and the CB 312, indicated as 324. The difference 324 is a block-sized difference between corresponding samples in the PB 320 and the CB 312. The difference 324 is transformed and quantized and represented as a transform block (TB), indicated by arrow 336. The PB 320 and the associated TB 336 are typically selected from one of many possible candidate CBs, for example, based on estimated cost or distortion.

[0068] A candidate coding block (CB) is a CB that results from one of the prediction modes available to the video encoder 114 for the associated PB and the resulting residual. The TB 336, when combined with the predicted PB in the video decoder 114, reduces the difference between the decoded CB and the original CB 312, at the expense of additional signaling in the bitstream.

[0069] Thus, each candidate coding block (CB), i.e., a prediction block (PB) combined with a transform block (TB), has an associated coding cost (or “rate”) and an associated differential (or “distortion”). The distortion of a CB is typically estimated as a sample value difference, such as the sum of absolute differences (SAD) or the sum of squared differences (SSD). The resulting estimate from each candidate PB may be determined by a mode selector 386 using the differential 324 to determine a prediction mode 387. The prediction mode 387 indicates a decision to use a particular prediction mode for the current CB, e.g., intraframe prediction or interframe prediction. Estimating the coding cost associated with each candidate prediction mode and the corresponding residual coding can be performed at a significantly lower cost than entropy coding of the residual. Thus, several candidate modes can be evaluated to determine the optimal mode for rate-distortion detection, even in a real-time video encoder.

[0070] Determining the optimal mode in terms of rate distortion is typically accomplished using a variation of Lagrangian optimization.

[0071] A Lagrangian or similar optimization process may be used both to select the optimal partitioning of the CTUs into CBs (by the block partitioner 310) as well as to select the best prediction mode from multiple possibilities. Through application of a Lagrangian optimization process of the candidate modes in the mode selector module 386, the intra-prediction mode with the lowest cost measure is selected as the "best" mode. The lowest-cost mode includes a selected secondary transform index 388, which is also encoded in the bitstream 115 by the entropy encoder 338.

[0072] In the second stage of operation of the video encoder 114 (called the "coding" stage), iterations over the determined coding tree for each CTU are performed in the video encoder 114. For CTUs using separate trees, for each 64x64 luma region of the CTU, the luma coding tree is coded first, followed by the chroma coding tree. Within the luma coding tree, only the luma CB is coded, and within the chroma coding tree, only the chroma CB is coded. For CTUs using a shared tree, a single tree describes the CU, i.e., the luma CB and the chroma CB, according to the common block structure of the shared tree.

[0073] The entropy encoder 338 supports both variable-length coding of syntax elements and arithmetic coding of syntax elements. Portions of the bitstream, such as "parameter sets," e.g., the sequence parameter set (SPS) and picture parameter set (PPS), use a combination of fixed-length and variable-length codewords. A slice (also called a continuous portion) has a slice header that uses variable-length coding, followed by slice data that uses arithmetic coding. The slice header defines parameters specific to the current slice, such as slice-level quantization parameter offsets. The slice data contains syntax elements for each CTU in the slice. The use of variable-length coding and arithmetic coding requires sequential parsing within each portion of the bitstream. Portions may be delineated with start codes to form "network abstraction layer units" or "NAL units." Arithmetic coding is supported using a context-adaptive binary arithmetic coding process. Arithmetically coded syntax elements consist of a sequence of one or more "bins." Bins, like bits, have a value of "0" or "1." However, bins are not coded as discrete bits in the bitstream 115. Bins have an associated predicted (or "likely" or "most likely") value and an associated probability, known as a "context." If the actual bin to be coded matches the predicted value, a "most probable symbol" (MPS) is coded. Coding the most probable symbol is relatively inexpensive in terms of consumed bits in the bitstream 115 and involves a cost equivalent to less than one discrete bit. If the actual bin to be coded does not match a possible value, a "least probable symbol" (LPS) is coded. Coding the least probable symbol has a relatively high cost in terms of consumed bits. The bin coding technique allows for efficient coding of bins with distorted "0" vs. "1" probabilities. For syntax elements with two possible values ​​(i.e., "flags"), a single bin is sufficient.For syntax elements that have many possible values, a sequence of bins is required.

[0074] The presence of a later bin in a sequence may be determined based on the value of a previous bin in the sequence. Furthermore, each bin may be associated with two or more contexts. The selection of a particular context may depend on the previous bin in the syntax element, the bin values ​​of adjacent syntax elements (i.e., from adjacent blocks), etc. Each time a context-coded bin is coded, the context selected for that bin (if any) is updated in a manner that reflects the new bin value. In this way, binary arithmetic coding schemes are said to be adaptive.

[0075] Also supported by the video encoder 114 are bins that lack context ("bypass bins"). Bypass bins are coded assuming an equal probability distribution between "0" and "1". Thus, each bin has a coding cost of 1 bit in the bitstream 115. The lack of context saves memory and reduces complexity, and therefore bypass bins are used when the distribution of values ​​for a particular bin is skewed. One example of an entropy coder that uses context and adaptation is known in the art as CABAC (Context-Adaptive Binary Arithmetic Coder), and many variations of this coder are used in video coding.

[0076] The entropy encoder 338 encodes the quantization parameter 392 and, if used for the current CB, encodes the LFNST index 388 using a combination of context-coded and bypass-coded bins. The quantization parameter 392 is coded using a "delta QP." The delta QP is signaled at most once in each area known as a "quantization group." The quantization parameter 392 is applied to the residual coefficients of the luma CB. An adjusted quantization parameter is applied to the residual coefficients of the co-located chroma CB. The adjusted quantization parameter may include mapping from the luma quantization parameter 392 according to a mapping table and CU level offset selected from a list of offsets. The secondary transform index 388 is signaled when the residual associated with the transform block contains significant residual coefficients only at coefficient positions that are transformed into primary coefficients by application of a secondary transform.

[0077] The multiplexer module 384 outputs the PB 320 from the intra-frame prediction module 364 according to a determined best intra-prediction mode selected from the tested prediction modes of each candidate CB. The candidate prediction modes need not include all possible prediction modes supported by the video encoder 114. Intra-prediction is classified into three types: "DC intra-prediction" involves populating a PB with a single value representing the average of nearby reconstructed samples. "Planar intra-prediction" involves populating a PB with samples according to a plane, where the DC offset and vertical and horizontal gradients are derived from nearby reconstructed neighboring samples. The nearby reconstructed samples typically include a row of reconstructed samples above the current PB, extending to the right of the PB to some extent, and a column of reconstructed samples to the left of the current PB, extending downward beyond the PB to some extent. "Angular intra-prediction" involves populating a PB with reconstructed neighboring samples filtered in a particular direction (or "angle") and propagated across the PB. In VVC, 65 angles are supported, and rectangular blocks can utilize additional angles not available to square blocks, for a total of 87 angles. A fourth type of intra prediction is available for chroma PB, whereby the PB is generated from co-located luma reconstructed samples according to a "cross-component linear model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived model is used to generate a block of samples for the chroma PB from the co-located luma samples.

[0078] If no previously reconstructed samples are available, for example at the edge of a frame, a default halftone value of half the range of samples is used. For example, for 10-bit video, a value of 512 is used. Because no samples are previously available for the CB located in the top-left position of the frame, angular and planar intra prediction modes produce the same output as DC prediction mode, i.e., a plane of samples with halftone values ​​as magnitudes.

[0079] For inter-frame prediction, the prediction block 382 is generated by the motion compensation module 380 using samples from one or two frames preceding the current frame in the coding order in the bitstream and output as the PB 320 by the multiplexer module 384. Furthermore, for inter-frame prediction, a single coding tree is typically used for both the luma and chroma channels. The order in which frames in the bitstream are coded may differ from the order in which the frames were captured or displayed. When one frame is used for prediction, the block is said to be "uni-predictive" and has one associated motion vector. When two frames are used for prediction, the block is said to be "bi-predictive" and has two associated motion vectors. For P slices, each CU can be intra-predicted or uni-predicted. For B slices, each CU can be intra-predicted, uni-predictive, or bi-predictive. Frames are typically coded using a "group of pictures" structure, allowing for temporal hierarchy of frames. A frame may be divided into multiple slices, each of which codes a portion of the frame. The temporal hierarchy of frames allows frames to refer to previous and subsequent pictures in the order in which they are displayed. Pictures are coded in the order necessary to ensure that dependencies for decoding each frame are satisfied.

[0080] Samples are selected according to a motion vector 378 and a reference picture index. The motion vector 378 and the reference picture index apply to all color channels; therefore, inter prediction is primarily described in terms of operation on the PU rather than the PB; i.e., the decomposition of each CTU into one or more inter-predicted blocks is described using a single coding tree. Inter prediction methods may vary in the number of motion parameters and their precision. The motion parameters typically comprise a reference frame index indicating which reference frame from a list of reference frames should be used and a spatial transformation for each of the reference frames, but may include more frames, dedicated frames, or complex affine parameters such as scaling and rotation. In addition, a predetermined motion refinement process may be applied to generate a dense motion estimate based on the referenced sample block.

[0081] After determining and selecting PB 320 and subtracting it from the original sample block in subtractor 322, a residual with the lowest coding cost, represented as 324, is obtained and subjected to lossy compression. The lossy compression process includes the steps of transform, quantization, and entropy coding. A forward linear transform module 326 applies a forward transform to the difference 324, converting it from the spatial domain to the frequency domain and generating linear transform coefficients, represented by arrow 328. The maximum linear transform size in one dimension is either a 32-point DCT-2 or a 64-point DCT-2 transform, as set by the "sps_max_luma_transform_size_64_flag" in the sequence parameter set. If the CB to be coded is larger than the maximum supported linear transform size, represented as a block size, e.g., 64x64 or 32x32, the linear transform 326 is applied in a tiled manner to transform all samples of the difference 324. When non-square CBs are used, tiling is also performed using the largest available transform size in each dimension of the CB. For example, if a maximum transform size of 32 is used, a 64x16 CB uses two 32x16 primary transforms arranged in a tiled fashion. If a CB is larger than the maximum supported transform size, the CB is filled with TBs in a tiled fashion. For example, a 128x128 CB with a 64-pt transform maximum size is filled with four 64x64 TBs arranged in a 2x2 configuration. A 64x128 CB with a 32-pt transform maximum size is filled with eight 32x32 TBs arranged in a 2x4 configuration.

[0082] Application of the transform 326 results in multiple TBs for the CB. If each application of the transform operates on a difference 324 larger than 32x32, e.g., a 64x64 TB, all resulting primary transform coefficients 328 outside the top-left 32x32 area of ​​the TB are set to zero, i.e., discarded. The remaining primary transform coefficients 328 are passed to a quantizer module 334. The primary transform coefficients 328 are quantized according to a quantization parameter 392 associated with the CB to generate primary transform coefficients 332. In addition to the quantization parameter 392, the quantizer module 334 may also apply a "scaling list" to enable non-uniform quantization within the TB by further scaling the residual coefficients according to their spatial position within the TB. The quantization parameter 392 may be different for the luma CB versus each chroma CB. The primary transform coefficients 332 are passed to a forward secondary transform module 330, which generates transform coefficients represented by arrow 336 by performing a non-separable secondary transform (NSST) operation or by bypassing the secondary transform. The forward primary transform is typically separable, transforming a set of rows and then a set of columns for each TB. The forward primary transform module 326 uses either a Type II Discrete Cosine Transform (DCT-2) in the horizontal and vertical directions, or bypassing the transform in the horizontal and vertical directions, or a combination of a Type VII Discrete Sine Transform (DST-7) and a Type VIII Discrete Cosine Transform (DCT-8) in the horizontal or vertical directions for luma TBs whose width and height do not exceed 16 samples. The use of a combination of DST-7 and DCT-8 is referred to as a "Multiple Transform Selection Set" (MTS) in the VVC standard.

[0083] The forward secondary transform of module 330 is generally a non-separable transform, which is applied only to the residual of intra-predicted CUs and may nevertheless be bypassed. The forward secondary transform operates on either 16 samples (organized as a 4x4 sub-block at the top left of the primary transform coefficients 328) or 48 samples (organized as three 4x4 sub-blocks at the top left of the primary transform coefficients 328) to generate a set of secondary transform coefficients. The set of secondary transform coefficients may be fewer in number than the set of primary transform coefficients from which they are derived. By applying the secondary transform only to sets of coefficients that are adjacent to each other and include a DC coefficient, the secondary transform is referred to as a "low-frequency non-separable secondary transform" (LFNST). Furthermore, when an LFNST is applied, all remaining coefficients in the TB must be zero in both the primary transform domain and the secondary transform domain.

[0084] The quantization parameter 392 is constant for a given TB, thus resulting in uniform scaling for the generation of residual coefficients in the primary transform domain for the TB. The quantization parameter 392 may vary periodically with the signaled "delta quantization parameter." The delta quantization parameter (delta QP) is signaled once for CUs contained within a given area, called a "quantization group." If the CU is larger than the quantization group size, the delta QP is signaled once with one of the TBs of the CU. That is, the delta QP is signaled by the entropy encoder 338 once for the first quantization group of the CU and not for any subsequent quantization groups of the CU. Non-uniform scaling is also possible by applying a "quantization matrix," whereby the scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter 392 and a corresponding entry in the scaling matrix. The scaling matrix may have a size smaller than the TB size, and when applied to the TB, a nearest-neighbor approach is used to provide a scaling value for each residual coefficient from the scaling matrix, which has a size smaller than the TB size. The residual coefficients 336 are provided to an entropy encoder 338 for encoding in the bitstream 115. Typically, the residual coefficients of each TB having at least one significant residual coefficient of a TU are scanned according to a scan pattern to generate an ordered list of values. The scan pattern generally scans the TB as a sequence of 4x4 "subblocks," providing a regular scan operation with a granularity of 4x4 sets of residual coefficients, with the arrangement of the subblocks depending on the size of the TB. The scan within each subblock and the progression from one subblock to the next typically follows a backward diagonal scan pattern. Additionally, a quantization parameter 392 is coded into the bitstream 115 using a delta QP syntax element, and a secondary transform index 388 is coded into the bitstream 115.

[0085] As described above, the video encoder 114 needs to access a frame representation that corresponds to the decoded frame representation seen by the video decoder 134. Accordingly, the residual coefficients 336 pass through an inverse secondary transform module 344, operating according to secondary transform indices 388 to generate intermediate inverse transform coefficients represented by arrow 342. The intermediate inverse transform coefficients are inverse quantized by an inverse quantization module 340 according to a quantization parameter 392 to generate inverse transform coefficients represented by arrow 346. The inverse quantization module 340 may also perform inverse non-uniform scaling of the residual coefficients using a scaling list, corresponding to the forward scaling performed in the quantization module 334. The intermediate inverse transform coefficients 346 are passed to an inverse primary transform module 348 to generate residual samples of the TUs, represented by arrow 350. The inverse primary transform module 348 applies a DCT-2 transform horizontally and vertically, constrained by the maximum available transform size, as described with reference to the forward primary transform module 326. The type of inverse transform performed by the inverse secondary transform module 344 corresponds to the type of forward transform performed by the forward secondary transform module 330. The type of inverse transform performed by the inverse primary transform module 348 corresponds to the type of primary transform performed by the primary transform module 326. The summation module 352 sums the residual samples 350 and the PU 320 to generate reconstructed samples of the CU (indicated by arrow 354).

[0086] The reconstructed samples 354 are passed to a reference sample cache 356 and an in-loop filter module 368. The reference sample cache 356, typically implemented using static RAM on the ASIC (thus avoiding expensive off-chip memory accesses), provides the minimum sample storage required to satisfy the dependencies for generating intra-frame PBs for subsequent CUs in the frame. The minimum dependencies typically include a "line buffer" of samples along the bottom of a row of CTUs for use by the next row of CTUs, and column buffering, whose extent is set by the height of the CTU. The reference sample cache 356 supplies reference samples (represented by arrow 358) to a reference sample filter 360. The sample filter 360 applies a smoothing operation to generate filtered reference samples (indicated by arrow 362). The filtered reference samples 362 are used by an intra-frame prediction module 364 to generate intra-predicted blocks of samples, represented by arrow 366. For each candidate intra-prediction mode, the intra-frame prediction module 364 generates a block of samples, i.e., 366. The block of samples 366 is generated by the module 364 using techniques such as DC, planar, or angular intra-prediction.

[0087] The in-loop filter module 368 applies several filtering stages to the reconstructed samples 354. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned to CU boundaries to reduce artifacts due to discontinuities. Another filtering stage present in the in-loop filter module 368 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. A further filtering stage available in the in-loop filter module 368 is a "sample adaptive offset" (SAO) filter. The SAO filter operates by first classifying the reconstructed samples into one or more categories and then applying an offset at the sample level according to the assigned category.

[0088] The filtered samples, represented by arrow 370, are output from the in-loop filter module 368. The filtered samples 370 are stored in a frame buffer 372. The frame buffer 372 typically has the capacity to store several pictures (e.g., up to 16) and is therefore stored in the memory 206. The frame buffer 372 is not typically stored using on-chip memory due to the large memory consumption required. Therefore, accessing the frame buffer 372 is costly in terms of memory bandwidth. The frame buffer 372 provides reference frames (represented by arrow 374) to a motion estimation module 376 and a motion compensation module 380.

[0089] The motion estimation module 376 estimates several “motion vectors” (denoted as 378), each of which is a Cartesian spatial offset from the position of the current CB and references a block in one of the reference frames in the frame buffer 372. A filtered block of reference samples (denoted as 382) is generated for each motion vector. The filtered reference samples 382 form further candidate modes available for potential selection by the mode selector 386. Furthermore, for a given CU, the PU 320 may be formed using one reference block (“uni-predicted”) or two reference blocks (“bi-predicted”). For a selected motion vector, the motion compensation module 380 generates the PB 320 according to a filtering process that supports sub-pixel accuracy in motion vectors. Thus, the motion estimation module 376 (which operates on many candidate motion vectors) can perform a simplified filtering process compared to that of the motion compensation module 380 (which operates on only the selected candidate), achieving a reduced amount of computation. If video encoder 114 selects inter prediction for the CU, motion vector 378 is coded into bitstream 115.

[0090] 3 is described with reference to generic video coding (VVC), other video coding standards or implementations may also employ the processing stages of modules 310-390. The frame data 113 (and bitstream 115) may also be stored in memory 206, hard disk drive 210, CD-ROM, Blu-ray disc TM, or other computer-readable storage medium. Additionally, frame data 113 (and bitstream 115) may be received from (or transmitted to) an external source, such as a server connected to a communications network 220 or a radio frequency receiver. Communications network 220 may provide limited bandwidth, necessitating the use of rate control in video encoder 114 to avoid saturating the network when it is difficult to compress frame data 113. Furthermore, bitstream 115 may be constructed from one or more slices representing spatial sections (collections of CTUs) of frame data 113, generated by one or more instances of video encoder 114 operating cooperatively under the control of processor 205. In the context of this disclosure, a slice may also be referred to as a "contiguous portion" of the bitstream. Slices are contiguous within the bitstream and can be encoded or decoded as separate portions, for example, when parallel processing is used.

[0091] The video decoder 134 is shown in Figure 4. The video decoder 134 of Figure 4 is an example of a generic video coding (VVC) video decoding pipeline, although other video codecs may be used to perform the processing stages described herein. As shown in Figure 4, a bitstream 133 is input to the video decoder 134. The bitstream 133 may be stored in memory 206, a hard disk drive 210, a CD-ROM, a Blu-ray disc TM or other non-transitory computer-readable storage medium. Alternatively, bitstream 133 may be received from an external source, such as a server connected to communications network 220 or a radio frequency receiver. Bitstream 133 includes encoded syntax elements representing imaged frame data to be decoded.

[0092] The bitstream 133 is input to the entropy decoder module 420. The entropy decoder module 420 extracts syntax elements from the bitstream 133 by decoding a sequence of "bins" and passes the values ​​of the syntax elements to other modules in the video decoder 134. The entropy decoder module 420 uses variable-length and fixed-length decoding to decode the SPS, PPS, or slice header of the arithmetic decoding engine and decodes the syntax elements of the slice data as a sequence of one or more bins. Each bin can use one or more "contexts," which describe the probability levels used to code the bin's "1" and "0" values. If multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts to decode the bin. The bin decoding process forms a sequential feedback loop, so each slice can be decoded in its entirety by a given instance of the entropy decoder 420. A single (or a few) high-performance entropy decoder 420 instances can decode all slices of a frame from bitstream 115, and multiple low-performance entropy decoder 420 instances can simultaneously decode slices of a frame from bitstream 133.

[0093] The entropy decoder module 420 applies an arithmetic coding algorithm, such as "context-adaptive binary arithmetic coding" (CABAC), to decode syntax elements from the bitstream 133. The decoded syntax elements are used to reconstruct parameters within the video decoder 134. The parameters include residual coefficients (represented by arrow 424), quantization parameters 474, secondary transform indices 470, and mode selection information such as intra-prediction modes (represented by arrow 458). The mode selection information also includes information such as motion vectors and the partitioning of each CTU into one or more CBs. The parameters are used to generate PBs, typically combined with sample data from previously decoded CBs.

[0094] The residual coefficients 424 are passed to an inverse secondary transform module 436 if no operation is performed (bypass) according to the secondary transform index or if a secondary transform is applied. The inverse secondary transform module 436 generates reconstructed transform coefficients 432, i.e., primary transform domain coefficients, from the secondary transform domain coefficients. The reconstructed transform coefficients 432 are input to an inverse quantizer module 428. The inverse quantizer module 428 performs inverse quantization (or “scaling”) on the residual coefficients 432, i.e., in the primary transform coefficient domain, according to a quantization parameter 474 to create reconstructed intermediate transform coefficients, represented by arrow 440. The inverse quantizer module 428 can also apply a scaling matrix to provide non-uniform inverse quantization within the TB, corresponding to the operation of the inverse quantizer module 340. If the bitstream 133 indicates that a non-uniform inverse quantization matrix is ​​to be used, the video decoder 134 reads the quantization matrix from the bitstream 133 as a sequence of scaling factors and places the scaling factors in the matrix. Inverse scaling uses a quantization matrix in combination with the quantization parameters to create reconstructed intermediate transform coefficients 440 .

[0095] The reconstructed transform coefficients 440 are passed to an inverse primary transform module 444. The module 444 converts the coefficients 440 from the frequency domain back to the spatial domain. The inverse primary transform module 444 applies an inverse DCT-2 transform horizontally and vertically, constrained by the maximum available transform size, as described with reference to the forward primary transform module 326. The result of the operation of the module 444 is a block of residual samples, represented by arrow 448. The block of residual samples 448 is equal in size to the corresponding CB. The residual samples 448 are provided to a summation module 450. In the summation module 450, the residual samples 448 are added to the decoded PB (represented as 452) to generate a block of reconstructed samples, represented by arrow 456. The reconstructed samples 456 are provided to a reconstructed sample cache 460 and an in-loop filtering module 488. The in-loop filtering module 488 generates a reconstructed block of frame samples, represented as 492. The frame samples 492 are written to a frame buffer 496 .

[0096] The reconstructed sample cache 460 operates similarly to the reconstructed sample cache 356 of the video encoder 114. The reconstructed sample cache 460 provides storage for reconstructed samples needed to intra-predict subsequent CBs without the memory 206 (e.g., by using the data 232, which is typically on-chip memory, instead). Reference samples, represented by arrow 464, are retrieved from the reconstructed sample cache 460 and provided to a reference sample filter 468 to generate filtered reference samples, indicated by arrow 472. The filtered reference samples 472 are provided to an intra-frame prediction module 476. The module 476 generates blocks of intra-predicted samples, represented by arrow 480, according to the intra-prediction mode parameter 458 signaled in the bitstream 133 and decoded by the entropy decoder 420. The sample blocks 480 are generated using modes such as DC, planar, or angular intra-prediction.

[0097] If the prediction mode of a CB is indicated to use intra prediction in the bitstream 133, the intra-predicted samples 480 form the decoded PB 452 via the multiplexer module 484. Intra prediction generates a predictive block (PB) of samples, i.e., a block in one color component derived using "neighboring samples" in the same color component. Neighboring samples are samples that neighbor the current block and have already been reconstructed by preceding them in block decoding order. When a luma block and a chroma block are co-located, the luma block and the chroma block may use different intra-prediction modes. However, the two chroma CBs share the same intra-prediction mode.

[0098] If the prediction mode of the CB is indicated to be inter-prediction in the bitstream 133, the motion compensation module 434 uses the motion vector (decoded from the bitstream 133 by the entropy decoder 420) and the reference frame index to select and filter a block of samples 498 from the frame buffer 496 to generate a block of inter-predicted samples represented as 438. The block of samples 498 is obtained from a previously decoded frame stored in the frame buffer 496. In the bi-predictive case, two blocks of samples are generated and blended together to generate samples for the decoded PB 452. The frame buffer 496 is populated with filtered block data 492 from the in-loop filtering module 488. Similar to the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 applies any of DBF, ALF, and SAO filtering operations. Generally, motion vectors are applied to both the luma and chroma channels, although the filtering processes for sub-sample interpolation in the luma and chroma channels are different.

[0099] 5 is a schematic block diagram illustrating a set 500 of available divisions or splits of a region into one or more sub-regions in a generic video coding tree structure. As described with reference to FIG. 3, the block partitioner 310 of the encoder 114 can utilize the divisions shown in set 500 to divide each CTU into one or more CUs or CBs according to the coding tree.

[0100] Although set 500 shows only square regions that are divided into other, possibly non-square, sub-regions, it should be understood that set 500 illustrates a potential division of a parent node in a coding tree into child nodes in the coding tree and does not require the parent node to correspond to a square region. If the containing region is non-square, the dimensions of the blocks resulting from the division are scaled according to the aspect ratio of the containing block. If a region is not further divided, i.e., at a leaf node of the coding tree, a CU occupies the region.

[0101] The process of subdividing a region into subregions must terminate when the resulting subregions reach a minimum CU size, typically 4x4 luma samples. In addition to constraining CUs to prohibit block regions smaller than a predetermined minimum size, e.g., 16 samples, CUs are constrained to have a minimum width or height of 4. Other minimum values ​​are possible for both width and height, or for width or height. The subdivision process can also terminate before the deepest level of decomposition, resulting in a CU larger than the minimum CU size. It is possible for no splitting to occur, resulting in a single CU occupying the entire CTU. A single CU occupying the entire CTU is the largest available coding unit size. Due to the use of subsampled chroma formats such as 4:2:0, the video encoder 114 and video decoder 134 configurations may terminate splitting regions in the chroma channels earlier than in the luma channel, including in the case of a shared coding tree that defines the block structure for the luma and chroma channels. If separate coding trees are used for luma and chroma, constraints on the available splitting operations guarantee a minimum chroma CB region of 16 samples, but such a CB is co-located with a larger luma region, e.g., 64 luma samples.

[0102] At leaf nodes of the coding tree, there are CUs and no further subdivisions. For example, leaf node 510 contains one CU. At non-leaf nodes of the coding tree, there are divisions into two or more further nodes, each of which can be a leaf node forming one CU or a non-leaf node containing further divisions into smaller regions. At each leaf node of the coding tree, there is one coding block for each color channel. A split that terminates at the same depth for both luma and chroma results in three co-located CBs. A split that terminates at a deeper depth for luma than for chroma results in multiple luma CBs being co-located with the chroma channel CBs.

[0103] The quadtree split 512 divides the inclusion region into four equally sized regions, as shown in Figure 5. Compared to HEVC, generic video coding (VVC) achieves additional flexibility with additional splits, including a horizontal binary split 514 and a vertical binary split 516. Splits 514 and 516 each divide the inclusion region into two equally sized regions. The splits are along either a horizontal boundary (514) or a vertical boundary (516) within the containing block.

[0104] Further flexibility is achieved in generalized video coding by adding a ternary horizontal split 518 and a ternary vertical split 520. The ternary splits 518 and 520 divide a block into three regions bounded either horizontally (518) or vertically (520) along ¼ and ¾ of the width or height of the containing region. The combination of a quadtree, binary tree, and ternary tree is called a QTBTTT. The root of the tree contains zero or more quadtree splits (the “QT” section of the tree). When the QT section ends, zero or more binary or ternary splits (the “multi-tree” or “MT” section of the tree) arise, finally terminating in a tree leaf node CB or CU. If the tree describes all color channels, the tree leaf node is a CU. If the tree describes the luma or chroma channels, the tree leaf node is a CB.

[0105] Compared to HEVC, which only supports quadtree trees and therefore only square blocks, QTBTT results in more possible CU sizes, especially considering the possible recursive application of binary and / or ternary tree partitioning. When only quadtree partitioning is available, each increase in coding tree depth corresponds to a reduction in CU size to one-quarter of the size of the parent area. In VVC, the availability of binary and ternary partitioning means that coding tree depth does not directly correspond to CU area. The possibility of unusual (non-square) block sizes can be reduced by constraining split options to eliminate splits that result in block widths or heights that are either less than four samples or are not a multiple of four samples. Generally, constraints are applied when considering luma samples. However, in the described configuration, constraints can be applied separately to blocks for chroma channels. Applying constraints to split options to chroma channels can result in different minimum block sizes for luma versus chroma, for example, when the frame data is in 4:2:0 chroma format or 4:2:2 chroma format. Each division produces a subregion whose side dimensions are unchanged, halved, or quartered with respect to the containing region, and since the CTU size is a power of two, all CU side dimensions are also powers of two.

[0106] 6 is a schematic flow diagram illustrating a data flow 600 of a QTBTTT (or "coding tree") structure used in general-purpose video coding. The QTBTTT structure is used for each CTU to define the partitioning of the CTU into one or more CUs. The QTBTTT structure for each CTU is determined by a block partitioner 310 in video encoder 114 and encoded into bitstream 115 or decoded from bitstream 133 by an entropy decoder 420 in video decoder 134. Data flow 600 further characterizes the permissible combinations available to block partitioner 310 for partitioning the CTU into one or more CUs according to the partitioning shown in FIG. 5.

[0107] Starting from the top level of the hierarchy, i.e., CTU, zero or more quadtree splits are first performed. Specifically, a quadtree (QT) split decision 610 is made by the block partitioner 310. A decision at 610 returning a "1" symbol indicates a decision to split the current node into four subnodes according to the quadtree split 512. This results in four new nodes, such as 620, being generated, and for each new node, a return is made to the QT split decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates that no further splits should be performed (returning a "0" symbol), the quadtree split stops and a multi-tree (MT) split is then considered.

[0108] First, an MT split decision 612 is made by the block partitioner 310. At 612, a decision to perform MT splitting is indicated. Returning a "0" symbol at decision 612 indicates that no further splitting of the node into subnodes is performed. If no further splitting of the node is performed, the node is a leaf node of the coding tree and corresponds to a CU. The leaf node is output at 622. Alternatively, if MT split 612 indicates a decision to perform MT splitting (returning a "1" symbol), the block partitioner 310 proceeds to direction decision 614.

[0109] The direction decision 614 indicates the direction of the MT partition as either horizontal ("H" or "0") or vertical ("V" or "1"). The block partitioner 310 proceeds to decision 616 if decision 614 returns a "0" indicating a horizontal direction. The block partitioner 310 proceeds to decision 618 if decision 614 returns a "1" indicating a vertical direction.

[0110] In each of decisions 616 and 618, the number of partitions for the MT split is indicated as either two (binary split or "BT" node) or three (ternary split or "TT") for the BT / TT split. That is, BT / TT split decision 616 is made by block partitioner 310 when the indicated direction from 614 is horizontal, and BT / TT split decision 618 is made by block partitioner 310 when the indicated direction from 614 is vertical.

[0111] The BT / TT split decision 616 indicates whether the horizontal split is a binary split 514, indicated by returning a "0," or a ternary split 518, indicated by returning a "1." If the BT / TT split decision 616 indicates a binary split, then in an HBT CTU node generation step 625, two nodes are generated by the block partitioner 310 according to the horizontal binary split 514. If the BT / TT split 616 indicates a ternary split, then in an HTT CTU node generation step 626, three nodes are generated by the block partitioner 310 according to the horizontal ternary split 518.

[0112] The BT / TT split decision 618 indicates whether the vertical split is a binary split 516, indicated by returning a "0," or a ternary split 520, indicated by returning a "1." If the BT / TT split 618 indicates a binary split, then in a VBT CTU node generation step 627, two nodes are generated by the block partitioner 310 according to the vertical binary split 516. If the BT / TT split 618 indicates a ternary split, then in a VTT CTU node generation step 628, three nodes are generated by the block partitioner 310 according to the vertical ternary split 520. For each node resulting from steps 625-628, the recursion of the data flow 600 back to the MT split decision 612 is applied in either a left-to-right or top-to-bottom order, depending on the direction 614. As a result, binary and ternary tree splitting can be applied to generate CUs of various sizes.

[0113] 7A and 7B provide an example division 700 of a CTU 710 into several CUs or CBs. An example of a CU 712 is shown in FIG. 7A. FIG. 7A shows the spatial arrangement of CUs in the CTU 710. The example division 700 is also shown as a coding tree 720 in FIG. 7B.

[0114] At each non-leaf node in CTU 710 in FIG. 7A , e.g., nodes 714, 716, and 718, the contained nodes (which may be further split or may be CUs) are scanned or traversed in “Z-order” to create a list of nodes, represented as columns in coding tree 720. In the case of quadtree splitting, the Z-order scan is from top-left to right, followed by bottom-left to right. In the case of horizontal and vertical splits, the Z-order scan (traversal) simplifies to a top-to-bottom scan and a left-to-right scan, respectively. Coding tree 720 in FIG. 7B lists all nodes and CUs according to the applied scan order. Each split generates a list of two, three, or four new nodes at the next level of the tree until a leaf node (CU) is reached.

[0115] Once the block partitioner 310 has decomposed the image into CTUs and then into CUs, and the CUs are used to generate respective residual blocks (324) as described with reference to FIG. 3, the residual blocks are forward transformed and quantized by the video encoder 114. The resulting TBs 336 are then scanned to form a sequential list of residual coefficients as part of the operation of the entropy coding module 338. An equivalent process is performed in the video decoder 134 to obtain the TBs from the bitstream 133.

[0116] FIG. 8 shows a syntax structure 800 for a bitstream 801 having one or more slices. Each slice contains multiple coding units. The bitstream 801 encodes image frame data 113 and is divided into multiple coding tree units, each of which is divided into one or more coding units, which are then divided into one or more transform blocks. The bitstream 801 may be generated by the video encoder 114, e.g., as bitstream 115, or parsed by the video decoder 134, e.g., as bitstream 133. The bitstream 801 is divided into portions, e.g., network abstraction layer (NAL) units, and delineation is achieved by prefixing each NAL unit with a NAL unit header, such as 808. The NAL unit header contains a NAL unit type that identifies the content of the following NAL unit. The video parameter set (VPS) 810 has a NAL unit type named "VPS_NUT" and contains parameters applicable to all layers of the bitstream. The VPS 810 may include a profile_tier_level syntax structure 830. The structure 830 specifies the profile of the bitstream with a "general_profile_idc" syntax element and a general_constraint_info syntax structure 832 that specifies the subprofile (if any) of the selected profile. The general_constraint_info syntax structure 832 includes a flag no_luma_transform_size_64_constraint_flag 832a and a codeword max_log2_ctu_size_constraint_idc 832b. The flag 832a provides the maximum transform block size constraint (e.g., 64) for the bitstream, providing a high-level (in the context of the bitstream's structure) indication of the tools required to encode or decode the bitstream. The 64-point transform differs from other transforms in that only the first 32 residual coefficients are scanned and coded.For example, a 64x64TB may only have significant (non-zero) residual coefficients in the upper-left 32x32 region. Flag 832a indicates a constraint that the maximum primary transform size is limited to 32 points horizontally and vertically, or left unconstrained; in this case, the maximum primary transform size supported by the VVC standard is 64 points horizontally and vertically. Flag 832a is a fixed-length codeword or a flag typically 1 bit in size. Further limiting the maximum transform size to a smaller value, such as 16 points or 8 points, can be implemented as well. Similarly, codeword 832b provides a maximum CTU size constraint for the bitstream, providing a high-level indication of the tools required to encode or decode the bitstream. Codeword 832b is a fixed-length codeword typically 2 bits in size. Structure 832 defines the specific sub-profile associated with the implementation of video encoder 114 and video decoder 134. Syntax structure 832 includes no_scaling_list_constraint_flag 832c. Flag 832c, when active (value equal to 1), indicates that no scaling lists are allowed in the bitstream.

[0117] The sequence parameter set (SPS) 812 has a NAL unit type named "SPS_NUT" and may also include a profile_tier_level syntax structure 830. The profile_level_tier syntax structure 830 is included in either the VPS 810 or the SPS 812, but not both. The sequence parameter set (SPS) 812 defines sequence-level parameters such as the profile (set of tools) used to encode and decode the bitstream, chroma format, sample bit depth, and frame resolution. The SPS 812 also specifies which coding tools may be used in a particular bitstream; the selection is a subset of the tools indicated as available by the profile and subprofile. Structure 834 indicates examples of tools available by the subprofile. The flag sps_max_luma_transform_size_64_flag 834a indicates whether a 64pt linear transform may be used. Flag 834a can indicate the use of a 64pt linear transform only if the constraint flag 832a does not prohibit the use of a 64pt linear transform. The sps_log2_ctu_size_minus5 codeword 834b indicates the size of the CTU (i.e., the maximum coding unit size) using a 2-bit fixed-length codeword. Codeword 834b may have values ​​0, 1, or 2, indicating a CTU size of 32x32, 64x64, or 128x128, respectively. The value 3 is reserved in the initial ("Version 1") profile of VVC. Codeword 834b may not exceed the value of 832b; thus, 832b imposes a limit on CTU size as part of the subprofile definition. The sps_explicit_scaling_list_enabled_flag 834c indicates whether a scaling list can be used in the bitstream for non-uniform quantization within a given TB. If flag 832c is active (e.g., a value equal to 1), flag 834c must be inactive (a value equal to 0).

[0118] Flag 834a indicates the maximum valid transform block size of the bitstream. Flag 834a is constrained based on the corresponding constraint flag 832a, but is not conditioned (set) based on the constraint flag 832a. In a compliance bitstream, the maximum transform block size enabled by flag 834a corresponds to the constraint set by the constraint flag 832a, e.g., has a maximum value less than or equal to the constraint flag 832a. Flag 834a is a fixed-length codeword or flag, typically 1 bit in size.

[0119] Similarly, codeword 834b indicates the maximum valid CTU size for the bitstream. Codeword 834b is constrained based on the corresponding constraint codeword 832b, but is not conditioned (set) based on the constraint codeword 832b. In a compliant bitstream, the maximum CTU size enabled by codeword 834b corresponds to the constraint set by codeword 832b, e.g., has a maximum value less than or equal to codeword 832b. Flag 834b is a fixed-length codeword, typically 2 bits in size.

[0120] An adaptation parameter set (APS) 813 is coded before a frame using a NAL unit type named "PREFIX_APS_NUT" or after a frame (not shown) using a NAL unit type named "SUFFIX_APS_NUT." Multiple APSs may be included between frames in the bitstream 800 (not shown). Each APS (e.g., 813) contains parameters for configuring one of three coding tools: a scaling list, ALF filter parameters, and LMCS model parameters. Which of the three coding tools is configured in a given APS is specified by the "aps_params_type" codeword included in the respective APS. The aps_params_type codeword uses a 3-bit fixed-length codeword and has values ​​0 through 2 for the three aforementioned tools, with values ​​3 through 7 reserved for future use. The syntax element aps_params_type equal to 0 indicates that the APS is named "ALF_APS" and contains parameters for the adaptive loop filter, aps_params_type equal to 1 indicates that the APS is named "LMCS_APS" and contains parameters for the luma model chroma scaling tool, and aps_params_type equal to 2 indicates that the APS is named "SCALING_APS" and contains parameters for the scaling list.

[0121] A sequence of slices that form one picture is known as an access unit (AU), such as AU 0 814. AU 0 814 contains three slices, such as slices 0-2. Slice 1 is marked as 816. Like the other slices, slice 1 (816) contains a slice header 818 and slice data 820.

[0122] FIG. 9 shows a syntax structure 900 of slice data 820 of bitstream 801 (e.g., 115 or 133). CTU 910 includes one or more CUs, an example of which is shown as CU 914. The size of each CTU is set by codeword 834b, and its value is constrained by constraint 832b. CU 914 includes a signaled prediction mode (not shown) followed by a transform tree 916. If the size of CU 914 does not exceed the maximum transform size (32 or 64 points horizontally and vertically), the transform tree 916 includes one transform unit. If the size of a CU, e.g., CU 914, exceeds the maximum transform size (32 or 64 points horizontally and vertically), the transform tree 916 includes multiple TUs, spatially arranged in tiles and stored sequentially in the bitstream, e.g., as shown as TUs 918a and 918b.

[0123] FIG. 10 illustrates a method 1000 for encoding frame data 113 into a bitstream 115, where the bitstream 115 includes one or more slices as a sequence of coding tree units. The method 1000 may be implemented by an apparatus such as a configured FPGA, ASIC, or ASSP. Furthermore, the method 1000 may be executed by the video encoder 114 under the execution of the processor 205. For frame encoding workloads, the steps of the method 1000 may be executed on different processors to share the workload, e.g., using modern multi-core processors, such that different slices are encoded by different processors. The generated bitstream 115 may conform to a subprofile that includes constraints on aspects of the block structure, including maximum transform size and maximum CTU size. The method 1000 may be stored in a computer-readable storage medium and / or memory 206.

[0124] Method 1000 begins with a coding constraint parameters step 1010. In step 1010, video encoder 114 encodes a profile_tier_level syntax structure 830, which includes a general_constraint_info syntax structure 832, into either VPS 810 or SPS 812 in bitstream 115 as a sequence of fixed-length coding parameters. Constraints on maximum transform size (832a) and maximum CTU size (832b) are coded as part of the general_constraint_info syntax structure 832 and contribute to the definition of a sub-profile of bitstream 115. Constraints on the use of scaling lists are coded as part of the general_constraint_info syntax structure 832 with flag 832c and also contribute to the definition of a sub-profile of bitstream 115.

[0125] Method 1000 proceeds from step 1010 to a coded block structure parameter step 1015. In step 1015, video encoder 114 encodes the selected maximum transform size of the bitstream as flag 834a in SPS 812 and encodes the selected CTU size of bitstream 115 as codeword 834b in SPS 812. Flag 834a and codeword 834b are coded regardless of the value of the corresponding constraint flag, i.e., 832a and 832b, respectively. However, flag 834a and codeword 834b are prohibited from indicating a higher capability (larger transform size or larger CTU size) than that constrained by flag 832a and codeword 832b. In step 1015, video encoder 114 codes the use of a scaling list by coding flag 834c, and codes flag 834c, which occurs regardless of the value of the corresponding constraint flag 832c. However, the valid flag 834c is constrained by the constraint flag 832c such that the scaling list may not be valid due to the flag 834c if its use is prohibited by the constraint flag 832c.

[0126] From step 1015, method 1000 continues with step 1020, which involves dividing the frame into slices. In performing step 1020, processor 205 divides frame data 113 into one or more slices or consecutive portions. If parallelism is desired, a separate instance of video encoder 114 encodes each slice somewhat independently. A single video encoder 114 may process each slice sequentially, or some intermediate degree of parallelism may be implemented. Generally, the division of a frame into slices (consecutive portions) is aligned with boundaries of the division of the frame into regions known as "subpictures," tiles, or the like.

[0127] The method 1000 continues from step 1020 with a slice header encoding step 1030. In step 1030, the entropy encoder 338 encodes the slice header 818 into the bitstream 115.

[0128] Method 1000 continues from step 1030 with step 1040, which involves dividing the slice into CTUs. In performing step 1040, video encoder 114 divides slice 816, for example, into a sequence of CTUs. Slice boundaries are aligned with CTU boundaries, and CTUs within a slice are ordered according to a CTU scan order, typically a raster scan order. The division of a slice into CTUs establishes which portions of frame data 113 should be processed by video encoder 113 when encoding the current slice.

[0129] Method 1000 continues from step 1040 with step 1050 of determining a coding tree. In step 1050, video encoder 114 determines a coding tree for a currently selected CTU in a slice. Method 1000 starts with the first CTU in the slice 816 in the first invocation of step 1050 and proceeds to subsequent CTUs in the slice 816 in subsequent invocations. In determining the coding tree for a CTU, various combinations of quadtree, binary, and ternary splits are generated and tested by block partitioner 310.

[0130] Method 1000 continues from step 1050 to step 1060 of determining a coding tree. In step 1060, video encoder 114 performs, using known methods, to determine an encoding for the CU resulting from the various coding trees under evaluation. Determining the encoding involves, for example, determining a prediction mode (e.g., intra prediction with a specific mode or inter prediction with motion vectors) and a transform selection (primary transform type and optional secondary transform type) based on coding cost. If the primary transform type for the luma TB is determined to be DCT-2 or any quantized primary transform coefficients not subject to a forward secondary transform are significant, the secondary transform index for the luma TB may indicate application of a secondary transform. Otherwise, the secondary transform index for the luma indicates bypass of the secondary transform. For the luma channel, the primary transform type is determined to be one of DCT-2, transform skip, or an MTS option for the chroma channels, with DCT-2 being an available transform type. In determining the individual coding units, the optimal coding tree is also determined in a joint manner. When encoding a coding unit using intra prediction, a luma intra prediction mode and a chrominance intra prediction mode are determined.

[0131] Method 1000 continues from step 1060 with step 1070 of encoding the coding unit. In step 1070, video encoder 114 encodes the determined coding unit of step 1060 into bitstream 115.

[0132] Method 1000 continues from step 1070 to step 1080 of testing for the last coding unit. In step 1080, processor 205 tests whether the current coding unit is the last coding unit in a CTU. If not ("NO" at step 1080), control in processor 205 proceeds to step 1060 of determining the coding unit. Otherwise, if the current coding unit is the last coding unit ("YES" at step 1080), control in processor 205 proceeds to step 1090 of testing for the last CTU.

[0133] In step 1090 of testing the last CTU, processor 205 tests whether the current CTU is the last CTU in slice 816. If it is not the last CTU in slice 816 ("NO" in step 1090), control in processor 205 returns to step 1050 of determining the coding tree. Otherwise, if the current CTU is the last ("YES" in step 1090), control in processor 205 proceeds to step 10100 of testing the last slice.

[0134] In last slice testing step 10100, processor 205 tests whether the current slice being coded is the last slice in the frame. If it is not the last slice ("NO" in step 10100), control in processor 205 passes to slice header coding step 1030. Otherwise, if the current slice is the last and all slices (consecutive portions) have been coded ("YES" in step 10100), method 1000 ends.

[0135] 11 shows a method 1100 for decoding a frame from a bitstream as a sequence of coding units arranged into slices. Method 1100 may be implemented by an apparatus such as an configured FPGA, ASIC, or ASSP. Furthermore, method 1100 may be performed by video decoder 134 under execution by processor 205. Accordingly, method 1100 may be stored in a computer-readable storage medium and / or memory 206.

[0136] Method 1100 decodes a bitstream encoded using method 1000 in which a subprofile of bitstream 115 includes constraints on block sizes, such as a maximum transform size and a maximum CTU size. The maximum transform size applies to luma and chroma, regardless of the chroma format of the video data. The CTU size indicates the area covered by the root node of the coding tree, which always applies to both the luma and chroma channels, regardless of subsequent splitting of CTUs into separate coding trees for luma and chroma that may occur deeper in the coding tree.

[0137] Method 1100 begins with step 1110, which involves decoding constraint parameters. In performing step 1110, video decoder 134 determines the sub-profile of bitstream 133 by decoding a general_constraint_info syntax structure 832 from either VPS 810 or SPS 812, which are present in bitstream 133 as a sequence of fixed-length parameters. Constraints on maximum transform size (832a) and maximum CTU size (832b) are decoded as part of general_constraint_info syntax structure 832 and contribute to determining the sub-profile of bitstream 133. Step 1110 may, for example, decode a maximum transform block size constraint and / or a maximum CTU size constraint from the bitstream. Constraints on the use of scaling lists are decoded as part of general_constraint_info syntax structure 832 with flag 832c and also contribute to defining the sub-profile of bitstream 133.

[0138] Method 1100 proceeds from step 1110 to step 1120, where it decodes block structure parameters. In step 1120, video decoder 134 decodes the selected maximum valid transform size of the bitstream as flag 834a from SPS 812 and the selected maximum valid CTU size of bitstream 115 as codeword 834b from SPS 812. Flag 834a and codeword 834b are decoded regardless of the corresponding constraint flag, i.e., 832a and 832b, respectively. However, flag 834a and codeword 834b are prohibited (in a "conforming" bitstream) from indicating a higher capability (larger transform size or larger CTU size) than that constrained by flag 832a and codeword 832b. In step 1125, video decoder 134 decodes the use of scaling lists by decoding flag 834c, and the decoding of flag 834c occurs regardless of the value of the corresponding constraint flag 832c. However, the enable flag 834c is constrained by the constraint flag 832c such that the scaling list may not be enabled by the flag 834c if its use is prohibited by the constraint flag 832c. If the prohibitions defined in the general_constraint_info syntax structure 832 are violated by the decoded block structure parameters, the bitstream may be considered "non-conforming" by the video decoder 134 and further decoding may be terminated.

[0139] The method 1100 continues from step 1120 with step 1130 of decoding the slice header. In step 1130, the entropy decoder 420 decodes the slice header 818 from the bitstream 133.

[0140] Method 1100 continues from step 1130 with step 1140 of dividing the slice into CTUs. In step 1140, video decoder 134 divides slice 816 into a sequence of CTUs. Slice boundaries are aligned with CTU boundaries, and CTUs within the slice are ordered according to a CTU scan order, which is typically a raster scan order. The division of a slice into CTUs establishes which portions of frame data 113 should be processed by video decoder 134 when decoding the current slice. Slices are divided into CTUs based on the largest available decoded CTU size.

[0141] Method 1100 continues from step 1140 with step 1150 of decoding the coding tree. In performing step 1150, video decoder 133 decodes the coding tree of the current CTU in the slice from bitstream 133, starting with the first CTU in slice 816 in the first invocation of step 1150. The coding tree of the CTU is decoded by decoding the split flag in entropy decoder 420 according to FIG. 6 and based on the maximum valid CTU size. In subsequent iterations of step 1150 for the CTU, decoding is performed for subsequent CTUs in slice 816.

[0142] Method 1100 continues from step 1160 with step 1170 of decoding the coding units. In step 1170, video decoder 134 decodes the coding units from bitstream 133. Each coding unit is decoded or determined from a corresponding CTU, and the CTU is determined according to the decoded maximum valid coding tree size and split flags decoded from the bitstream.

[0143] Method 1100 continues from step 1110 to step 1180 of testing for the last coding unit. In step 1180, processor 205 tests whether the current coding unit is the last coding unit in the CTU. If it is not the last coding unit (“NO” at step 1180), control in processor 205 returns to step 1170 of decoding a coding unit to decode the next coding unit in the coding tree unit. If the current coding unit is the last coding unit (“YES” at step 1180), control in processor 205 proceeds to step 1190 of testing for the last CTU.

[0144] In test last CTU step 1190, processor 205 tests whether the current CTU is the last CTU in slice 816. If not, and it is the last CTU in the slice (“NO” at step 1190), control of processor 205 returns to decoding coding tree step 1150 to decode the next coding tree unit in slice 816. If the current CTU is the last CTU in slice 816 (“YES” at step 1190), control in processor 205 proceeds to test last slice step 11100.

[0145] In last slice testing step 11100, processor 205 tests whether the current slice being decoded is the last slice in the frame. If it is not the last slice in the frame ("NO" at step 11100), control of processor 205 returns to slice header decoding step 1130, which operates to decode the slice header of the next slice in the frame (e.g., "Slice 2" in FIG. 11). If the current slice is the last slice in the frame ("YES" at step 1100), method 1100 ends.

[0146] Steps 1140 through 11100 operate to decode an image frame by determining transform blocks for each of the coding units of the coding units according to a decoded maximum valid transform block size and / or a decoded maximum valid CTU size and a split flag decoded from the bitstream. Operation of method 1610 for multiple coding units operates as described with respect to device 130 of FIG. 1 to generate an image frame.

[0147] The configurations of the video encoder 114 using method 1000 and the video decoder 134 using method 1100 can support sub-profile definitions with granularity that include block structure aspects, i.e., maximum transform size and CTU size (corresponding to maximum CU size). Controlling block structure aspects using maximum transform size constraints and / or maximum CTU size constraints means that sub-profiles provide granularity of control over standard block structure behavior that affects all color channels, regardless of the chroma format used. Constraining the maximum transform size and / or maximum CTU size using a general constraint related to a set of sequence-level tools allows early knowledge in decoding to determine which tools are needed. The constraint flags are placed in a fixed position relative to the start of the VPS 810 or SPS 812, so that the profile and sub-profile of a bitstream can be determined without having to perform variable-length decoding. Furthermore, implementations or tools that are known to be problematic can be disabled without affecting other aspects of the implementation, e.g., other coding tools, and without the creation of ad hoc or non-standard sub-profiles. Thus, vendors implementing the VVC standard have more flexibility in implementing video encoders and decoders that are suitable and adaptable for real-world applications and implementations in their products.

[0148] In the configuration of the video encoder 114 and the video decoder 134, the maximum coding unit size is constrained using a 1-bit flag that limits the CTU size to 64x64 when active and allows a CTU size of 128x128 when inactive.

[0149] In another configuration of the video encoder 114 and the video decoder 134, a "no_scaling_list_constraint_flag" (832c) is also present in the general_constraint_info() 832 and is encoded by the video encoder 114 in step 1110 and decoded by the video decoder 134 in step 1110. The video encoder 114 encodes the sps_explicit_scaling_list_enabled_flag (834c) into the SPS 812 to indicate whether scaling lists should be used in quantization / dequantization. The video decoder 134 parses the sps_explicit_scaling_list_enabled_flag 834c from the SPS 812 to determine whether the dequantization should utilize scaling lists in the dequantization performed by the inverse quantizer module 428. When no_scaling_list_constraint_flag 832c is active (e.g., has a value equal to 1), the sps_explicit_scaling_list_enabled_flag 834c encoded in SPS 812 indicates that a scaling list is not in use (value equal to 0). When no_scaling_list_constraint_flag 832c indicates that a scaling list is not in use (value equal to 1), the aps_params_type is prohibited from having the value 2 ("SCALING_APS"). When no_alf_constraint_flag encoded in general_constraint_info() 832 indicates that an adaptive loop filter is not in use (value equal to 1), the aps_params_type of any APS (e.g., APS 813) associated with the bitstream is prohibited from having the value 0 ("ALF_APS").When the no_lmcs_constraint_flag coded in general_constraint_info() 832 indicates that luma model chroma scaling should not be used (value equal to 1), the aps_params_type of any APS associated with the bitstream (e.g., APS 813) is prohibited from having the value 1 ("LMCS_APS").

[0150] Industrial Applicability The described arrangements are applicable to the computer and data processing industry, particularly in digital signal processing for decoding and encoding signals such as video and image signals, to achieve high compression efficiency. Providing one or more of the constraint flags described above allows for the selection of a subset of tools for a given profile ("sub-profiling"). The selection of a subset of tools provides several advantages, such as VVC vendor implementation advantages, since the vendor can specify a subset of a profile that excludes unnecessary or otherwise problematic coding tools, for example, from a complexity standpoint.

[0151] The above describes only some embodiments of the present invention, and modifications and / or variations can be made to the present invention without departing from the scope and spirit of the invention; the embodiments are illustrative and not limiting.

[0152] Appendix A. 1 is an example of working draft text for the VVC standard adapted to correspond to the methods described herein.

[0153] 7.3.3.2 General constraint information syntax

[0154] [Table 1]

[0155] [Table 2]

[0156] 7.4.4.2 General Constraint Information Semantics ... max_log2_ctu_size_constraint_idc specifies that sps_log2_ctu_size_minus5 is in the range of 0 to max_log2_ctu_size_constraint_idc.

[0157] no_luma_transform_size_64_constraint_flag equal to 1 specifies that sps_max_luma_transform_size_64_flag is equal to 0. no_luma_transform_size_64_constraint_flag equal to 0 imposes no such constraint. no_scaling_list_constraint_flag equal to 1 specifies that sps_explicit_scaling_list_enabled_flag is equal to 0 and aps_params_type is not equal to 2. no_scaling_list_constraint_flag equal to 0 imposes no such constraint. ... no_alf_constraint_flag equal to 1 specifies that sps_alf_enabled_flag is equal to 0 and aps_params_type is not equal to 0. no_alf_constraint_flag equal to 0 imposes no such constraint. ... no_lmcs_constraint_flag equal to 1 specifies that sps_lmcs_enabled_flag is equal to 0 and aps_params_type is not equal to 0. no_lmcs_constraint_flag equal to 0 imposes no such constraint.

[0158] The claims defining the invention are as follows:

Claims

1. 1. A method for decoding an image frame from a bitstream, comprising: dividing the image frame into a plurality of coding tree units, each of the plurality of coding tree units being divided into one or more transform blocks; decoding a first flag for a maximum transform block size constraint from a syntax structure for constraint information in a profile_tier_level syntax structure in the bitstream; decoding a second flag included in a sequence parameter set in the bitstream regardless of the value of the first flag, the second flag being a flag for a maximum transform block size; determining the one or more transform blocks for each of the plurality of coding tree units according to the second flag for the maximum transform block size and a split flag decoded from the bitstream; decoding each of the determined one or more transform blocks from the bitstream to decode the image frame; decoding first information for a coding tree unit size constraint from a syntax structure for constraint information in a profile_tier_level syntax structure in the bitstream; decoding second information indicating a value for specifying a coding tree unit size from a sequence parameter set in the bitstream; determining one or more coding units for each of the plurality of coding tree units according to the second information for the coding tree unit size and a split flag decoded from the bitstream; decoding each of the determined one or more coding units from the bitstream to decode the image frame; Including, If the maximum transform block size is not constrained by the first flag, the second flag may indicate that the maximum transform block size is 64; if the maximum transform block size is constrained by the first flag, then the maximum transform block size is constrained to 32; the value of the second information ranges from 0 to an integer value based on the first information; The sequence parameter set may include the profile_tier_level syntax structure. A method characterized by:

2. 2. The method of claim 1, wherein a 1-bit fixed length codeword is used for the first flag for the maximum transform block size constraint.

3. 2. The method of claim 1, wherein a 1-bit fixed length codeword is used for the second flag for the maximum transform block size.

4. 2. The method of claim 1, wherein the second flag is sps_max_luma_transform_size_64_flag.

5. If the value of the second information is 0, the coding tree unit size is 32; If the value of the second information is 1, the coding tree unit size is 64; The method of claim 1 , wherein when the value of the second information is 2, the coding tree unit size is 128.

6. The method of claim 1 , wherein the second information is sps_log2_ctu_size_minus5.

7. The method of claim 1, characterized in that if the value of the first flag is 1, the value of the second flag is signaled as 0.

8. 1. A video decoding apparatus for decoding an image frame from a bitstream, the image frame being divided into a plurality of coding tree units, each of the plurality of coding tree units being divided into one or more transform blocks; means for decoding a first flag for a maximum transform block size constraint from a syntax structure for constraint information in a profile_tier_level syntax structure in the bitstream; means for decoding a second flag, which is included in a sequence parameter set in the bitstream regardless of the value of the first flag and is a flag for a maximum transform block size; means for determining the one or more transform blocks for each of the plurality of coding tree units according to the second flag for the maximum transform block size and a split flag decoded from the bitstream; means for decoding each of the determined one or more transform blocks from the bitstream to decode the image frame; means for decoding first information for coding tree unit size constraints from a syntax structure for constraint information in a profile_tier_level syntax structure in the bitstream; means for decoding second information indicating a value for specifying a coding tree unit size from a sequence parameter set in the bitstream; means for determining one or more coding units for each of the plurality of coding tree units according to the second information for the coding tree unit size and a split flag decoded from the bitstream; means for decoding each of the determined one or more coding units from the bitstream to decode the image frame; Equipped with If the maximum transform block size is not constrained by the first flag, the second flag may indicate that the maximum transform block size is 64; if the maximum transform block size is constrained by the first flag, then the maximum transform block size is constrained to 32; the value of the second information ranges from 0 to an integer value based on the first information; The sequence parameter set may include the profile_tier_level syntax structure.

1. A video decoding device comprising:

9. 1. A method for encoding an image frame into a bitstream, comprising: dividing the image frame into a plurality of coding tree units, each of the plurality of coding tree units being divided into one or more transform blocks; encoding a first flag for a maximum transform block size constraint in a syntax structure for constraint information in a profile_tier_level syntax structure in the bitstream; encoding a second flag for a maximum transform block size in a sequence parameter set in the bitstream regardless of the value of the first flag; encoding each of the one or more transform blocks; encoding first information for a coding tree unit size constraint into a syntax structure for constraint information in a profile_tier_level syntax structure in the bitstream; encoding second information for a coding tree unit size into a sequence parameter set in the bitstream; Each of the plurality of coding tree units is divided into one or more coding units, and each of the one or more coding units is encoded. Including, If the maximum transform block size is not constrained by the first flag, the second flag may indicate that the maximum transform block size is 64; if the maximum transform block size is constrained by the first flag, then the maximum transform block size is constrained to 32; the value of the second information ranges from 0 to an integer value based on the first information; The sequence parameter set may include the profile_tier_level syntax structure. A method characterized by:

10. 10. The method of claim 9, wherein a 1-bit fixed length codeword is used for the first flag for the maximum transform block size constraint.

11. 10. The method of claim 9, wherein a 1-bit fixed length codeword is used for the second flag for the maximum transform block size.

12. 10. The method of claim 9, wherein the second flag is sps_max_luma_transform_size_64_flag.

13. If the value of the second information is 0, the coding tree unit size is 32; If the value of the second information is 1, the coding tree unit size is 64; The method of claim 9 , wherein when the value of the second information is 2, the coding tree unit size is 128.

14. The method of claim 9 , wherein the second information is sps_log2_ctu_size_minus5.

15. The method of claim 9, wherein if the value of the first flag is 1, the value of the second flag is signaled as 0.

16. 1. A video encoding apparatus for encoding an image frame into a bitstream, the image frame being divided into a plurality of coding tree units, each of the plurality of coding tree units being divided into one or more transform blocks; means for encoding a first flag for a maximum transform block size constraint into a syntax structure for constraint information in a profile_tier_level syntax structure in the bitstream; means for encoding a second flag for a maximum transform block size into a sequence parameter set in the bitstream regardless of the value of the first flag; means for encoding each of the one or more transform blocks; means for encoding first information for a coding tree unit size constraint into a syntax structure for constraint information in a profile_tier_level syntax structure in the bitstream; means for encoding second information for a coding tree unit size into a sequence parameter set in the bitstream; each of the plurality of coding tree units is divided into one or more coding units, and means for encoding each of the one or more coding units; Including, If the maximum transform block size is not constrained by the first flag, the second flag may indicate that the maximum transform block size is 64; if the maximum transform block size is constrained by the first flag, then the maximum transform block size is constrained to 32; the value of the second information ranges from 0 to an integer value based on the first information; The sequence parameter set may include the profile_tier_level syntax structure.

1. A video encoding device comprising:

17. A computer program product for causing a computer to carry out the method according to any one of claims 1 to 7.

18. A computer program product for causing a computer to carry out the method according to any one of claims 9 to 15.

Images