Method, apparatus and system for encoding and decoding a block of video samples

Abstract

A system and method for decoding an image frame from a bitstream, the image frame being segmented into multiple coding tree units. The method includes: decoding a maximum transform block size constraint and / or a maximum coding tree unit (CTU) size constraint from the bitstream; and decoding a maximum enabled transform block size and / or a maximum enabled CTU size from the bitstream. The decoded maximum enabled transform block size is less than or equal to the decoded maximum transform block size constraint. The decoded maximum enabled CTU size is less than or equal to the decoded maximum CTU size constraint. The method operates to determine transform blocks in one or more transform blocks of each coding tree unit in the plurality of coding tree units based on the decoded maximum enabled transform block size, the maximum enabled CTU, and a splitting flag decoded from the bitstream; and to decode the determined transform blocks in one or more transform blocks from the bitstream to decode the image frame.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Australian Patent Application 2020203330, filed on 21 May 2020, pursuant to 35 U.SC §119, the entire contents of which are incorporated herein by reference for all purposes, as if fully set forth herein. Technical Field

[0003] This invention generally relates to digital video signal processing, and more particularly to methods, apparatus, and systems for encoding and decoding video sample blocks. The invention also relates to computer program products comprising a computer-readable medium containing computer programs for encoding and decoding video sample blocks. Background Technology

[0004] Currently, there are numerous video coding applications, including those for transmitting and storing video data. Many video coding standards have been developed, and others are currently under development. Recent advancements in video coding standardization have led to the formation of a group known as the Joint Video Experts Group (JVET). The JVET comprises members from two Standards Setting Organizations (SSOs): Study Group 16, Question 6 (SG16 / Q6) of the Telecommunication Standardization Sector (ITU-T) of the International Telecommunication Union (ITU), also known as the Video Coding Experts Group (VCEG); and Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the Moving Picture Experts Group (MPEG).

[0005] The Joint Video Experts Group (JVET) issued a Call for Proposals (CfP) and analyzed the responses at its 10th meeting in San Diego, USA. The submitted responses demonstrated video compression capabilities significantly superior to those of the current state-of-the-art video compression standard, namely, "High Efficiency Video Coding" (HEVC). Based on this superior performance, a project was initiated to develop a new video compression standard named "Various Video Coding" (VVC). VVC is expected to address the ongoing demand for even higher compression performance, particularly as video formats become more capable (e.g., with higher resolution and higher frame rates), and the growing market demand for services delivered over WANs (where bandwidth costs are relatively high). VVC must be implementable in contemporary silicon processes and offer an acceptable trade-off between the achieved performance and implementation cost. Implementation cost can be considered in one or more aspects, such as silicon area, CPU processor load, memory utilization, and bandwidth. Part of the diversity of the VVC standard lies in the wide selection of tools available for compressing video data and the broad range of applications suitable for VVC.

[0006] Video data consists of a sequence of frames of image data, each frame containing one or more color channels. Typically, one primary color channel and two secondary color channels are required. The primary color channel is usually referred to as the "luminance" channel, and the (one or more) secondary color channels are usually referred to as the "chrominance" channels. Although video data is typically displayed in the RGB (Red-Green-Blue) color space, this color space is highly correlated between its three corresponding components. The video data representation seen by the encoder or decoder typically uses a color space such as YCbCr. YCbCr concentrates the luminance (mapped to "luminance" according to the transfer function) in the Y (primary) channel and the chrominance in the Cb and Cr (secondary) channels. Due to the use of decorrelated YCbCr signals, the statistics for the luminance channel are significantly different from those for the chrominance channel. The main difference is that, after quantization, the chrominance channel contains relatively fewer effective coefficients for a given block compared to the coefficients for the corresponding luminance channel block. Furthermore, the Cb and Cr channels can be spatially sampled at a lower rate (subsampling) compared to the luminance channel (e.g., half horizontally and half vertically (known as the "4:2:0 chroma format")). The 4:2:0 chroma format is commonly used in applications such as internet video streaming, broadcast television, and Blu-ray. TMUsed in "consumer" applications (such as on-disk storage). Subsampling the Cb and Cr channels horizontally at half rate instead of vertically is called the "4:2:2 chroma format". The 4:2:2 chroma format is commonly used in professional applications (including capturing scripts for filmmaking, etc.). The higher sampling rate of the 4:2:2 chroma format makes the resulting video more flexible for editing operations such as color grading. Before being distributed to consumers, 4:2:2 chroma format material is usually converted to 4:2:0 chroma format and then encoded for distribution. In addition to chroma format, video is also characterized by resolution and frame rate. Example resolutions are Ultra High Definition (UHD) of 3840×2160 or "8K" of 7680×4320, and example frame rates are 60 or 120Hz. Luminance sampling rates can range from approximately 500 megasamples per second to several gigabits per second. For the 4:2:0 chroma format, the sampling rate of each chroma channel is one-quarter of the luminance sampling rate, and for the 4:2:2 chroma format, the sampling rate of each chroma channel is half of the luminance sampling rate.

[0007] The VVC standard is a block-based codec, where frames are first divided into an array of square regions called "Code Tree Units" (CTUs). CTUs typically occupy relatively large areas, such as 128×128 luma samples. However, the area of ​​CTUs at the right and bottom edges of individual frames may be smaller. Associated with each CTU is a "coding tree" ("shared tree") for both luma and chroma channels, or separate trees for each channel. The coding tree defines the breakdown of the CTU's regions into sets of blocks (also called "coded blocks" (CBs)). When using a shared tree, a single coding tree specifies blocks for both luma and chroma channels; in this case, the set of juxtaposed coding blocks is called a "Code Unit" (CU), meaning each CU has coding blocks for its respective color channel. CBs are processed in a specific order for encoding or decoding. Because of the use of the 4:2:0 chroma format, a CTU with a luma coding tree for a 128×128 luma sample region has a corresponding chroma coding tree for a 64×64 chroma sample region (juxtaposed with the 128×128 luma sample region). When a single coding tree is used for both luma and chroma channels, the set of juxtaposed blocks for a given region is typically referred to as a “unit,” such as the CU mentioned above, as well as “prediction units” (PUs) and “transform units” (TUs). A single tree with CUs spanning the color channels of 4:2:0 chroma format video data produces chroma blocks that are half the width and half the height of the corresponding luma blocks. When a separate coding tree is used for a given region, the CBs mentioned above, as well as “prediction blocks” (PBs) and “transform blocks” (TBs), are used.

[0008] Despite the aforementioned distinction between “cell” and “block”, the term “block” can be used as a general term for a frame area or region for which operations are applied across all color channels.

[0009] For each CU, a prediction unit (PU) (“prediction unit”) is generated for the content (sample values) of the corresponding region of the generated frame data. Furthermore, a representation of the difference (or “residual” in the spatial domain) between the prediction seen at the encoder input and the region content is formed. The differences of the color channels can be transformed and encoded into a sequence of residual coefficients, thus forming one or more TUs for a given CU. The applied transformation can be a Discrete Cosine Transform (DCT) or other transformation applied to individual blocks of residual values. The transformation is applied separately, i.e., the two-dimensional transformation is performed in two passes. First, the block is transformed by applying a one-dimensional transformation to each row of samples in the block. Then, a portion of the result is transformed by applying a one-dimensional transformation to each column of the partial result to produce a final block of transform coefficients that are essentially decorrelated on the residual samples. The VVC standard supports transforms of various sizes, including transforms of rectangular blocks (each side a power of 2). The transform coefficients are quantized for entropy encoding into the bitstream.

[0010] VVC is characterized by intra-frame prediction and inter-frame prediction. Intra-frame prediction involves generating a prediction for the current sample block in the frame using previously processed samples from the frame being used. Inter-frame prediction involves generating a prediction for the current sample block in the frame using sample blocks obtained from previously decoded frames. Sample blocks obtained from previously decoded frames are offset from the spatial location of the current block according to a motion vector, which has typically been filtered. Intra-frame predicted blocks can be (i) uniform sample values ​​(“DC intra-frame prediction”), (ii) planes with offsets as well as horizontal and vertical gradients (“planar intra-frame prediction”), (iii) a group of blocks with neighboring samples applied in a specific direction (“angular intra-frame prediction”), or (iv) the result of matrix multiplication using neighboring samples and selected matrix coefficients. Further discrepancies between the predicted block and the corresponding input samples can be corrected to some extent by encoding the 'residual' into the bitstream. The residual is typically transformed from the spatial domain to the frequency domain to form residual coefficients (in the “primary transform domain”), and the residual coefficients can be further transformed by applying a “secondary transform” (to produce residual coefficients in the “secondary transform domain”). Quantizing the residual coefficients according to the quantization parameters leads to a loss of accuracy in the reconstruction of the samples generated at the decoder, while the bit rate within the bit stream is also reduced.

[0011] Intra-Block Copy (IBC) mode allows the use of sample blocks from the same frame to generate block predictions. An alternative Matrix Intra-Frame Prediction (MIP) mode is available, which uses a predetermined vector and matrix multiplication of adjacent samples of the block to generate the predicted block. Blocks can be color-coded instead of using a transform. The three color channels of the block can be generated using an Adaptive Color Transform (ACT), typically decorrelated from the RGB color space to the YCbCr color space. The Cb and Cr channel residuals can be jointly encoded (JCbCr). A choice of primary transform (MTS) is available, including DCT-2, DCT-8, and DST-7, as well as the Transform Skip (TS) mode. Optional secondary transforms are also available, whereby a kernel from a set of kernels is selected based on the intra-frame prediction mode and the block size, and this kernel is suitable for the low-frequency region of the transformed block (called Low-Frequency Inseparable Transform (LFNST)). The residuals can be quantized and encoded using a grid scheme called Dependent Quantization (DQ). Inter-frame predicted blocks can be predicted as a set of sub-blocks, each with a different motion vector derived from an affine motion model. Blocks can be generated by combining (uniformly) mixing intra-prediction and inter-prediction operations. Blocks can also be generated using geometrically oriented mixing (CIIP) of two different inter-prediction blocks. Luminance mapping with chroma scaling (LMCS) processing can be applied as part of the decoding process, mapping luminance samples to specific values, and scaling operations can be applied to the values ​​of chroma samples. An adaptive loop filter (ALF) can be applied, thereby spatially filtering the luminance and chroma samples using one of several filters sent in the bitstream, primarily for smoothing purposes. The residual coefficients of the transform block can be dequantized in a non-uniform manner based on the (spatial) scaling matrix (or "scaling list") applied in combination with the quantization parameters of the transform block. Summary of the Invention

[0012] The object of the present invention is to substantially overcome or at least improve one or more disadvantages of the existing arrangement.

[0013] A first aspect of the present invention provides a method for decoding an image frame from a bitstream, the image frame being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more transform blocks, the method comprising: decoding a maximum transform block size constraint from the bitstream; decoding a maximum enabled transform block size from the bitstream, wherein the decoded maximum enabled transform block size is less than or equal to the decoded maximum transform block size constraint; determining one or more transform blocks of each coding tree unit in the plurality of coding tree units based on the decoded maximum enabled transform block size and a split flag decoded from the bitstream; and decoding each transform block in the determined one or more transform blocks from the bitstream to decode the image frame.

[0014] According to another aspect, the maximum transform block size constraint uses a fixed-length codeword of 1 bit.

[0015] According to another aspect, the maximum enabled transform block size constraint is decoded from the general_constraint_info syntax structure in the bit stream.

[0016] According to another aspect, the maximum transform block size constraint is decoded from one of the video parameter set and sequence parameter set of the bitstream.

[0017] According to another aspect, the maximum enabled transform block size uses a fixed-length codeword of 1 bit.

[0018] According to another aspect, the maximum enabled transform block size is decoded from the seq_parameter_set_rbsp() syntax structure in the bit stream.

[0019] Another aspect of the present invention provides a method for decoding an image frame from a bitstream, the image frame being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more coding units, the method comprising: decoding a maximum coding tree unit size constraint from the bitstream; decoding a maximum enabled coding tree unit size from the bitstream, wherein the decoded maximum enabled coding tree unit size is less than or equal to the decoded maximum coding unit size constraint; determining one or more coding units of each coding tree unit among the plurality of coding tree units based on the decoded maximum enabled coding tree size and a split flag decoded from the bitstream; and decoding each coding unit of the determined one or more coding units from the bitstream to decode the image frame.

[0020] According to another aspect, the maximum coding tree unit size constraint uses a fixed-length codeword of 1 bit.

[0021] According to another aspect, the maximum enabled coding tree unit size constraint is decoded from the general_constraint_info syntax structure in the bit stream.

[0022] According to another aspect, the maximum coding tree unit size constraint is decoded from one of the video parameter set and sequence parameter set of the bitstream.

[0023] According to another aspect, the maximum enabled coding tree unit size uses a fixed-length codeword of 1 bit.

[0024] According to another aspect, the maximum enabled coding tree unit size is decoded from the seq_parameter_set_rbsp() syntax structure in the bit stream.

[0025] Another aspect of the present invention provides a non-transitory computer-readable medium having a computer program stored thereon for implementing a method for decoding an image frame from a bitstream, the image frame being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more transform blocks, the method comprising: decoding a maximum transform block size constraint from the bitstream; decoding a maximum enabled transform block size from the bitstream, wherein the decoded maximum enabled transform block size is less than or equal to the decoded maximum transform block size constraint; determining one or more transform blocks of each coding tree unit in the plurality of coding tree units based on the decoded maximum enabled transform block size and a split flag decoded from the bitstream; and decoding each transform block in the determined one or more transform blocks from the bitstream to decode the image frame.

[0026] Another aspect of the invention provides a video decoder configured to implement a method for decoding an image frame from a bitstream, the image frame being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more coding units, the method comprising: decoding a maximum coding tree unit size constraint from the bitstream; decoding a maximum enabled coding tree unit size from the bitstream, the decoded maximum enabled coding tree unit size being less than or equal to the decoded maximum coding unit size constraint; determining one or more coding units of each coding tree unit among the plurality of coding tree units based on the decoded maximum enabled coding tree size and a split flag decoded from the bitstream; and decoding each coding unit of the determined one or more coding units from the bitstream to decode the image frame.

[0027] Another aspect of the present invention provides a system comprising: a memory; and a processor, wherein the processor is configured to execute code stored in the memory for implementing a method of decoding an image frame from a bitstream, the image frame being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more transform blocks, the method comprising: decoding a maximum transform block size constraint from the bitstream; decoding a maximum enabled transform block size from the bitstream, the decoded maximum enabled transform block size being less than or equal to the decoded maximum transform block size constraint; determining one or more transform blocks of each coding tree unit in the plurality of coding tree units based on the decoded maximum enabled transform block size and a split flag decoded from the bitstream; and decoding each transform block in the determined one or more transform blocks from the bitstream to decode the image frame.

[0028] Another aspect of the present invention provides a method for decoding an image frame from a bitstream, the image frame being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more coding blocks, the method comprising: decoding from the bitstream a first flag indicating a constraint on the use of a scaling list, the flag being included in a general constraint information syntax; decoding from the bitstream a second flag indicating the enabling of the scaling list, the second flag being constrained by the first flag; decoding from the bitstream the one or more coding blocks using scaling based on the value of the second flag; and decoding the image frame using the decoded one or more coding units.

[0029] Another aspect of the present invention provides a video decoding apparatus for decoding an image frame from a bitstream, the image frame being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more transform blocks, the video decoding apparatus comprising: a first decoding unit configured to decode a maximum transform block size constraint from the bitstream; a second decoding unit configured to decode a maximum enabled transform block size from the bitstream, wherein the decoded maximum enabled transform block size is less than or equal to the decoded maximum transform block size constraint; a determining unit configured to determine one or more transform blocks of each coding tree unit among the plurality of coding tree units based on the decoded maximum enabled transform block size and a split flag decoded from the bitstream; and a third decoding unit configured to decode each transform block of the determined one or more transform blocks from the bitstream to decode the image frame.

[0030] Another aspect of the present invention provides a video decoding apparatus for decoding an image frame from a bitstream, the image frame being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more coding units, the video decoding apparatus comprising: a first decoding unit configured to decode a maximum coding tree unit size constraint from the bitstream; a second decoding unit configured to decode a maximum enabled coding tree unit size from the bitstream, wherein the decoded maximum enabled coding tree unit size is less than or equal to the decoded maximum coding unit size constraint; a determining unit configured to determine one or more coding units of each coding tree unit among the plurality of coding tree units based on the decoded maximum enabled coding tree size and a splitting flag decoded from the bitstream; and a third decoding unit configured to decode the image frame using one or more decoded coding units.

[0031] Another aspect of the present invention provides a video decoding apparatus for decoding an image frame from a bitstream, the image frame being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more coding blocks, the video decoding apparatus comprising: a first decoding unit configured to decode from the bitstream a first flag indicating a constraint on the use of a scaling list, the flag being included in a general constraint information syntax; a second decoding unit configured to decode from the bitstream a second flag indicating the enabling of the scaling list, the second flag being constrained by the first flag; a third decoding unit configured to decode from the bitstream the one or more coding blocks using scaling based on the value of the second flag; and a fourth decoding unit configured to decode the image frame using the decoded one or more coding units.

[0032] Another aspect of the present invention provides a non-transitory computer-readable medium having a computer program stored thereon for implementing a method for decoding an image frame from a bitstream, the image frame being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more coding units, the method comprising: decoding a maximum coding tree unit size constraint from the bitstream; decoding a maximum enabled coding tree unit size from the bitstream, the decoded maximum enabled coding tree unit size being less than or equal to the decoded maximum coding unit size constraint; determining one or more coding units of each coding tree unit among the plurality of coding tree units based on the decoded maximum enabled coding tree size and a split flag decoded from the bitstream; and decoding each coding unit of the determined one or more coding units from the bitstream to decode the image frame.

[0033] Another aspect of the present invention provides a non-transitory computer-readable medium having a computer program stored thereon for implementing a method for decoding an image frame from a bitstream, the image frame being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more coding blocks, the method comprising: decoding from the bitstream a first flag indicating a constraint on the use of a scaling list, the flag being included in a general constraint information syntax; decoding from the bitstream a second flag indicating the enabling of the scaling list, the second flag being constrained by the first flag; decoding from the bitstream the one or more coding blocks using scaling based on the value of the second flag; and decoding the image frame using the decoded one or more coding units to decode the image frame.

[0034] Other aspects were also disclosed. Attached Figure Description

[0035] At least one embodiment of the invention will now be described with reference to the following figures and appendices, wherein:

[0036] Figure 1 This is a schematic block diagram illustrating a video encoding and decoding system;

[0037] Figure 2A and 2B Composition can be put into practice Figure 1 A schematic block diagram of a general computer system that includes one or both of the video encoding and decoding systems;

[0038] Figure 3 This is a schematic block diagram showing the functional modules of a video encoder;

[0039] Figure 4 This is a schematic block diagram illustrating the functional modules of the video decoder;

[0040] Figure 5 This is a schematic block diagram illustrating the available segmentation of a block into one or more blocks in the tree structure of general video coding;

[0041] Figure 6 This is a schematic diagram used to implement the data stream of permissive segmentation from one or more blocks in the tree structure of general video coding;

[0042] Figure 7A and 7B This illustrates an example of a coding tree unit (CTU) splitting into multiple coding units (CUs);

[0043] Figure 8 The syntactic structure of a bitstream with multiple stripes is shown, each stripe comprising multiple coding units;

[0044] Figure 9 The syntactic structure of a bitstream with coding tree units is shown.

[0045] Figure 10 This illustrates a method for encoding frames in a bitstream that includes one or more stripes as a sequence of coding units;

[0046] Figure 11 This illustrates a method for decoding frames from a bitstream that is a sequence of coding units arranged in stripes; and

[0047] Appendix A shows an example of a working draft text suitable for a VVC standard corresponding to the methods disclosed herein. Detailed Implementation

[0048] Where steps and / or features with the same reference numerals are referenced in any one or more figures, unless the contrary is intended, these steps and / or features have the same (one or more) function or (one or more) operation for the purposes of this specification.

[0049] VVC encoders and decoders include a capability signaling mechanism called “constraints.” Early in the bitstream, a set of constraints exists indicating which capabilities of the VVC standard are not used in the bitstream. Constraints are signaled along with the bitstream’s “grades” and “levels.” Grades broadly indicate which toolset is needed to decode the bitstream. Constraints also provide fine-grained control over which tools are further constrained within a given grade. Further constraints are called “subgrades.” Subgrades allow for the effective removal of specific tools from a grade even after it has been defined, where implementers agree on common subgrades to deploy. For example, if a given tool is found difficult to implement (despite SSO analysis efforts), it can be removed later. One use of subgrades is to reduce the “proliferation” of grades, some of which may never be used. Defining fewer grades helps avoid market segmentation of VVC standard implementations. Grades and subgrades also define what are called “interoperability points.” Interoperability points are the sets of tools that manufacturers agree on to be supported by their implementations. Grades are agreed upon when the standard is finalized, and subgrades can be agreed upon at a later date. This allows for addressing unforeseen implementation complexities or other issues with specific tools without adding additional grades to the standard. Specific encoders may choose to further restrict the use of encoding tools, for example, for complexity reduction purposes, without deviating from the agreed grade and subgrade definitions of the resulting bitstream. Any further restrictions selected by the tool can be signaled early in the bitstream, thus suppressing signaling for unused tools. Alternatively, the encoder may make the tool available but never signal its use (reserving control over the tool in the signaling, but always selecting a disabled value).

[0050] Figure 1 This is a schematic block diagram illustrating the functional modules of a video encoding and decoding system 100. In addition to constraints on specific encoding tools, system 100 includes constraints on block structure, thereby providing further flexibility in sub-architecture.

[0051] System 100 includes a source device 110 and a destination device 130. A communication channel 120 is used to communicate encoded video information from the source device 110 to the destination device 130. In some arrangements, one or both of the source device 110 and the destination device 130 may each include a mobile phone or "smartphone," in which case the communication channel 120 is a wireless channel. In other arrangements, the source device 110 and the destination device 130 may include video conferencing equipment, in which case the communication channel 120 is typically a wired channel such as an Internet connection. Furthermore, the source device 110 and the destination device 130 may include a wide range of arbitrary devices, including those supporting over-the-air television broadcasting, cable television applications, Internet video applications (including streaming), and applications that capture encoded video data on some computer-readable storage medium (such as hard disk drives in a file server).

[0052] like Figure 1 As shown, source device 110 includes a video source 112, a video encoder 114, and a transmitter 116. Video source 112 typically includes a source of captured video frame data (denoted as 113), such as a camera sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote camera sensor. Video source 112 can also be the output of a computer graphics card (e.g., video output from a display operating system and various applications running on a computing device (e.g., a tablet computer)). Examples of source devices 110 that may include a camera sensor as video source 112 include smartphones, video camcorders, professional cameras, and webcams.

[0053] For further reference Figure 3The video encoder 114 converts (or "encodes") the captured frame data (indicated by arrow 113) from the video source 112 into a bitstream (indicated by arrow 115). The video encoder 114 uses a specific set of VVC encoding tools (or "tiers") to encode the captured frame data 113. An indication of which tier to use is encoded into the bitstream 115 using a "profile_tier_level" syntax structure embedded in the "video parameter set" (VPS) or "sequence parameter set" (SPS). The SPS syntax structure is also named "seq_parameter_set_rbsp()", and the VPS syntax structure is also named "video_parameter_set_rbsp()". Additionally, a "general_constraint_info" syntax structure, which is part of the aforementioned profile_tier_level syntax structure, can be used to further constrain the encoding toolset used to the bitstream 115. The scaling list, the filter for ALF, and the parameters for LMCS are sent in one or more "Adaptive Parameter Sets" (APS). Each APS includes parameters for one of the three tools identified by the "aps_params_type" syntax element in the corresponding APS.

[0054] Bitstream 115 is transmitted by transmitter 116 via communication channel 120 as encoded video data (or "encoded video information"). In some implementations, bitstream 115 may be stored in a non-transitory storage device 122, such as "flash" memory or a hard disk drive, until subsequently transmitted via communication channel 120 or as a replacement for transmission via communication channel 120. For example, the encoded video data may be made available to customers via a wide area network (WAN) for video streaming applications when needed.

[0055] Destination device 130 includes receiver 132, video decoder 134, and display device 136. Receiver 132 receives encoded video data from communication channel 120 and transmits the received video data as a bitstream (indicated by arrow 133) to video decoder 134. Video decoder 134 then outputs decoded frame data (indicated by arrow 135) to display device 136 for video display. Decoded frame data 135 has the same chroma format as frame data 113. Examples of display devices 136 include cathode ray tubes, liquid crystal displays (such as in smartphones, tablets, computer monitors, or standalone televisions). Alternatively, the functions of source device 110 and destination device 130 can be embodied in a single device, examples of which include mobile phones and tablets. The decoded frame data can be further transformed before being presented to the user. For example, a projection format can be used to render a "viewport" with specific latitude and longitude from the decoded frame data to represent a 360° view of the scene.

[0056] Although the example apparatus has been described above, the source apparatus 110 and the destination apparatus 130 can each typically be configured within a general-purpose computer system via a combination of hardware and software components. Figure 2A This computer system 200 is illustrated and includes: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 configured as a video source 112, and a microphone 280; and output devices including a printer 215, a display device 214 configured as a display device 136, and a speaker 217. The computer module 201 can communicate with a communication network 220 via a connection 221 using an external modulator-demodulator (modem) transceiver device 216. The communication network 220, which may represent the communication channel 120, can be a WAN, such as the Internet, a cellular telecommunications network, or a private WAN. If the connection 221 is a telephone line, the modem 216 can be a conventional dial-up modem. Alternatively, if the connection 221 is a high-capacity (e.g., cable or optical) connection, the modem 216 can be a broadband modem. A wireless modem can also be used for wireless connection to the communication network 220. The transceiver device 216 can provide the functions of a transmitter 116 and a receiver 132, and the communication channel 120 can be embodied in the wiring 221.

[0057] Computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, memory unit 206 may have semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). Computer module 201 also includes multiple input / output (I / O) interfaces, including: an audio-video interface 207 connected to a video display 214, speakers 217, and microphone 280; an I / O interface 213 connected to a keyboard 202, mouse 203, scanner 226, camera 227, and optionally a joystick or other human-machine interface device (not shown); and an interface 208 for external modem 216 and printer 215. Signals from audio-video interface 207 to computer monitor 214 are typically output from the computer's graphics card. In some implementations, modem 216 may be integrated within computer module 201, for example, within interface 208. Computer module 201 also has a local network interface 211, which allows computer system 200 to connect via cable 223 to a local area network (LAN) known as a local area network (LAN). Figure 2A As shown, the local area communication network 222 can also be connected to the wide area network 220 via cable 224, wherein the local area communication network 222 typically includes a so-called "firewall" device or a device with similar functionality. The local network interface 211 may include Ethernet. TM Circuit card, Bluetooth TM The interface 211 can be a wireless or IEEE 802.11 wireless interface; however, various other types of interfaces can be implemented for interface 211. The local network interface 211 can also provide the functions of transmitter 116 and receiver 132, and the communication channel 120 can also be embodied in the local communication network 222.

[0058] I / O interfaces 208 and 213 can provide either serial or parallel connections, or both, with the former typically implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided, and storage device 209 typically includes a hard disk drive (HDD) 210. Other storage devices (not shown), such as floppy disk drives and tape drives, may also be used. An optical disc drive 212 is typically provided as a non-volatile source of data. Optical discs, such as CD-ROMs, DVDs, and Blu-ray Discs, may be used. TMPortable storage devices such as USB-RAM, portable external hard disk drives, and floppy disks serve as suitable sources of data for computer system 200. Typically, any of HDD 210, optical disc drive 212, network 220, and 222 can also be configured to operate as a video source 112 or as a destination for decoded video data to be stored for reproduction via display 214. The source device 110 and destination device 130 of system 100 can be embodied within computer system 200.

[0059] Components 205-213 of computer module 201 typically communicate via interconnect bus 204 and in a manner known to those skilled in the art to operate computer systems 200 in a conventional mode of operation. For example, processor 205 is connected to system bus 204 via wiring 218. Similarly, memory 206 and optical disc drive 212 are connected to system bus 204 via wiring 219. Examples of computers in which this arrangement can be practiced include IBM-PC and compatible machines, Sun SPARCstation, and Apple Macs. TM Or a similar computer system.

[0060] Where appropriate or desired, the video encoder 114 and the video decoder 134, as well as the methods described below, can be implemented using the computer system 200. Specifically, the video encoder 114, the video decoder 134, and the methods to be described can be implemented as one or more software applications 233 executable within the computer system 200. In particular, instructions 231 (see [link to instructions]) executed within the computer system 200 in the software 233 are utilized. Figure 2B The steps of implementing the video encoder 114, the video decoder 134, and the method are as follows. The software instructions 231 can be configured as one or more code modules, each for performing one or more specific tasks. Alternatively, the software can be divided into two separate parts, where a first part and its corresponding code modules perform the method, and a second part and its corresponding code modules manage the user interface between the first part and the user.

[0061] For example, the software can be stored in a computer-readable medium including the storage means described below. The software is loaded from the computer-readable medium into computer system 200 and then executed by computer system 200. The computer-readable medium having such software, or the computer program recorded on such computer-readable medium, is a computer program product. Using the computer program product in computer system 200 preferably implements advantageous means for carrying out the video encoder 114, video decoder 134, and the methods.

[0062] Software 233 is typically stored in HDD 210 or memory 206. The software is loaded from a computer-readable medium into computer system 200 and executed by computer system 200. Thus, for example, software 233 can be stored on an optically readable disk storage medium (e.g., CD-ROM) 225 read by optical disk drive 212.

[0063] In some instances, application 233 is supplied to the user encoded on one or more CD-ROMs 225 and read via the corresponding drive 212, or alternatively, the user may read application 233 from network 220 or 222. Furthermore, software may also be loaded into computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, and Blu-ray Discs. TM This includes hard disk drives, ROMs or integrated circuits, USB storage devices, magneto-optical disks, or computer-readable cards such as PCMCIA cards, regardless of whether these devices are inside or outside the computer module 201. Examples of temporary or intangible computer-readable transmission media that may also be used to provide software, applications, instructions, and / or video data or encoded video data to the computer module 401 include: radio or infrared transmission channels and network connections to other computers or networked devices, as well as the Internet or intranets including email sending and information recorded on websites.

[0064] The second part of the aforementioned application 233 and the corresponding code modules can be executed to implement one or more graphical user interfaces (GUIs) to be drawn or otherwise presented on the display 214. By typically operating the keyboard 202 and mouse 203, users and applications of the computer system 200 can operate the interface in a functionally applicable manner to provide control commands and / or input to applications associated with these (one or more) GUIs. Other functionally applicable forms of user interfaces can also be implemented, such as audio interfaces utilizing voice prompts output via speaker 217 and user voice commands input via microphone 280.

[0065] Figure 2B This is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents... Figure 2A The computer module 201 can access the logical aggregation of all memory modules (including HDD 209 and semiconductor memory 206).

[0066] When the computer module 201 is initially powered on, the power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in... Figure 2A The ROM 249 of the semiconductor memory 206 is sometimes referred to as firmware. The POST program 250 checks the hardware within the computer module 201 to ensure proper operation, and typically checks the processor 205, the memory 234 (209, 206), and the Basic Input / Output System Software (BIOS) module 251, which is also typically stored in the ROM 249, for correct operation. Once the POST program 250 has successfully run, the BIOS 251 boots. Figure 2A The hard disk drive 210 is booted. Booting the hard disk drive 210 causes the bootloader 252 residing on the hard disk drive 210 to be executed via the processor 205. This loads the operating system 253 into the RAM memory 206, where the operating system 253 begins operation. The operating system 253 is a system-level application executable by the processor 205 to implement various advanced functions, including processor management, memory management, device management, storage management, software application interfaces, and a general user interface.

[0067] The operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on computer module 201 has sufficient memory to execute without conflicting with memory allocated to other processes. Furthermore, appropriate use of memory is essential. Figure 2A The computer system 200 contains different types of memory so that each process can run efficiently. Therefore, aggregate memory 234 is not intended to illustrate how specific segments of memory are allocated (unless otherwise stated), but rather to provide an overview of the memory accessible to the computer system 200 and how that memory is used.

[0068] like Figure 2B As shown, processor 205 includes multiple functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and local or internal memory 248, sometimes referred to as cache memory. Cache memory 248 typically includes multiple storage registers 244-246 in a register segment. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using wiring 218. Memory 234 is connected to bus 204 using wiring 219.

[0069] Application program 233 includes a sequence of instructions 231, which may contain conditional branch instructions and loop instructions. Program 233 may also include data 232 used when executing program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative size of instructions 231 and memory locations 228-230, as described by the instructions shown in memory location 230, a particular instruction may be stored in a single memory location. Alternatively, as described by the instruction segments shown in memory locations 228 and 229, an instruction may be divided into multiple parts, each stored in a separate memory location.

[0070] Typically, a set of instructions is assigned to processor 205, which is then executed within processor 205. Processor 205 awaits subsequent input, which it responds to by executing another set of instructions. Inputs can be provided from one or more sources, including data generated by one or more input devices 202, 203, data received from external sources via one of networks 220, 202, data retrieved from one of storage devices 206, 209, or data retrieved from storage medium 225 inserted into the corresponding reader 212 (all of which are within...). Figure 2A (As shown in the diagram). Executing a set of instructions may result in output data in some cases. Execution may also involve storing data or variables into memory 234.

[0071] The video encoder 114, video decoder 134, and the method can use input variables 254 stored in corresponding memory locations 255, 256, and 257 within memory 234. The video encoder 114, video decoder 134, and the method generate output variables 261 stored in corresponding memory locations 262, 263, and 264 within memory 234. Intermediate variables 258 can be stored in memory locations 259, 260, 266, and 267.

[0072] refer to Figure 2B The processor 205, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 work together to perform micro-operation sequences required for "fetch, decode, and execute" cycles of instructions in the instruction set constituting program 233. Each fetch, decode, and execute cycle includes:

[0073] The fetch operation is used to fetch or read instruction 231 from memory locations 228, 229, and 230;

[0074] Decoding operation, in which the control unit 239 determines which instruction was extracted; and

[0075] The operation is performed, in which the control unit 239 and / or ALU 240 executes the instruction.

[0076] Then, a further fetch, decode, and execute cycle for the next instruction can be performed. Similarly, a storage cycle can be performed, in which the control unit 239 stores or writes a value to or writes it to memory location 232.

[0077] To explain Figure 10 and Figure 11 Each step or subprocess in the method is associated with one or more segments of program 233 and is typically performed by working together in the register units 244, 245, 247, ALU 240 and control unit 239 in processor 205 to fetch, decode and execute cycles of instructions in the instruction set of the segment of program 233.

[0078] Figure 3 This is a schematic block diagram showing the functional modules of the video encoder 114. Figure 4 This is a schematic block diagram illustrating the functional modules of the video decoder 134. Typically, data is transferred between the functional modules within the video encoder 114 and the video decoder 134 in groups of samples or coefficients (such as the partitioning of blocks into fixed-size sub-blocks) or as arrays. Figure 2A and 2BAs shown, the video encoder 114 and video decoder 134 can be implemented using a general-purpose computer system 200, wherein various functional modules can be implemented using dedicated hardware within the computer system 200 and executable software within the computer system 200 (such as one or more software code modules of a software application program 233 residing on the hard disk drive 205 and executed by the processor 205). Alternatively, the video encoder 114 and video decoder 134 can be implemented using a combination of dedicated hardware and software executable within the computer system 200. Alternatively, the video encoder 114, video decoder 134, and the method can be implemented in dedicated hardware such as one or more integrated circuits that perform the functions or sub-functions of the method. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), a dedicated standard product (ASSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or one or more microprocessors and associated memory. Specifically, the video encoder 114 includes modules 310-390, and the video decoder 134 includes modules 420-496, wherein each of these modules can be implemented as one or more software code modules of the software application 233.

[0079] although Figure 3 The video encoder 114 is an example of a Universal Video Coding (VVC) video coding pipeline, but other video codecs can also be used for the processing stages described herein. The video encoder 114 receives captured frame data 113, such as a series of frames, each frame including one or more color channels. Frame data 113 can be any chroma format, such as 4:0:0, 4:2:0, 4:2:2, or 4:4:4 chroma formats. The block partitioner 310 first divides the frame data 113 into CTUs, which are typically square in shape and configured to use a specific size. For example, the maximum enabled size of a CTU can be 32×32, 64×64, or 128×128 luma samples configured by the "sps_log2_ctu_size_minus5" syntax element present in the "sequence parameter set". The CTU size also provides the maximum CU size, as a CTU without further subdivision will contain one CU. Block partitioner 310 further divides each CTU into one or more CBs according to the luma coding tree or chroma coding tree. The luma channel can also be referred to as the primary color channel. Each chroma channel can also be referred to as a secondary color channel. CBs have various sizes and can include both square and non-square aspect ratios. (Reference) Figure 10The operation of block partitioner 310 is further described. However, in the VVC standard, CB, CU, PU, ​​and TU always have side lengths that are powers of 2. Therefore, based on the luminance coding tree and chrominance coding tree of the CTU, and according to the iterative progress on one or more blocks of the CTU, the current CB, represented as 312, is output from block partitioner 310. See below for reference. Figure 5 and Figure 6 To further illustrate the options for partitioning a CTU into CBs. Although the operation is generally described in units of CTU, the video encoder 114 and video decoder 134 can operate on smaller regions to reduce memory consumption. For example, individual CTUs can be partitioned into smaller regions referred to as “Virtual Pipeline Data Units” (VPDUs) of size 64×64. VPDU formation provides a more suitable data granularity for pipelined processing in the hardware architecture, where the reduction in memory footprint compared to operation on a full CTU reduces silicon area and thus cost. When the CTU size is 128×128, restrictions on the allowed coding tree are in place to ensure that a VPDU is fully processed before proceeding to the next VPDU. For example, at the root node of the coding tree for a 128×128 CTU, ternary splitting is prohibited because the resulting CUs (such as 32×128 / 128×32 or further decompositions thereof) cannot be processed with the required progression from a 64×64 region to subsequent 64×64 regions. When the CTU size is 64×64, regardless of the encoding tree selected by the encoder, the processing must complete one 64×64 region before moving to the next 64×64 region, i.e., from one CTU to the next CTU.

[0080] The CTUs obtained from the first segmentation of frame data 113 can be scanned in raster scan order and can be grouped into one or more "strips". A strip can be an "intra-frame" (or "I") strip. An intra-frame strip (I strip) indicates that each CU in the strip is intra-predicted. Alternatively, a strip can be single-predicted or double-predicted ("P" or "B" strips, respectively), indicating the additional availability of single and double prediction within the strip.

[0081] In I-slices, the coding trees of individual CTUs can diverge into two separate coding trees below the 64×64 level, one for luma and the other for chroma. Using separate trees allows different block structures to exist between luma and chroma within the luma 64×64 region of the CTU. For example, a large chroma block can be juxtaposed with many smaller luma blocks, and vice versa. In P- or B-slices, a single coding tree of the CTU defines a common block structure for both luma and chroma. The resulting blocks from a single tree can be intra-predictive or inter-predictive.

[0082] For each CTU, the video encoder 114 operates in two phases. In the first phase (referred to as the “search” phase), the block partitioner 310 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has an associated “candidate” CB. The first phase involves testing various candidate CBs to select the CB that provides relatively high compression efficiency and relatively low distortion. The testing typically involves Lagrangian optimization, thereby evaluating the candidate CBs based on a weighted combination of rate (coding cost) and distortion (error relative to the input frame data 113). The “best” candidate CB (the CB with the lowest evaluated rate / distortion) is selected for subsequent encoding into the bitstream 115. Included in the evaluation of the candidate CBs are the options of using the CB for a given region, or splitting further regions according to various splitting options and encoding the individual smaller resulting regions using further CBs, or even further splitting the regions. Thus, both the coding tree and the CB are selected in the search phase.

[0083] Video encoder 114 generates prediction blocks (PBs) indicated by arrow 320 for each CB (e.g., CB 312). PB 320 is a prediction of the content of the associated CB 312. Subtractor module 322 generates a difference (or “residual”, which refers to the difference in the spatial domain) between PB 320 and CB 312, represented as 324. Difference 324 is the block size difference between the corresponding samples in PB 320 and CB 312. Difference 324 is transformed, quantized, and represented as a transform block (TB) indicated by arrow 336. PB 320 and the associated TB 336 are typically selected from one of several possible candidate CBs, for example, based on the evaluated cost or distortion.

[0084] The candidate coded block (CB) is a CB obtained from the video encoder 114 for one of the prediction modes available for the associated PB and the resulting residual. When combined with the prediction PB in the video decoder 114, the TB 336 reduces the difference between the decoded CB and the original CB 312 at the cost of additional signal notification in the bitstream.

[0085] Therefore, each candidate coded block (CB, i.e., the prediction block (PB) combined with the transform block (TB)) has an associated coding cost (or "rate") and an associated difference (or "distortion"). The distortion of the CB is typically estimated as the difference in sample values, such as the sum of absolute differences (SAD) or the sum of squared differences (SSD). The mode selector 386 uses the difference 324 to determine the estimate obtained from each candidate PB to determine the prediction mode 387. The prediction mode 387 indicates the decision to use a specific prediction mode (e.g., intra-frame prediction or inter-frame prediction) for the current CB. The coding cost associated with each candidate prediction mode and the corresponding residual coding can be estimated at a significantly lower cost compared to entropy coding of the residual. Therefore, even in a real-time video encoder, multiple candidate modes can be evaluated to determine the optimal mode in the sense of rate distortion.

[0086] The best mode in terms of determination distortion is usually achieved using a Lagrange-optimized variation.

[0087] Lagrange or similar optimization processes can be used to select the optimal partition from the CTU to the CB (using block partitioner 310) and the optimal prediction mode from multiple possibilities. By applying Lagrange optimization of candidate modes to the mode selector module 386, the intra-prediction mode with the lowest cost metric is selected as the optimal mode. The lowest cost mode includes the selected minor transform index 388, which is also encoded into bitstream 115 by entropy encoder 338.

[0088] In the second phase of the operation of the video encoder 114 (referred to as the "encoding" phase), the determined coding trees for each CTU (one or more) are iterated within the video encoder 114. For CTUs using individual trees, the luma coding tree is encoded first for each 64×64 luma region of the CTU, followed by the chroma coding tree. Within the luma coding tree, only the luma CB is encoded, and within the chroma coding tree, only the chroma CB is encoded. For CTUs using shared trees, a single tree describes the CUs, i.e., the luma CB and the chroma CB, according to the common block structure of the shared tree.

[0089] The entropy encoder 338 supports both variable-length encoding and arithmetic encoding of syntax elements. Parts of the bitstream, such as “parameter sets” (e.g., Sequence Parameter Sets (SPS) and Picture Parameter Sets (PPS)), use a combination of fixed-length and variable-length codewords. Stripes (also called continuous sections) have a stripe header using variable-length encoding, followed by stripe data using arithmetic encoding. The stripe header defines parameters specific to the current stripe, such as stripe-level quantization parameter offsets. The stripe data includes the syntax elements of the individual CTUs within the stripe. Using both variable-length and arithmetic encoding requires sequential parsing within the various sections of the bitstream. These sections can be described using start codes to form “Network Abstraction Layer Units” or “NAL Units.” Arithmetic encoding is supported using context-adaptive binary arithmetic encoding processing. Arithmetic-encoded syntax elements consist of a sequence of one or more “bins”. Like bits, bins have values ​​of “0” or “1”. However, bins are not encoded as discrete bits in the bitstream 115. A bin has an associated predicted (or "possible" or "maximum probability") value and an associated probability (called "context"). The "maximum probability symbol" (MPS) is encoded when the actual bin to be encoded matches the predicted value. Encoding the MPS is relatively inexpensive in terms of bit consumption (including the cost of a total of less than one discrete bit) in bit stream 115. When the actual bin to be encoded does not match the possible value, the "minimum probability symbol" (LPS) is encoded. Encoding the LPS is relatively expensive in terms of bit consumption. Bin encoding techniques enable efficient encoding of bins that can skew the probabilities of "0" vs "1". For syntactic elements with two possible values ​​(i.e., "flag"), a single bin is sufficient. For syntactic elements with many possible values, a sequence of bins is required.

[0090] The presence of later bins in a sequence can be determined based on the value of an earlier bin. Furthermore, each bin can be associated with more than one context. The selection of a specific context can depend on earlier bins in a syntactic element, bins of adjacent syntactic elements (i.e., bins from adjacent blocks), etc. Each time a context-encoded bin is encoded, the context selected for that bin (if it exists) is updated to reflect the new bin value. Therefore, the binary arithmetic coding scheme is called adaptive.

[0091] Video encoder 114 also supports bins lacking context (“bypass bins”). Bypass bins are encoded using an equal probability distribution between “0” and “1”. Therefore, each bin occupies the encoding cost of one bit in bitstream 115. The absence of context saves memory and reduces complexity, thus using bypass bins with an unskewed distribution of values ​​for a particular bin. An example of a context-adaptive entropy encoder is known in the art as CABAC (Context-Adaptive Binary Arithmetic Encoder), and many variations of this encoder are used in video coding.

[0092] The entropy encoder 338 encodes the quantization parameter 392 using a combination of context-coded bins and bypass-coded bins, and encodes the LFNST index 388 if used for the current CB. The quantization parameter 392 is encoded using an "incremental QP". The incremental QP is signaled at most once in each region called a "quantization group". The quantization parameter 392 is applied to the residual coefficients of the luma CB. Adjusted quantization parameters are applied to the residual coefficients of the juxtaposed chroma CB. Adjusted quantization parameters may include mapping from the luma quantization parameter 392 according to a mapping table and a CU level offset selected from an offset list. The secondary transform index 388 is signaled when the residual associated with the transform block includes valid residual coefficients only in those coefficient positions that have undergone transformation to the primary coefficients by applying a secondary transform.

[0093] Multiplexer module 384 outputs PB 320 from intra-prediction module 364 based on the optimal intra-prediction mode determined from the test prediction modes selected from the various candidate CBs tested. Candidate prediction modes need not include every conceivable prediction mode supported by video encoder 114. Intra-prediction is divided into three types. "DC intra-prediction" involves filling the PB with a single value representing the average of nearby reconstructed samples. "Plane intra-prediction" involves filling the PB with samples based on a plane, where the DC offset, along with vertical and horizontal gradients, is derived from neighboring reconstructed samples. Neighboring reconstructed samples typically include a row of reconstructed samples above the current PB (extending to the right of the PB) and a column of reconstructed samples to the left of the current PB (extending downwards beyond the PB). "Angular intra-prediction" involves filling the PB with neighboring reconstructed samples that are filtered and propagated across the PB in a specific direction (or "angle"). In VVC, 65 angles are supported, with rectangular blocks able to utilize additional angles unavailable to square blocks to generate a total of 87 angles. A fourth type of intra-frame prediction can be used for chroma PB, thereby generating PB from juxtaposed luma reconstruction samples according to a "Cross-Component Linear Model" (CCLM) mode. Three different CCLM modes are available, each using a different model derived from adjacent luma and chroma samples. The derived model is used to generate sample blocks for chroma PB from juxtaposed luma samples.

[0094] If previously reconstructed samples are unavailable (e.g., at the edge of a frame), a default halftone value of half the sample range is used. For example, for 10-bit video, a value of 512 is used. Since no previous samples are available for the CB located at the top left of the frame, the angular and planar intra-frame prediction modes produce the same output as the DC prediction mode, i.e., a flat plane of samples with halftone values ​​as amplitude.

[0095] For inter-frame prediction, the motion compensation module 380 generates a prediction block 382 using samples from one or two frames preceding the current frame in the order of the coded frames in the bitstream, and the multiplexer module 384 outputs it as a PB 320. Furthermore, for inter-frame prediction, a single coding tree is typically used for both the luma and chroma channels. The order of the coded frames in the bitstream may differ from the order of the frames at capture or display. When one frame is used for prediction, the block is called a "single prediction" and has two associated motion vectors. When two frames are used for prediction, the block is called a "double prediction" and has two associated motion vectors. For P-strips, each CU can be intra-frame prediction or single prediction. For B-strips, each CU can be intra-frame prediction, single prediction, or double prediction. Frames are typically encoded using a "picture group" structure to achieve a temporal hierarchy. Frames can be divided into multiple stripes, each strip encoding a portion of the frame. The temporal hierarchy of frames allows frames to reference previous and subsequent pictures in the order of the displayed frames. The image is encoded in the order required to satisfy the dependencies of decoding each frame.

[0096] Samples are selected based on motion vector 378 and a reference image index. Motion vector 378 and the reference image index apply to all color channels; therefore, inter-frame prediction is described primarily in terms of operation on the PU rather than the PB. That is, a single coding tree is used to describe the decomposition of each CTU into one or more inter-frame prediction blocks. Inter-frame prediction methods can vary in the number and precision of motion parameters. Motion parameters typically include a reference frame index, indicating which reference frame(s) from the reference frame list will be used, along with spatial translations of the individual reference frames. However, this can include additional frames, special frames, or complex affine parameters (such as scaling and rotation). Additionally, predetermined motion refinement processes can be applied to generate dense motion estimates based on the reference sample blocks.

[0097] With PB 320 determined and selected, and subtracted from the original sample block at subtractor 322, the residual with the lowest encoding cost, represented as 324, is obtained and lossy compression is performed on this residual. The lossy compression process includes transform, quantization, and entropy coding steps. A forward principal transform module 326 applies a forward transform to the difference 324, thereby transforming the difference 324 from the spatial threshold to the frequency domain and generating principal transform coefficients, indicated by arrow 328. The maximum principal transform size in one dimension is a 32-point DCT-2 or 64-point DCT-2 transform configured by the "sps_max_luma_transform_size_64_flag" in the sequence parameter set. If the forward encoded CB is larger than the maximum supported principal transform size represented as a block size (e.g., 64×64 or 32×32), the principal transform 326 is applied in a block-wise manner to transform all samples of the difference 324. In the case of using a non-square CB, the maximum available transform size in each dimension of the CB is also used for blockization. For example, when using a maximum transform size of 32, a 64×16 CB uses two 32×16 primary transforms arranged in blocks. When the size of the CB is larger than the maximum supported transform size, the CB is filled with TBs arranged in blocks. For example, a 2×2 arrangement uses four 64×64 TBs to fill a 128×128 CB with a maximum transform size of 64-pt. A 2×4 arrangement uses eight 32×32 TBs to fill a 64×128 CB with a maximum transform size of 32-pt.

[0098] The application of transform 326 yields multiple TBs of the CB. When each transform is applied to a TB with a difference 324 greater than 32×32 (e.g., 64×64), all resulting principal transform coefficients 328 outside the top-left 32×32 region of the TB are set to zero, i.e., discarded. The remaining principal transform coefficients 328 are passed to quantizer module 334. The principal transform coefficients 328 are quantized according to quantization parameter 392 associated with the CB to produce principal transform coefficients 332. In addition to quantization parameter 392, quantizer module 334 can also apply a "scaling list" to allow non-uniform quantization within the TB by further scaling the residual coefficients according to their spatial location within the TB. Quantization parameter 392 can vary for the luminance CB relative to the individual chrominance CBs. The principal transform coefficients 332 are passed to forward secondary transform module 330 by performing an inseparable minor transform (NSST) operation or bypassing the minor transform to produce the transform coefficients indicated by arrow 336. The forward primary transform is typically separable, transforming a set of rows for each TB and then a set of columns. For luminance TBs with a width and height not exceeding 16 samples, the forward primary transform module 326 uses a Type II Discrete Cosine Transform (DCT-2) in both the horizontal and vertical directions, or bypasses the transform in both the horizontal and vertical directions, or uses a combination of a Type VII Discrete Sine Transform (DST-7) and a Type VIII Discrete Cosine Transform (DCT-8) in either the horizontal or vertical direction. The use of the combination of DST-7 and DCT-8 is referred to as the "Multiple Transform Selection Set" (MTS) in the VVC standard.

[0099] The forward minor transform of module 330 is typically a non-separable transform, applied only to the residuals of the intra-predicted CU and still bypassable. The forward minor transform operates on 16 samples (arranged as the top-left 4×4 sub-block of the primary transform coefficients 328) or 48 samples (arranged as three 4×4 sub-blocks of the top-left 8×8 coefficients of the primary transform coefficients 328) to produce a set of minor transform coefficients. The number of minor transform coefficient sets can be less than the number of primary transform coefficient sets from which they are derived. Because the minor transform is applied only to the set of coefficients that are adjacent to each other and include DC coefficients, the minor transform is called the “Low-Frequency Non-Separable Minor Transform” (LFNST). Furthermore, when applying LFNST, all residual coefficients in the TB must be zero in both the primary and minor transform domains.

[0100] The quantization parameter 392 is constant for a given TB, and thus a uniform scaling of the residual coefficients generated for the main transform domain of the TB is obtained. The quantization parameter 392 can be periodically varied by a “decremental quantization parameter” signaled. For a CU contained within a given region (called a “quantization group”), the incremental quantization parameter (decremental QP) is signaled once. If the CU is larger than the quantization group size, the incremental QP is signaled once for one of the TBs of the CU. That is, for the first quantization group of the CU, the entropy encoder 338 signals the incremental QP once, and for any subsequent quantization group of the CU, the incremental QP is not signaled. Non-uniform scaling can also be achieved by applying a “quantization matrix,” whereby the scaling factor applied to each residual coefficient is derived from a combination of the quantization parameter 392 and the corresponding entries in the scaling matrix. The scaling matrix can have a size smaller than the TB, and when applied to the TB, a nearest neighbor method is used to provide scaling values ​​for each residual coefficient from a scaling matrix smaller than the TB size. The residual coefficients 336 are fed to the entropy encoder 338 for encoding in the bit stream 115. Typically, according to the scan pattern, the residual coefficients of each TB with at least one valid residual coefficient are scanned from the TU to produce an ordered list of values. The scan pattern typically scans the TB as a sequence of 4×4 “sub-blocks,” thus providing regular scan operations at a granularity of 4×4 sets of residual coefficients, where the arrangement of the sub-blocks depends on the size of the TB. The scan within each sub-block and the progression from one sub-block to the next typically follow a backward diagonal scan pattern. Additionally, quantization parameter 392 is encoded into bitstream 115 using incremental QP syntax elements, and minor transform index 388 is encoded into bitstream 115.

[0101] As described above, the video encoder 114 needs access to the frame representation corresponding to the encoded frame representation seen in the video decoder 134. Therefore, the residual coefficients 336 are passed through the inverse secondary transform module 344 (operating according to the secondary transform coefficients 388) to produce intermediate inverse transform coefficients, indicated by arrow 342. The intermediate inverse transform coefficients 346 are inversely quantized by the dequantization module 340 according to the quantization parameter 392 to produce residual samples, indicated by arrow 346. The dequantization module 340 can also perform inverse non-uniform scaling of the residual coefficients using a scaling list, corresponding to the forward scaling performed in the quantizer module 334. The intermediate inverse transform coefficients 346 are passed to the inverse primary transform module 348 to produce residual samples of TU, ​​indicated by arrow 350. The inverse primary transform module 348 applies a DCT-2 transform horizontally and vertically, constrained by the maximum available transform size as described in the reference forward primary transform module 326. The type of inverse transformation performed by the inverse secondary transformation module 344 corresponds to the type of forward transformation performed by the forward secondary transformation module 330. The type of inverse transformation performed by the inverse primary transformation module 348 corresponds to the type of primary transformation performed by the primary transformation module 326. The summation module 352 adds the residual sample 350 to PU 320 to generate the reconstructed sample of CU (indicated by arrow 354).

[0102] Reconstructed sample 354 is passed to reference sample cache 356 and in-loop filter module 368. Reference sample cache 356, typically implemented using static RAM on an ASIC (thus avoiding expensive off-chip memory access), provides the minimum sample storage required to generate intra-frame PB dependencies for subsequent CUs in a frame. Minimal dependencies typically include a “line buffer” of samples below a row of CTUs for use by the next row of CTUs and a column buffer whose range is set by the height of the CTU. Reference sample cache 356 feeds reference samples (indicated by arrow 358) to reference sample filter 360. Sample filter 360 applies a smoothing operation to produce filtered reference samples (indicated by arrow 362). Filtered reference samples 362 are used by intra-frame prediction module 364 to generate intra-frame prediction blocks of samples, indicated by arrow 366. For each candidate intra-frame prediction mode, intra-frame prediction module 364 generates sample block 366. Sample block 366 is generated by module 364 using techniques such as DC, planar, or angular intra-frame prediction.

[0103] The in-loop filter module 368 applies several filtering stages to the reconstructed samples 354. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned with the CU boundaries to reduce artifacts caused by discontinuities. Another filtering stage present in the in-loop filter module 368 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. Another available filtering stage in the in-loop filter module 368 is a "sample adaptive offset" (SAO) filter. The SAO filter works by first classifying the reconstructed samples into one or more categories and then applying an offset at the sample level according to the assigned category.

[0104] The filtered sample, indicated by arrow 370, is output from the in-loop filter module 368. The filtered sample 370 is stored in the frame buffer 372. The frame buffer 372 typically has a capacity to store several (e.g., up to 16) images and is therefore stored in memory 206. Due to the large memory consumption required, the frame buffer 372 is typically not stored in on-chip memory. Thus, access to the frame buffer 372 is expensive in terms of memory bandwidth. The frame buffer 372 provides reference frames (indicated by arrow 374) to the motion estimation module 376 and the motion compensation module 380.

[0105] Motion estimation module 376 estimates multiple "motion vectors" (denoted as 378), each a Cartesian space offset relative to the current CB position, thus referencing a block of one of the reference frames in frame buffer 372. A filtered block (denoted as 382) of reference samples is generated for each motion vector. The filtered reference samples 382 form further candidate modes for potential selection by mode selector 386. Furthermore, for a given CU, PB 320 can be formed using one reference block ("single prediction") or two reference blocks ("double prediction"). For the selected motion vector, motion compensation module 380 generates PB 320 based on filtering that supports sub-pixel precision in the motion vector. Thus, motion estimation module 376 (which operates on many candidate motion vectors) can perform simplified filtering compared to motion compensation module 380 (which operates only on the selected candidate), achieving reduced computational complexity. When video encoder 114 selects inter-frame prediction for a CU, motion vector 378 is encoded into bitstream 115.

[0106] Although the reference to Universal Video Coding (VVC) explains Figure 3The video encoder 114 can be used, but other video coding standards or implementations can also employ the processing stages of modules 310-390. Frame data 113 (and bitstream 115) can also be read from (or written to) memory 206, hard disk 210, CD-ROM, Blu-ray disc, or other computer-readable storage media. Additionally, frame data 113 (and bitstream 115) can be received from (or sent to) an external source, such as a server or RF receiver connected to communication network 220. Communication network 220 can provide limited bandwidth, thus requiring rate control in the video encoder 114 to avoid network saturation when frame data 113 is difficult to compress. Furthermore, bitstream 115 can be constructed from one or more stripes representing spatial portions (CTU sets) of frame data 113, these stripes being generated by one or more instances of the video encoder 114 and operating in a coordinated manner under the control of processor 205. In the context of this invention, a stripe may also be referred to as a “continuous portion” of a bitstream. A stripe is continuous within a bitstream and (e.g., if parallel processing is being used) can be encoded or decoded into separate portions.

[0107] exist Figure 4 The video decoder 134 is shown. Although... Figure 4 The video decoder 134 is an example of a Universal Video Coding (VVC) video decoding pipeline, but other video codecs can also be used for the processing stages described herein. Figure 4 As shown, bitstream 133 is input to video decoder 134. Bitstream 133 can be sourced from memory 206, hard disk drive 210, CD-ROM, Blu-ray disc, etc. TM It can be read from other non-transitory computer-readable storage media. Alternatively, bit stream 133 can be received from an external source, such as a server or radio frequency receiver connected to communication network 220. Bit stream 133 contains encoded syntax elements representing the captured frame data to be decoded.

[0108] Bitstream 133 is input to entropy decoder module 420. Entropy decoder module 420 extracts syntactic elements from bitstream 133 by decoding a sequence of "bins" and passes the values ​​of the syntactic elements to other modules in video decoder 134. Entropy decoder module 420 uses variable-length and fixed-length decoding to decode SPS, PPS, or stripe headers and uses an arithmetic decoding engine to decode the syntactic elements of the stripe data into a sequence of one or more bins. Each bin can use one or more "contexts," where the context describes the probability levels used to encode "one" and "zero" values ​​for the bin. When multiple contexts are available for a given bin, a "context modeling" or "context selection" step is performed to select one of the available contexts to decode the bin. The processing of decoding bins forms a sequential feedback loop, so that the entire stripe of each stripe can be decoded by a given instance of entropy decoder 420. A single (or several) high-execution-entropy decoder 420 instance can decode all stripes of a frame from bit stream 115, while multiple low-execution-entropy decoder 420 instances can simultaneously decode stripes of a frame from bit stream 133.

[0109] The entropy decoder module 420 applies an arithmetic coding algorithm, such as Context Adaptive Binary Arithmetic Coding (CABAC), to decode syntax elements from the bitstream 133. The decoded syntax elements are used to reconstruct parameters within the video decoder 134. These parameters include residual coefficients (indicated by arrow 424), quantization parameters 474, minor transform index 470, and mode selection information such as intra-frame prediction modes (indicated by arrow 458). The mode selection information also includes information such as motion vectors and partitions of each CTU to one or more CBs. These parameters are typically combined with sample data from previously decoded CBs to generate a PB.

[0110] The residual coefficients 424 are passed to the inverse secondary transform module 436, where a secondary transform is applied or no operation (bypass) is performed based on the secondary transform index. The inverse secondary transform module 436 generates reconstructed transform coefficients 432, i.e., primary transform coefficients, from the secondary transform domain coefficients. The reconstructed transform coefficients 432 are input to the dequantizer module 428. The dequantizer module 428 performs inverse quantization (or "scaling") on the residual coefficients 432, i.e., in the primary transform coefficient domain, to create reconstructed intermediate transform coefficients, indicated by arrow 440, based on quantization parameters 474. The dequantizer module 428 can also apply a scaling matrix to provide non-uniform dequantization within the TB, corresponding to the operation of the dequantizer module 340. If a non-uniform inverse quantization matrix is ​​indicated in the bitstream 133, the video decoder 134 reads the quantization matrix from the bitstream 133 as a sequence of scaling factors and arranges the scaling factors into a matrix. Inverse scaling combines the quantization matrix with the quantization parameters to create the reconstructed intermediate transform coefficients 440.

[0111] The reconstructed transform coefficients 440 are passed to the inverse master transform module 444. Module 444 transforms coefficients 440 from the frequency domain back to the spatial domain. The inverse master transform module 444 applies an inverse DCT-2 transform horizontally and vertically, constrained by the maximum available transform size as described in the reference forward master transform module 326. The result of the operation of module 444 is a block of residual samples, indicated by arrow 448. The block of residual samples 448 is equal in size to the corresponding CB. The block of residual samples 448 is fed to the summing module 450. At the summing module 450, residual samples 448 are added to the decoder PB (denoted as 452) to produce a block of reconstructed samples, indicated by arrow 456. The reconstructed samples 456 are fed to the reconstructed sample cache 460 and the in-loop filtering module 488. The in-loop filtering module 488 produces a reconstructed block of frame samples, denoted as 492. Frame samples 492 are written to the frame buffer 496.

[0112] The reconstruction sample cache 460 operates in a manner similar to the reconstruction sample cache 356 of the video encoder 114. The reconstruction sample cache 460 provides storage for reconstructed samples required for intra-frame prediction of subsequent CBs in the absence of memory 206 (e.g., by using data 232, typically on-chip memory, instead). The reference sample, indicated by arrow 464, is obtained from the reconstruction sample cache 460 and supplied to the reference sample filter 468 to produce the filtered reference sample, indicated by arrow 472. The filtered reference sample 472 is supplied to the intra-frame prediction module 476. Module 476 generates blocks of intra-frame prediction samples, indicated by arrow 480, based on the intra-frame prediction mode parameters 458 represented in bitstream 133 and decoded by the entropy decoder 420. The blocks of samples 480 are generated using modes such as DC, planar, or angular intra-frame prediction.

[0113] When the prediction mode of a color block (CB) is indicated in bitstream 133 to use intra-prediction, intra-prediction samples 480 are used to form decoded blocks (PBs) 452 via multiplexer module 484. Intra-prediction produces a prediction block (PB) of samples, i.e., a block within a color component, which is derived using "neighboring samples" in the same color component. Neighboring samples are those adjacent to the current block and have already been reconstructed because they are earlier in the block decoding order. In the case of juxtaposition of luma and chroma blocks, luma and chroma blocks can use different intra-prediction modes. However, two chroma CBs share the same intra-prediction mode.

[0114] When the prediction mode of CB is indicated as intra-prediction in bitstream 133, motion compensation module 434 uses motion vectors (decoded from bitstream 133 by entropy decoder 420) and a reference frame index to select and filter sample blocks 498 from frame buffer 496 to produce a block of inter-prediction samples represented as 438. Sample block 498 is obtained from previously decoded frames stored in frame buffer 496. For dual prediction, two sample blocks are generated and blended together to produce samples for decoding PB 452. Frame buffer 496 is filled with filtered block data 492 from in-loop filtering module 488. Similar to in-loop filtering module 368 of video encoder 114, in-loop filtering module 488 applies any DBF, ALF, and SAO filtering operations. Generally, motion vectors are applied to both luma and chroma channels, but the filtering processes used for subsample interpolation in the luma and chroma channels are different.

[0115] Figure 5 This is a schematic block diagram illustrating a set 500 of available partitions or splits of a region into one or more sub-regions in a tree structure of general video coding. (See reference...) Figure 3 The segmentation shown in set 500 can be used by block partitioner 310 of encoder 114 to segment each CTU into one or more CUs or CBs according to a coding tree determined by Lagrange optimization.

[0116] Although set 500 only shows the division of a square region into other possible non-square subregions, it should be understood that set 500 is showing potential divisions from parent nodes to child nodes in the coding tree, and does not require the parent node to correspond to a square region. If the containing region is non-square, the size of the block obtained from the division is scaled according to the aspect ratio of the containing block. Once the region is not further divided, i.e., at the leaf node of the coding tree, the CU occupies the region.

[0117] The process of subsegmenting a region into subregions must terminate when the resulting subregion reaches the minimum CU size (typically 4×4 luma samples). In addition to constraining the CU to prevent block regions from being smaller than a predetermined minimum size (e.g., 16 samples), the CU is constrained to have a minimum width or height of 4. Other minimum values ​​are also possible, either in terms of width and height or in terms of both. The subsegmentation process can also terminate before the deepest decomposition level, resulting in CUs larger than the minimum CU size. It is possible that no splitting occurs, resulting in a single CU occupying the entire CTU. The single CU occupying the entire CTU is the maximum available coding unit size. Due to the use of a subsampled chroma format (such as 4:2:0), the arrangement of the video encoder 114 and video decoder 134 can terminate the splitting of regions in the chroma channels earlier than in the luma channels, including in the case of a shared coding tree defining the block structure of the luma and chroma channels. When separate coding trees are used for luma and chroma, constraints on available splitting operations ensure a minimum chroma CB region of 16 samples, even when such a CB is juxtaposed with a larger luma region (e.g., 64 luma samples).

[0118] In the absence of further sub-segments, a CU exists at the leaf nodes of the coding tree. For example, leaf node 510 contains one CU. At the non-leaf nodes of the coding tree, there are splits to two or more other nodes, where each node can be a leaf node forming a CU or a non-leaf node containing further splits to smaller regions. At each leaf node of the coding tree, there is a coding block for each color channel. A split terminating at the same depth for both luma and chroma results in three juxtaposed CBs. A split terminating at a depth deeper than chroma results in multiple luma CBs juxtaposed with the chroma channel CB.

[0119] like Figure 5 As shown, quadtree split 512 divides the containing region into four equal-sized regions. Compared to HEVC, Universal Video Coding (VVC) achieves additional flexibility through additional splits, including horizontal binary split 514 and vertical binary split 516. Splits 514 and 516 each divide the containing region into two equal-sized regions. The splits are along the horizontal boundary (514) or vertical boundary (516) within the containing block.

[0120] Further flexibility is achieved in general video coding by adding ternary horizontal splits 518 and ternary vertical splits 520. Ternary splits 518 and 520 divide a block into three regions bounded horizontally (518) or vertically (520) along 1 / 4 and 3 / 4 of the width or height of the containing region. A combination of quadtrees, binary trees, and ternary trees is called a “QTBTTT”. The root of the tree consists of zero or more than zero quadtree splits (the “QT” part of the tree). Once the QT part terminates, zero or more than zero binary or ternary splits may occur (the “multitree” or “MT” part of the tree), eventually ending in a CB or CU at the leaf node of the tree. The leaf node is a CU if the tree describes all color channels. The leaf node is a CB if the tree describes either a luma channel or a chroma channel.

[0121] Compared to HEVC, which only supports quadtrees and therefore only square blocks, QTBTTT offers a wider range of possible CU sizes, particularly considering the potential recursive application of binary and / or ternary splits. When only quadtree splits are available, each increase in the coding tree depth corresponds to a CU size reduction to one-quarter of the parent region's size. In VVC, the availability of binary and ternary splits means that the coding tree depth no longer directly corresponds to the CU region. The possibility of anomalous (non-square) block sizes can be reduced by constraining split options to eliminate splits that result in block widths or heights less than four samples or that are not multiples of four samples. Typically, constraints are applied when considering luma samples. However, in the aforementioned arrangement, constraints can also be applied individually to blocks in the chroma channel. Split options that apply constraints to the chroma channel can yield different minimum block sizes for luma vs. chroma (e.g., when the frame data is in 4:2:0 or 4:2:2 chroma format). Each split produces a sub-region with side sizes that are unchanged, halved, or quartered relative to the containing region. Then, since the size of the CTU is a power of 2, the side dimensions of all CUs are also powers of 2.

[0122] Figure 6 This is a schematic flowchart illustrating the data flow 600 of the QTBTTT (or "coding tree") structure used in general video coding. The QTBTTT structure is used for each CTU to define the partitioning of the CTU into one or more CUs. The QTBTTT structure of each CTU is determined by the block partitioner 310 in the video encoder 114 and is encoded into bit stream 115 or decoded from bit stream 133 by the entropy decoder 420 in the video decoder 134. Figure 5 As shown in the diagram, data stream 600 further exhibits features that allow block partitioner 310 to divide a CTU into one or more CUs using a combination of licenses.

[0123] Starting from the top level of the hierarchy, i.e., at CTU, zero or more quadtree partitions are performed first. Specifically, the block partitioner 310 makes a quadtree (QT) split decision 610. The decision at 610 returns a "1" sign, indicating that it is decided to split the current node into four child nodes according to the quadtree split 512. The result is the generation of four new nodes, such as at 620, and for each new node, the process recursively returns to the QT split decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates no further splitting (returns a "0" sign), the quadtree partitioning stops, and multitree (MT) splits are subsequently considered.

[0124] First, block partitioner 310 makes an MT split decision 612. At 612, a decision indicating whether to perform an MT split is made is given. Decision 612 returns a "0" sign, indicating that no further splitting of the node into its child nodes will be performed. If no further splitting of the node is performed, the node is a leaf node of the coding tree and corresponds to a CU. The leaf node is output at 622. Alternatively, if MT split 612 indicates a decision to perform an MT split (returning a "1" sign), block partitioner 310 proceeds to direction decision 614.

[0125] Direction decision 614 indicates the direction of MT splitting as horizontal (“H” or “0”) or vertical (“V” or “1”). If decision 614 returns “0” indicating the horizontal direction, block partitioner 310 proceeds to decision 616. If decision 614 returns “1” indicating the vertical direction, block partitioner 310 proceeds to decision 618.

[0126] In each of decisions 616 and 618, the number of partitions to be split in the MT partition during the BT / TT split is indicated as two (binary split or "BT" nodes) or three (ternary split or "TT" nodes). That is, when the direction indicated from 614 is horizontal, the block partitioner 310 makes the BT / TT split decision 616, and when the direction indicated from 614 is vertical, the block partitioner 310 makes the BT / TT split decision 618.

[0127] BT / TT split decision 616 indicates whether the horizontal split is a binary split 514 indicated by returning "0" or a ternary split 518 indicated by returning "1". When BT / TT split decision 616 indicates a binary split, at step 625 of generating HBT CTU nodes, block partitioner 310 generates two nodes based on binary horizontal split 514. When BT / TT split 616 indicates a ternary split, at step 626 of generating HTT CTU nodes, block partitioner 310 generates three nodes based on ternary horizontal split 518.

[0128] The BT / TT split decision 618 indicates whether the vertical split is a binary split 516 indicated by returning "0" or a ternary split 520 indicated by returning "1". When the BT / TT split 618 indicates a binary split, at step 627 of generating VBT CTU nodes, the block partitioner 310 generates two nodes according to the vertical binary split 516. When the BT / TT split 618 indicates a ternary split, at step 628 of generating VTT CTU nodes, the block partitioner 310 generates three nodes according to the vertical ternary split 520. For each node obtained from steps 625-628, the data flow 600 is applied according to direction 614 in a left-to-right or top-to-bottom order, recursively returning to the MT split decision 612. As a result, binary and ternary splits can be applied to generate CUs of various sizes.

[0129] Figure 7A and 7B Provides an example of CTU 710 splitting into multiple CUs or CBs, 700. Figure 7A Example CU 712 is shown in the figure. Figure 7A This illustrates the spatial arrangement of the CUs in a CTU 710. Example partition 700 is shown. Figure 7B It is also shown as coding tree 720.

[0130] exist Figure 7A At each non-leaf node in CTU 710 (e.g., nodes 714, 716, and 718), the contained nodes (which may be further splits or CUs) are scanned or traversed in "Z order" to create a list of nodes represented as columns in the coding tree 720. For quadtree splits, the Z-order scan results in a sequence from top left to right followed by a sequence from bottom left to right. For horizontal and vertical splits, the Z-order scan (traversal) simplifies to a top-to-bottom scan and a left-to-right scan, respectively. Figure 7B The encoding tree 720 lists all nodes and CUs according to the applied scan order. Each split generates a list of two, three, or four new nodes at the next level of the tree, until a leaf node (CU) is reached.

[0131] As per reference Figure 3 In the case where the image is decomposed into CTUs and further decomposed into CUs using block partitioner 310, and residual blocks (324) are generated using CUs, the residual blocks are forward transformed and quantized using video encoder 114. The resulting TBs 336 are then scanned to form an ordered list of residual coefficients as part of the operation of entropy coding module 338. Equivalent processing is performed in video decoder 134 to obtain TBs from bitstream 133.

[0132] Figure 8The syntax structure 800 of a bitstream 801 with one or more stripes is shown. Each stripe includes multiple coding units. The bitstream 801 encodes image frame data 113 and is segmented into multiple coding tree units, each coding tree unit being segmented into one or more coding units, and each coding unit being further segmented into one or more transform blocks. The bitstream 801 may be generated by a video encoder 114 (e.g., as bitstream 115) or may be parsed by a video decoder 134 (e.g., as bitstream 133). The bitstream 801 is segmented into multiple parts, such as Network Abstraction Layer (NAL) units, wherein depiction is achieved by setting a NAL unit header, such as 808, before each NAL unit. The NAL unit header includes a NAL unit type that identifies the content of the subsequent NAL unit. A Video Parameter Set (VPS) 810 has a NAL unit type named "VPS_NUT" and includes parameters applicable to all layers of the bitstream. The VPS 810 may include a profile_tier_level syntax structure 830. Structure 830 specifies the profile of the bitstream using the "general_profile_idc" syntax element, and the general_constraint_info syntax structure 832 specifies the sub-profiles of the selected profile (if they exist). The general_constraint_info syntax structure 832 includes the flag no_luma_transform_size_64_constraint_flag 832a and the codeword max_log2_ctu_size_constraint_idc 832b. Flag 832a provides a maximum transform block size constraint (e.g., 64) for the bitstream, thus providing a high-level indication (within the context of the bitstream's structure) of the tools needed to encode or decode the bitstream. The 64-point transform differs from other transforms in that only the first 32 residual coefficients are scanned and encoded. For example, a 64×64TB may have only valid (non-zero) residual coefficients in the top-left 32×32 region. Flag 832a indicates that the maximum major transform size is constrained to 32 points horizontally and vertically, or to the left without constraint, in which case the maximum major transform size supported by the VVC standard is 64 points horizontally and vertically. Flag 832a is a fixed-length codeword or a flag typically 1 bit in size. It can also be implemented in a similar manner to impose constraints on the maximum transform size to smaller values ​​such as 16 points or 8 points. Similarly, codeword 832b provides a maximum CTU size constraint for the bitstream, thus providing high-level indication of the tools needed to encode or decode the bitstream. Codeword 832b is a fixed-length codeword typically 2 bits in size. Structure 832 defines specific sub-files associated with the implementation of video encoder 114 and video decoder 134. Syntactic structure 832 includes no_scaling_list_constraint_flag 832c.Flag 832c, when active (value equal to 1), indicates that the scaling list cannot be used in the bitstream.

[0133] The Sequence Parameter Set (SPS) 812 has a NAL unit type named "SPS_NUT" and may also include a profile_tier_level syntax structure 830. The profile_tier_level syntax structure 830 is included in either VPS 810 or SPS 812, but not in either. The Sequence Parameter Set (SPS) 812 defines sequence-level parameters such as the profile (toolset) used for encoding and decoding the bitstream, chroma format, sample bit depth, and frame resolution. SPS 812 also specifies which encoding tools can be used in a particular bitstream; this selection indicates a subset of tools available for the profile and sub-profile. Structure 834 shows an example of tools available for a sub-profile. The flag sps_max_luma_transform_size64_flag 834a indicates whether a 64-pt primary transform can be used. If constraint flag 832a does not prohibit the use of a 64-pt primary transform, flag 834a can indicate only the use of a 64-pt primary transform. The Sps_log2_ctu_size_minus5 codeword 834b uses a two-bit fixed-length codeword to indicate the size of the CTU (i.e., the maximum coding unit size). Codeword 834b can have values ​​of 0, 1, or 2, indicating CTU sizes of 32×32, 64×64, or 128×128, respectively. The value 3 is reserved in the initial (“Version 1”) grade of the VVC. Codeword 834b can not exceed the value of 832b, thus 832b imposes a constraint on the CTU size as part of the subgrade definition. The sps_explicit_scaling_list_enabled_flag 834c indicates whether a scaling list is allowed in the bitstream to feed non-uniform quantization within a fixed TB. When flag 832c is active (e.g., a value equal to 1), flag 834c needs to be inactive (a value equal to 0).

[0134] Flag 834a indicates the maximum enabled transform block size of the bitstream. Flag 834a is constrained based on the corresponding constraint flag 832a, but not adjusted (set) based on constraint flag 832a. In a compliant bitstream, the maximum transform block size enabled by flag 834a corresponds to the constraint set by constraint flag 832a, for example, having a value less than or equal to the maximum value of constraint flag 832a. Flag 834a is a fixed-length codeword or a flag typically 1 bit in size.

[0135] Similarly, codeword 834b indicates the maximum enabled CTU size of the bitstream. Codeword 834b is constrained based on the corresponding constraint codeword 832b, but not adjusted (set) based on the constraint codeword 832b. In a conforming bitstream, the maximum CTU size enabled by codeword 834b corresponds to the constraint set by codeword 832b, for example, having a value less than or equal to the maximum value of codeword 832b. Flag 834b is typically a fixed-length codeword of 2 bits.

[0136] The Adaptive Parameter Set (APS) 813 is encoded using a NAL unit type named "PREFIX_APS_NUT" before frames, or using a NAL unit type named "SUFFIX_APS_NUT" after frames (not shown). Multiple APSs (not shown) can be included between frames in the bitstream 800. Each APS (e.g., 813) includes parameters for configuring one of three encoding tools: a scaling list, ALF filter parameters, and LMCS model parameters. Which of the three encoding tools is configured in a given APS is specified by the "aps_params_type" codeword included in the corresponding APS. The aps_params_type codeword uses a three-bit fixed-length codeword, where values ​​0-2 are used for the three tools mentioned above, and values ​​3-7 are reserved for future use. A syntax element aps_params_type equal to 0 is named "ALF_APS" and indicates that APS contains parameters for the adaptive loop filter; an aps_params_type equal to 1 is named "LMCS_APS" and indicates that APS contains parameters for the luminance model chromaticity scaling tool; and an aps_params_type equal to 2 is named "SCALING_APS" and indicates that APS contains parameters for scaling the list.

[0137] The sequence of stripes that form an image is called an Access Unit (AU), such as AU 0 814. AU 0 814 includes three stripes, such as stripes 0 to 2. Strip 1 is labeled 816. Like the other stripes, strip 1 (816) includes a strip header 818 and strip data 820.

[0138] Figure 9The syntax structure 900 of striped data 820 of bitstream 801 (e.g., 115 or 133) is shown. CTU 910 includes one or more CUs, exemplified as CU 914. The size of each CTU is set by codeword 834b, the value of which is constrained by constraint 832b. CU 914 includes a prediction mode (not shown) signaled by a signal, followed by a transform tree 916. If the size of CU 914 does not exceed the maximum transform size (32 points horizontally or 64 points vertically), then the transform tree 916 includes a transform unit. If the size of a CU (e.g., CU 914) does exceed the maximum transform size (32 points horizontally or 64 points vertically), then the transform tree 916 includes multiple TUs arranged spatially in blocks and stored sequentially in the bitstream, exemplified as TUs 918a and 918b.

[0139] Figure 10 A method 1000 for encoding frame data 113 into bitstream 115 is illustrated, bitstream 115 comprising one or more stripes as a sequence of coding tree units. Method 1000 can be embodied by a device such as a configured FPGA, ASIC, or ASSP. Alternatively, method 1000 can be performed by a video encoder 114 under the execution of processor 205. Due to the workload of encoding frames, the steps of method 1000 can be performed on different processors to distribute the workload, for example using modern multi-core processors, such that different stripes are encoded by different processors. The resulting bitstream 115 can conform to sub-grades including constraints on various aspects of the block structure, including maximum transform size and maximum CTU size. Method 1000 can be stored on a computer-readable storage medium and / or in memory 206.

[0140] Method 1000 begins with encoding constraint parameters step 1010. At step 1010, the video encoder 114 encodes the profile_tier_level syntax structure 830, which contains the general_constraint_info syntax structure 832, into the VPS 810 or SPS 812 as a fixed-length sequence of encoding parameters in bitstream 115. Constraints on the maximum transform size (832a) and the maximum CTU size (832b) are encoded as part of the general_constraint_info syntax structure 832, thus contributing to the definition of sub-profiles in bitstream 115. Constraints on the use of scaling lists are encoded as part of the general_constraint_info syntax structure 832 with flag 832c, which also contributes to the definition of sub-profiles in bitstream 115.

[0141] Method 1000 proceeds from step 1010 to the coding block structure parameter step 1015. At step 1015, the video encoder 114 encodes the selected maximum transform size of the bitstream as flag 834a in SPS 812, and the selected CTU size of the bitstream 115 as codeword 834b in SPS 812. Flags 834a and codeword 834b are encoded regardless of the values ​​of the corresponding constraint flags (i.e., 832a and 832b, respectively). However, flags 834a and codeword 834b are prohibited from indicating a higher capability (larger transform size or larger CTU size) than constrained by flags 832a and codeword 832b. At step 1015, the video encoder 114 encodes the use of the scaling list by encoding flag 834c, which occurs regardless of the value of the corresponding constraint flag 832c. However, the enable flag 834c is constrained by the constraint flag 832c, which means that if the constraint flag 832c prohibits the use of the scaling list, the scaling list may not be enabled by the flag 834c.

[0142] Method 1000 continues from step 1015 to step 1020, which involves segmenting the frame into strips. In the execution of step 1020, processor 205 segments the frame data 113 into one or more strips or continuous portions. Where parallelism is required, individual instances of video encoder 114 encode the individual strips to some extent independently. A single video encoder 114 can process the individual strips sequentially, or it can achieve some intermediate level of parallelism. Typically, segmenting the frame into strips (continuous portions) is aligned with the boundaries of regions that segment the frame into, such as "sub-pictures" or blocks.

[0143] Method 1000 continues from step 1020 to step 1030, which encodes the strip header. At step 1030, the entropy encoder 338 encodes the strip header 818 into bitstream 115.

[0144] Method 1000 continues from step 1030 to step 1040, which involves segmenting the strip into CTUs. In step 1040, the video encoder 114 segments the strip 816, for example, into a sequence of CTUs. The strip boundaries are aligned with the CTU boundaries, and the CTUs within the strip are ordered according to the CTU scan order (typically the raster scan order). Segmenting the strip into CTUs determines which portion of the frame data 113 will be processed by the video encoder 113 when encoding the current strip.

[0145] Method 1000 continues from step 1040 to step 1050, which determines the coding tree. At step 1050, the video encoder 114 determines the coding tree of the currently selected CTU in the strip. Method 1000 starts with the first CTU in strip 816 on the first call to step 1050 and progresses to subsequent CTUs in strip 816 on subsequent calls. In determining the coding tree of the CTU, various combinations of quadtrees, binary splits, and ternary splits are generated and tested by the block partitioner 310.

[0146] Method 1000 continues from step 1050 to step 1060, which determines the coding unit. At step 1060, the video encoder 114 performs an operation to determine the encoding of the CUs obtained from various coding trees in the evaluation using known methods. Determining the encoding involves, for example, determining the prediction mode (e.g., intra-frame prediction with a specific mode or inter-frame prediction with motion vectors) and transform selection (primary transform type and optional secondary transform type) based on coding cost. If the primary transform type of the luma TB is determined to be DCT-2, or if any quantized primary transform coefficients that do not undergo a positive secondary transform are valid, the secondary transform index of the luma TB can indicate the application of the secondary transform. Otherwise, the secondary transform index of the luma indicates bypassing the secondary transform. For the luma channel, the primary transform type is determined to be one of the MTS options for the chroma channel, DCT-2, or transform skip, with DCT-2 being the available transform type. When determining the individual coding units, the optimal coding tree is also determined jointly. When encoding the coding unit using intra-frame prediction, the luma intra-frame prediction mode and chroma intra-frame prediction are determined.

[0147] Method 1000 continues from step 1060 to step 1070 of encoding the coding unit. At step 1070, the video encoder 114 encodes the coding unit determined in step 1060 into bitstream 115.

[0148] Method 1000 continues from step 1070 to step 1080, which is the last coding unit test. At step 1080, processor 205 tests whether the current coding unit is the last coding unit in the CTU. If not (step 1080 is "No"), control in processor 205 proceeds to step 1060, which determines the coding unit. Otherwise, if the current coding unit is the last coding unit (step 1080 is "Yes"), control in processor 205 proceeds to step 1090, which is the last CTU test.

[0149] At step 1090 of the final CTU test, processor 205 tests whether the current CTU is the last CTU in stripe 816. If it is not the last CTU in stripe 816 (step 1090 is "No"), control in processor 205 returns to step 1050 of determining the coding tree. Otherwise, if the current CTU is the last (step 1090 is "Yes"), control in processor 205 proceeds to step 10100 of the final stripe test.

[0150] At step 10100 of the last stripe test, processor 205 tests whether the current stripe being encoded is the last stripe in the frame. If it is not the last stripe (step 10100 is "No"), control in processor 205 proceeds to step 1030, which encodes the stripe header. Otherwise, if the current stripe is the last and all stripes (continuous portions) have been encoded (step 10100 is "Yes"), method 1000 terminates.

[0151] Figure 11 A method 1100 for decoding frames from a bitstream that is a sequence of coding units arranged in stripes is illustrated. Method 1100 can be embodied by a device such as a configured FPGA, ASIC, or ASSP. Alternatively, method 1100 can be performed by a video decoder 134 under the execution of processor 205. Therefore, method 1100 can be stored on a computer-readable storage medium and / or in memory 206.

[0152] Method 1100 decodes bitstreams encoded using Method 1000, where the sub-levels of bitstream 115 include constraints on block size, such as maximum transform size and maximum CTU size. The maximum transform size applies to both luma and chroma, regardless of the chroma format of the video data. The CTU size indicates the area covered by the root node of the coding tree, and the root node of the coding tree always applies to both the luma and chroma channels, regardless of subsequent splitting into separate coding trees for luma and chroma (which may appear deeper within the CTU coding tree).

[0153] Method 1100 begins with the step 1110 of decoding constraint parameters. In the execution of step 1110, the video decoder 134 decodes the general_constraint_info syntax structure 832 from the VPS 810 or SPS 812 present in the bitstream 133 as a fixed-length parameter sequence to determine the sub-grades of the bitstream 133. Constraints on the maximum transform size (832a) and the maximum CTU size (832b) are decoded as part of the general_constraint_info syntax structure 832, thus contributing to the determination of the sub-grades of the bitstream 133. For example, step 1110 may decode the maximum transform block size constraint and / or the maximum CTU size constraint from the bitstream. Constraints on the use of scaling lists are decoded as part of the general_constraint_info syntax structure 832 with flag 832c, also contributing to the definition of the sub-grades of the bitstream 133.

[0154] Method 1100 proceeds from step 1110 to the decoding block structure parameter step 1120. At step 1120, the video decoder 134 decodes the selected maximum enabled transform size of the bitstream into flag 834a from SPS 812, and decodes the selected maximum enabled CTU size of the bitstream 115 into codeword 834b from SPS 812. Flags 834a and codeword 834b are decoded regardless of the corresponding constraint flags (i.e., 832a and 832b, respectively). However, flags 834a and codeword 834b are disabled (in the "compliant" bitstream) to indicate a higher capability (larger transform size or larger CTU size) than constrained by flags 832a and codeword 832b. At step 1125, the video decoder 134 decodes the use of the scaling list by decoding flag 834c, where the decoding of flag 834c occurs regardless of the value of the corresponding constraint flag 832c. However, the enable flag 834c is constrained by the constraint flag 832c, such that if the constraint flag 832c prohibits the use of the scaling list, then the scaling list may not be enabled by the flag 834c. If the decoded block structure parameters violate the prohibitions defined in the general_constraint_info syntax structure 832, the video decoder 134 considers the bitstream to be "inconforming" and may stop further decoding.

[0155] Method 1100 continues from step 1120 to step 1130, which involves decoding the stripe header. At step 1130, the entropy decoder 420 decodes the stripe header 818 from bit stream 133.

[0156] Method 1100 continues from step 1130 to step 1140, which involves segmenting the stripe into CTUs. At step 1140, the video decoder 134 segments the stripe 816 into a sequence of CTUs. The stripe boundaries are aligned with the CTU boundaries, and the CTUs within the stripe are ordered according to the CTU scan order. The CTU scan order is typically the raster scan order. Segmenting the stripe into CTUs determines which portion of the frame data 113 will be processed by the video decoder 134 when decoding the current stripe. The stripe is segmented into CTUs based on the maximum enabled CTU size after decoding.

[0157] Method 1100 continues from step 1140 to step 1150, which involves decoding the encoding tree. During the execution of step 1150, the video decoder 133, on the first invocation of step 1150, decodes the encoding tree of the current CTU in the slice 816 from the bit stream 133, starting from the first CTU in the slice 816. This is achieved by... Figure 6 Furthermore, the encoding tree of the CTU is decoded by decoding the split flag at entropy decoder 420 based on the maximum enabled CTU size. In subsequent iterations of step 1150, subsequent CTUs in stripe 816 are decoded for the CTU.

[0158] Method 1100 continues from step 1160 to step 1170 of decoding the coding unit. At step 1170, video decoder 134 decodes the coding unit from bitstream 133. Each coding unit is decoded or determined according to a corresponding CTU, which is determined based on the maximum enabled coding tree size after decoding and the split flag decoded from the bitstream.

[0159] Method 1100 continues from step 1110 to step 1180, which is the last coding unit test. At step 1180, processor 205 tests whether the current coding unit is the last coding unit in the CTU. If it is not the last coding unit (step 1180 is "No"), control in processor 205 returns to step 1170, which decodes the next coding unit in the coding tree unit. If the current coding unit is the last coding unit (step 1180 is "Yes"), control in processor 205 proceeds to step 1190, which is the last CTU test.

[0160] At step 1190 of the final CTU test, processor 205 tests whether the current CTU is the last CTU in stripe 816. If it is not the last CTU in the stripe (step 1190 is "No"), control in processor 205 returns to step 1150 of decoding the coding tree to decode the next coding tree unit of stripe 816. If the current CTU is the last CTU in stripe 816 (step 1190 is "Yes"), control in processor 205 proceeds to step 11100 of the final stripe test.

[0161] At the last stripe test step 11100, processor 205 tests whether the currently being decoded stripe is the last stripe in the frame. If it is not the last stripe in the frame (step 11100 is "No"), control in processor 205 returns to step 1130, which decodes the stripe header, and step 1130 operates to decode the next stripe in the frame (e.g., ...). Figure 11 The header of the stripe ("strip 2"). If the current stripe is the last stripe in the frame ("Yes" in step 1100), then method 1100 terminates.

[0162] Steps 1140 to 11100 perform operation 1 to decode the image frame by determining the transform blocks of each coding unit of the coding tree unit based on the decoded maximum enabled transform block size and / or the decoded maximum enabled CTU size and the split flag decoded from the bitstream. (See also: Regarding...) Figure 1 The apparatus 130 described herein operates in a method 1610 for multiple coding units to generate an image frame.

[0163] The arrangement of the video encoder 114 using method 1000 and the video decoder 134 using method 1100 enables sub-grade definitions with granularity including block structure aspects, namely, the maximum transform size and the CTU size (which corresponds to the maximum CU size). Control over the block structure aspects using maximum transform size constraints and / or maximum CTU size constraints means that the sub-grade provides granular control over the standard block structure behavior affecting all color channels, regardless of the chroma format used. General constraints are used in association with the sequence-level toolset to constrain the maximum transform size and / or maximum CTU size, allowing early knowledge during decoding to determine which tools are needed. The constraint flags are located at a fixed position relative to the start of the VPS 810 or SPS 812, thus allowing the grade and sub-grade of the bitstream to be determined without requiring variable-length decoding. Furthermore, problematic implementations or tools can be disabled without affecting other aspects of the implementation (e.g., other encoding tools) and without creating ad hoc or non-standard sub-grades. Therefore, vendors implementing the VVC standard have greater flexibility in implementing video encoders and decoders that are suitable for and applicable to real-world applications and products.

[0164] In the arrangement of the video encoder 114 and the video decoder 134, a one-bit flag is used to constrain the maximum coding unit size. When active, the flag limits the CTU size to 64×64 and allows a CTU size of 128×128 when inactive.

[0165] In another arrangement of video encoder 114 and video decoder 134, "no_scaling_list_constraint_flag" (832c) also exists in general_constraint_info() 832, and is encoded by video encoder 114 at step 1010 and decoded by video decoder 134 at step 1110. Video encoder 114 encodes sps_explicit_scaling_list_enabled_flag (834c) in SPS 812, thereby indicating whether the scaling list will be used for quantization / inverse quantization. Video decoder 134 parses sps_explicit_scaling_list_enabled_flag 834c from SPS 812 to determine whether the scaling list should be utilized for inverse quantization performed by inverse quantizer module 428. If no_scaling_list_constraint_flag 832c is active (e.g., its value equals 1), then sps_explicit_scaling_list_enabled_flag 834c, encoded in SPS 812, indicates that the scaling list is not used (its value equals 0). If no_scaling_list_constraint_flag 832c indicates that the scaling list is not used (its value equals 1), then aps_params_type is prohibited from having a value of 2 (“SCALING_APS”). When no_alf_constraint_flag, encoded in general_constraint_info() 832, indicates that the adaptive loop filter is not in use (its value equals 1), then aps_params_type with a value of 0 (“ALF_APS”) for any AP associated with the bitstream (e.g., APS 813) is prohibited from having a value of 0. When the no_lmcs_constraint_flag, encoded in general_constraint_info()832, indicates that luminance model chroma scaling will not be used (value equal to 1), it is prohibited for any APS (e.g., APS 813) associated with the bitstream to have a value of 1 in aps_params_type (“LMCS_APS”).

[0166] Industrial availability

[0167] The described arrangement is applicable to the computer and data processing industry, and particularly suitable for digital signal processing that encodes or decodes signals such as video and image signals, thereby achieving high compression efficiency. Providing one or more of the constraint flags described above allows selection of a subset of tools (“sub-grades”) of a given grade. The selection of this tool subset offers benefits such as implementation advantages for VVC vendors, as vendors can specify a subset of grades that excludes unnecessary or otherwise problematic (e.g., from a complexity perspective) encoding tools.

[0168] The foregoing only illustrates some embodiments of the present invention, and modifications and / or changes can be made to the present invention without departing from the scope and spirit of the present invention, wherein these embodiments are merely exemplary and not restrictive.

[0169] Appendix A

[0170] Examples of the working draft text of the VVC standard are suitable for the methods described herein.

[0171] 7.3.3.2 General Constraint Information Syntax

[0172]

[0173]

[0174]

[0175] 7.4.4.2 General Constraint Information Semantics

[0176] …

[0177] max_log2_ctu_size_constraint_idc specifies that sps_log2_ctu_size_minus5 should be in the range of 0 to max_log2_ctu_size_constraint_idc (inclusive).

[0178] The value of no_luma_transform_size_64_constraint_flag, which is equal to 1, specifies that sps_max_luma_transform_size_64_flag should be equal to 0.

[0179] A value of 0 for no_luma_transform_size_64_constraint_flag does not impose such a constraint.

[0180] A `no_scaling_list_constraint_flag` value of 1 specifies that `sps_explicit_scaling_list_enabled_flag` should be equal to 0 and `aps_params_type` should not be equal to 2. A `no_scaling_list_constraint_flag` value of 0 does not impose such a constraint.

[0181] …

[0182] The value of no_alf_constraint_flag, which is 1, specifies that sps_alf_enabled_flag should be 0 and aps_params_type should not be 0.

[0183] A no_alf_constraint_flag value of 0 does not impose such a constraint.

[0184] …

[0185] A `no_lmcs_constraint_flag` value of 1 specifies that `sps_lmcs_enabled_flag` should be equal to 0, and `aps_params_type` should not be equal to 0. A `no_lmcs_constraint_flag` value of 0 does not impose such a constraint.

Claims

1. A method for decoding an image frame from a bitstream, the image frame being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more coding units, the method comprising: Decode the first information of the encoding tree unit size constraint from the general constraint information syntax structure included in the profile_tier_level syntax structure in the bit stream; Decode second information from the set of sequence parameters in the bit stream, indicating a value for specifying the size of the coding tree unit, wherein the value is in the range of 0 to an integer value determined based on the first information; One or more coding units among the plurality of coding tree units are determined based on the second information of the coding tree unit size and the split flag decoded from the bit stream; Decode the first flag of the constraint for the use of the scaling list from the general constraint information syntax structure included in the profile_tier_level syntax structure in the bit stream; Regardless of the value of the first flag, a second flag enabling the use of the scaling list is decoded from the sequence parameter set in the bitstream, wherein, when the value of the first flag is 1, the value of the second flag is constrained by the first flag, such that the second flag indicates that the use of the scaling list is disabled; and The one or more coding units are decoded from the bit stream using scaling based on the value of the second flag.

2. The method according to claim 1, wherein, The first information of the coding tree unit size constraint uses a fixed-length codeword of 1 bit.

3. The method according to claim 1, wherein, The second information regarding the size of the coding tree unit uses a fixed-length codeword of 1 bit.

4. The method according to claim 1, wherein, The second information regarding the size of the coding tree unit is decoded from the sequence parameter set rbsp syntax structure in the bitstream.

5. A video decoding apparatus for decoding image frames from a bitstream, the image frames being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more coding units, the video decoding apparatus comprising: The first decoding unit is configured to decode first information of the coding tree unit size constraint from the general constraint information syntax structure included in the profile_tier_level syntax structure in the bit stream; The second decoding unit is configured to decode second information from a set of sequence parameters in the bit stream, indicating a value for specifying the size of the coding tree unit, wherein the value is in the range of 0 to an integer value determined based on the first information; A determining unit is configured to determine one or more coding units among the plurality of coding tree units based on second information about the size of the coding tree unit and a split flag decoded from the bit stream; The third decoding unit is configured to decode a first flag of the constraint for the use of the scaling list from the general constraint information syntax structure included in the profile_tier_level syntax structure in the bit stream. A fourth decoding unit is configured to decode, regardless of the value of the first flag, a second flag enabling the use of the scaling list from a set of sequence parameters in the bitstream, wherein, when the value of the first flag is 1, the value of the second flag is constrained by the first flag, such that the second flag indicates that the use of the scaling list is disabled; and The fifth decoding unit is configured to decode the one or more encoding units from the bit stream using scaling based on the value of the second flag.

6. A non-transitory computer-readable storage medium storing a program that, when executed by a computer, implements the steps of the method according to any one of claims 1 to 4.

7. A method for encoding an image frame in a bitstream, wherein the image frame is segmented into a plurality of coding tree units, and each coding tree unit is segmented into one or more coding units, the method comprising: The first information of the coding tree unit size constraint is encoded in the general constraint information syntax structure included in the profile_tier_level syntax structure in the bit stream; The second information, which indicates the value used to specify the size of the coding tree unit, is encoded in the set of sequence parameters in the bit stream, wherein the value is in the range of 0 to an integer value determined based on the first information; The first flag indicating the constraint on the use of the scaling list is encoded in the general constraint information syntax structure in the bit stream; A second flag indicating the enable of the scaling list is encoded in the sequence parameter set in the bit stream, the second flag being constrained by the first flag, wherein the second flag is signaled regardless of the value of the first flag; One or more coding units are encoded in the bit stream using scaling based on the value of the second flag; and Encode each coding unit in one or more coding units.

8. The method according to claim 7, wherein, The first information of the coding tree unit size constraint uses a fixed-length codeword of 1 bit.

9. The method according to claim 7, wherein, The second information regarding the size of the coding tree unit uses a fixed-length codeword of 1 bit.

10. The method according to claim 7, wherein, The second information about the size of the coding tree unit is encoded in the sequence parameter set rbsp syntax structure in the bit stream.

11. A video encoding apparatus for encoding image frames in a bitstream, the image frames being segmented into a plurality of coding tree units, each coding tree unit being segmented into one or more coding units, the video encoding apparatus comprising: The first coding unit is configured to encode first information of the coding tree unit size constraint in the general constraint information syntax structure included in the profile_tier_level syntax structure in the bit stream; The second encoding unit is configured to encode second information, which indicates a value for specifying the size of the encoding tree unit, into a set of sequence parameters in the bit stream, wherein the value is in the range of 0 to an integer value determined based on the first information; The third encoding unit is configured to encode a first flag indicating a constraint on the use of the scaling list in the general constraint information syntax structure in the bit stream; A fourth encoding unit is configured to encode a second flag indicating the enabling of the scaling list into the set of sequence parameters in the bit stream, the second flag being constrained by the first flag, wherein the second flag is signaled regardless of the value of the first flag; A fifth encoding unit, configured to encode one or more encoding units in the bitstream using scaling based on the value of the second flag; and The sixth coding unit is configured to encode the one or more coding units.

12. A non-transitory computer-readable storage medium storing a program that, when executed by a computer, implements the steps of the method according to any one of claims 7 to 10.

13. A computer program product comprising a program that, when executed by a computer, implements the steps of the method according to any one of claims 1 to 4 and 7 to 10.

Images