Sub-block coding with generalized intra prediction in video coding
By grouping predictive blocks into independent PBGs and reconstructing samples within each group independently, the solution addresses inefficiencies in video coding, particularly with narrow blocks, enhancing coding efficiency and reducing delays.
Patent Information
- Application Number
- JP2021549317
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-02-25
- Filing Date
- 2020-02-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-02-26
AI Technical Summary
Existing video coding techniques face inefficiencies in handling narrow predictive blocks due to excessive memory access overhead and delays in reconstruction processes, particularly when using intra-subpartition coding with vertical or horizontal partitions.
The proposed solution involves grouping predictive blocks into independent predictive block groups (PBGs) and reconstructing samples within each group independently, allowing for vertical, horizontal, or combined partitioning of coding units using intra-subpartitioning (ISP) to reduce dependency on other blocks and optimize memory access.
This approach reduces memory access overhead and reconstruction delays, enhancing the coding efficiency and performance of video encoders and decoders by allowing parallel processing of predictive blocks, thereby improving overall video coding efficiency.
Smart Images

Figure 0007579794000010 
Figure 0007579794000011 
Figure 0007579794000012
Abstract
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001]
[0001] This application claims priority to U.S. Provisional Patent Application No. 62 / 812,078, filed February 28, 2019, U.S. Provisional Patent Application No. 62 / 817,474, filed March 12, 2019, and U.S. Provisional Patent Application No. 62 / 862,936, filed June 18, 2019, which claims the benefit of U.S. Provisional Patent Application No. 16 / 800,467, filed February 25, 2020, the contents of each of which are incorporated by reference in their entirety. [Technical field]
[0002]
[0002] This disclosure relates to video encoding and decoding.
[0003]
[0003] Digital video capabilities may be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio telephones, so-called "smartphones," video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), and extensions to such standards. Video devices may more efficiently transmit, receive, encode, decode, and / or store digital video information by implementing such video coding techniques.
[0004]
[0004] Video coding (e.g., video encoding and / or video decoding) typically involves predicting blocks of video data either from already coded blocks of video data in the same picture (e.g., intra prediction) or from already coded blocks of video data in a different picture (e.g., inter prediction). Video coding techniques include spatial (intra picture) prediction and / or temporal (inter picture) prediction to reduce or remove redundancy inherent in a video sequence. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be partitioned into video blocks, which may also be referred to as coding tree units (CTUs), coding units (CUs) and / or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. A picture is sometimes referred to as a frame, and a reference picture is sometimes referred to as a reference frame.
[0005]
[0005] Generally, this disclosure describes techniques related to coding multiple predictive blocks into predictive block groups using intra subpartition (ISP) coding. In particular, a video coder (e.g., a video encoder and / or a video decoder) may group multiple predictive blocks, such as multiple sub-blocks of a coding unit, into a predictive block group. For example, a single predictive block group may include more than one predictive block or sub-block of a coding unit. In one example, a first predictive block group may include a first sub-block of a coding unit and a second sub-block of a coding unit.
[0006]
[0006] In some examples, the video coder may reconstruct samples of predictive blocks included in a first predictive block group independent of reconstruction of samples of other predictive blocks included in the same predictive block group. For example, the video coder may reconstruct samples of one predictive block before, without reconstruction, or in parallel with reconstruction of samples of another predictive block included in the same predictive block group. In some cases, the predictive block grouping techniques of this disclosure may be implemented when using an instance of intra-subpartition coding to include partitioning of coding units, such as vertical partitioning, horizontal partitioning, a combination of horizontal and vertical partitioning, etc. Furthermore, the predictive block grouping techniques of this disclosure may be implemented when predictive blocks have relatively narrow dimensions (e.g., narrow height, narrow width, etc.) such that the dimensions of the predictive blocks meet a predefined threshold for predictive block grouping. One or more techniques described herein for grouping predictive blocks into predictive block groups may be applied to any existing video codec, such as the HEVC (High Efficiency Video Coding) or H.266 / Versatile Video Coding (VVC) standard, or may be used in any future video coding standard.
[0007]
[0007] According to one example, a method of coding video data is disclosed, the method including: performing partitioning of a coding unit (CU) of the video data using an intra subpartition (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block, grouping a plurality of predictive blocks from the set of predictive blocks into a first predictive block group (PBG), and reconstructing samples of the predictive blocks included in the first PBG independent of samples of other predictive blocks included in the first PBG.
[0008]
[0008] According to another example, a device for coding video data is disclosed, the device including: a memory configured to store the video data; and one or more processors implemented in a circuit, the one or more processors configured to perform partitioning of a coding unit (CU) of the video data using an intra subpartition (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block, group a plurality of predictive blocks from the set of predictive blocks into a first predictive block group (PBG), and reconstruct samples of the predictive blocks included in the first PBG independent of samples of other predictive blocks included in the first PBG.
[0009]
[0009] According to another example, a non-transitory computer-readable storage medium is disclosed that stores instructions that, when executed, cause one or more processors to perform partitioning of a coding unit (CU) of video data using an intra subpartition (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block, group a plurality of predictive blocks from the set of predictive blocks into a first predictive block group (PBG), and reconstruct samples of the predictive blocks included in the first PBG independent of samples of other predictive blocks included in the first PBG.
[0010]
[0010] According to another example, an apparatus for coding video data is disclosed, the apparatus including: means for performing partitioning of a coding unit (CU) of the video data using an intra subpartition (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block, means for grouping a plurality of predictive blocks from the set of predictive blocks into a first predictive block group (PBG), and means for reconstructing samples of the predictive blocks included in the first PBG independent of samples of other predictive blocks included in the first PBG.
[0011]
[0011] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will become apparent from the description, drawings, and claims. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a block diagram illustrating an example video encoding and decoding system, which may implement the techniques of this disclosure. [Diagram 2]
[0013] FIG. 2 is a diagram illustrating example intra-prediction modes. [Diagram 3]
[0014] FIG. 3 is a block diagram illustrating an example of an 8×4 rectangular block. [Figure 4A]
[0015] FIG. 4A is a block diagram providing an illustration of the mode mapping process for out-of-diagonal modes. [Figure 4B] FIG. 4B is a block diagram providing an illustration of the mode mapping process for out-of-diagonal modes. [Figure 4C] FIG. 4C is a block diagram providing an illustration of the mode mapping process for out-of-diagonal modes. [Diagram 5]
[0016] FIG. 5 is a block diagram illustrating the wide angle employed in VTM2. [Figure 6]
[0017] FIG. 6 is a block diagram illustrating the wider angle employed in VTM3. [Figure 7]
[0018] FIG. 7 is a table specifying a mapping table between predModeIntra and the angle parameter intraPredAngle in VTM3. [Figure 8]
[0019] FIG. 8 is a block diagram illustrating an example division of a block for intra subpartition coding. [Figure 9]
[0020] FIG. 9 is a block diagram illustrating an example division of a block for intra subpartition coding. [Figure 10]
[0021] FIG. 10 is a block diagram illustrating multiple reference line prediction. [Figure 11]
[0022] FIG. 11 is a conceptual diagram illustrating an example of how a coding block may be partitioned into four horizontal planar prediction blocks. [Figure 12]
[0023] FIG. 12 is a conceptual diagram illustrating an example of how a coding block may be partitioned into four vertical planar prediction blocks. [Figure 13]
[0024] FIG. 13 is a conceptual diagram illustrating an example of such a planar prediction block structure. [Figure 14]
[0025] FIG. 14 is a block diagram illustrating how a coding block is divided into four predictive blocks. [Figure 15]
[0026] FIG. 15 is a block diagram illustrating an example in which a 4×N coding block is coded with intra subpartitions and vertical partitions. [Figure 16]
[0027] FIG. 16 is a conceptual diagram illustrating an example in which an M×N coding block is divided into four predictive blocks. [Figure 17]
[0028] FIG. 17 is a conceptual diagram illustrating an example in which only vertical intra-mode prediction is allowed for a 4×N coding block that has been coded with intra subpartition coding (ISP) and vertical partitioning. [Figure 18]
[0029] FIG. 18 is a conceptual diagram illustrating an example in which an 8×N (N>4) coding block is coded with ISP and partitioned vertically. [Figure 19A]
[0030] FIG. 19A is a conceptual diagram illustrating an exemplary quaternary tree binary tree (QTBT) structure and a corresponding coding tree unit (CTU). [Figure 19B] FIG. 19B is a conceptual diagram illustrating an exemplary quaternary tree binary tree (QTBT) structure and corresponding coding tree unit (CTU). [Figure 20]
[0031] FIG. 20 is a block diagram illustrating an example video encoder that may perform the techniques of this disclosure. [Figure 21]
[0032] FIG. 21 is a block diagram illustrating an example video decoder that may perform the techniques of this disclosure. [Figure 22]
[0033] FIG. 22 is a flow chart illustrating an exemplary method for encoding a current block. [Diagram 23]
[0034] FIG. 23 is a flow chart illustrating an example method for decoding a current block. [Figure 24]
[0035] FIG. 24 is a flowchart illustrating an example method for performing predictive block grouping on a current block of video data. Detailed Description
[0013]
[0036] Generally, this disclosure describes techniques for improving the coding and / or decoding of predictive blocks using predictive block groups in intra subpartition (ISP) coding. Such techniques may be applied to current or future video coding standards, including the Versatile Video Coding (VVC) standard currently under development.
[0014]
[0037] In some examples, the video coder may code a block of video data using ISP coding. For example, the video coder may partition the block of video data into sub-blocks (e.g., predictive blocks). In some examples, the sub-blocks may have a relatively narrow size in one or more dimensions. In an illustrative example, a coding unit (CU), such as a coding block, may have a dimension of width (W) x height (H). In one example, W of a CU may be equal to 4 and H may be equal to N>0. In an illustrative example, the video coder may code 4xN CUs using ISP coding. For example, the video coder may partition the CU using vertical or horizontal partitioning. In another example, the video coder may use a combination of vertical and horizontal partitioning. In such a case, the video coder may partition a first portion or portions of a CU using vertical partitioning and partition a second portion or portions of the same CU using horizontal partitioning.
[0015]
[0038] In an example involving vertical partitioning, the resulting sub-blocks may have a size of 1×N. Additionally, N may be greater than 8 (e.g., H is greater than 8). In such an example, the video coder may utilize a row-wise store of samples. When the video coder uses a row-wise store of samples, narrow sub-blocks, such as those having dimensions below a predefined threshold, may result in the video coder performing an inefficient and time-consuming memory access procedure. For example, in such a case, the video coder may access (e.g., read) more samples from memory compared to the number of samples in the sub-block when performing the memory access procedure. When a block is of size M×N, accessing M samples in each row from memory may often involve overhead, such as accessing more than M samples due to limitations in the minimum number of samples per access. For example, a given memory access procedure may require the video coder to read samples of a first row before reading samples from another row. In such an example, the video coder may therefore read a minimum of X samples before accessing a target sample in one narrow sub-block. X is greater than the number of samples in the narrow sub-block. This problem is exacerbated when the number of rows is larger for the narrow sub-block due to the narrow shape of 1×N and 2×N sub-blocks. For example, a 4×4 CU may have only four rows. In such an example, the video coder may need to access 16 rows for a 1×16 sub-block, even though the sub-block has the same number of samples as a 4×4 CU. The resulting overhead may cause the video coder to experience delays during the coding process, especially in the worst case where there are several small coding blocks in a high solution sequence. In addition, the reconstruction of a sub-block in a CU depends on the reconstruction of other sub-blocks in the coding block that precede it in the decoding order.In such cases, sub-blocking of this nature may cause the video coder to experience increased delays in coding or decoding the CU.
[0016]
[0039] The above-mentioned problems can be addressed by the disclosed prediction block grouping technique, among others, by grouping prediction blocks (e.g., subblocks) into prediction block groups (PBGs) and reconstructing samples of a prediction block included in a PBG independently of samples of other prediction blocks included in the same PBG. Specifically, intra subpartitioning (ISP) may be used to divide a coding unit vertically, horizontally, or a combination of horizontal and vertical to form prediction blocks that may then be grouped into various PBGs.
[0017]
[0040] Various techniques in this disclosure may be described with reference to a video coder or with reference to video coding, which are intended to be general terms that may refer to a video encoder and either video coding or video decoder. Unless explicitly stated, it should not be assumed that a technique described with respect to a video encoder or video decoder cannot be performed by the other of the video encoder or video decoder. For example, in many instances, a video decoder performs the same, or sometimes the inverse, coding technique as a video encoder to decode encoded video data. In many instances, a video encoder also includes a video decoding loop, and thus a video encoder performs video decoding as part of encoding video data. Thus, unless explicitly stated, a technique described in this disclosure with respect to a video decoder may also be performed by a video encoder, and vice versa.
[0018]
[0041] 1 is a block diagram illustrating an example video encoding and decoding system 100 that may implement the techniques of this disclosure. The techniques of this disclosure are generally directed to coding (encoding and / or decoding) video data. In general, video data includes any data for processing video. Thus, video data may include raw uncoded video, encoded video, decoded (e.g., reconstructed) video, and video metadata such as signaling data.
[0019]
[0042] As shown in Figure 1, the system 100 includes an originating device 102 that, in this example, provides encoded video data to be decoded and displayed by a destination device 116. In particular, the originating device 102 provides the video data to the destination device 116 via a computer-readable medium 110. The originating device 102 and the destination device 116 may include any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, or the like. In some cases, the originating device 102 and the destination device 116 may be equipped for wireless communication and thus may be referred to as wireless communication devices.
[0020]
[0043] In the example of FIG. 1, source device 102 includes video source 104, memory 106, video encoder 200, and output interface 108. Destination device 116 includes input interface 122, video decoder 300, memory 120, and display device 118. According to this disclosure, video encoder 200 of source device 102 and video decoder 300 of destination device 116 may be configured to apply techniques for generalized intra prediction. Thus, source device 102 represents an example of a video encoding device, while destination device 116 represents an example of a video decoding device. In other examples, source device and destination device may include other components or configurations. For example, source device 102 may receive video data from an external video source, such as an external camera. Similarly, destination device 116 may interface with an external display device rather than including an integrated display device.
[0021]
[0044] The system 100 shown in FIG. 1 is only one example. In general, any digital video encoding and / or decoding device may perform the techniques for generalized intra prediction. The source device 102 and the destination device 116 are only examples of such coding devices, where the source device 102 generates coded video data for transmission to the destination device 116. This disclosure refers to a "coding" device as a device that performs coding (encoding and / or decoding) of data. Thus, the video encoder 200 and the video decoder 300 represent examples of coding devices, in particular video encoders and video decoders, respectively. In some examples, the devices 102, 116 may operate substantially symmetrically, such that each of the devices 102, 116 includes a video encoding component and a video decoding component. Thus, the system 100 may support one-way or two-way video transmission between the video devices 102 and 116, for example, video streaming, video playback, video broadcasting, or video telephony.
[0022]
[0045] Generally, the video source 104 represents a source of video data (i.e., raw, uncoded video data) and provides a sequential series of pictures (also referred to as "frames") of the video data to the video encoder 200, which encodes the data for the pictures. The video source 104 of the originating device 102 may include a video capture device such as a video camera, a video archive containing previously captured raw video, and / or a video feed interface for receiving video from a video content provider. As a further alternative, the video source 104 may generate computer graphics-based data as the source video, or a combination of live, archival, and computer-generated video. In each case, the video encoder 200 encodes the captured, pre-captured, or computer-generated video data. The video encoder 200 may reconstruct the pictures from the order in which they are received (sometimes referred to as "display order") into a coding order for coding. The video encoder 200 may generate a bitstream including the encoded video data. The source device 102 may then output the encoded video data via output interface 108 onto computer-readable medium 110 for receipt and / or retrieval by, for example, an input interface 122 of the destination device 116 .
[0023]
[0046] The memory 106 of the source device 102 and the memory 120 of the destination device 116 represent general purpose memories. In some examples, the memories 106, 120 may store raw video data, e.g., raw video from the video source 104 and raw decoded video data from the video decoder 300. Additionally or alternatively, the memories 106, 120 may store software instructions executable by, e.g., the video encoder 200 and the video decoder 300, respectively. In this example, the memories 106 and 120 are shown separate from the video encoder 200 and the video decoder 300, but it should be understood that the video encoder 200 and the video decoder 300 may also include internal memories for functionally similar or equivalent purposes. Additionally, the memories 106, 120 may store encoded video data, e.g., output from the video encoder 200 and input to the video decoder 300. In some examples, a portion of the memory 106, 120 may be allocated as one or more video buffers, for example, to store raw decoded and / or encoded video data.
[0024]
[0047] The computer-readable medium 110 may represent any type of medium or device capable of transferring encoded video data from the source device 102 to the destination device 116. In one example, the computer-readable medium 110 represents a communication medium that allows the source device 102 to transmit the encoded video data directly to the destination device 116 in real time, for example, via a radio frequency network or a computer-based network. The output interface 108 may modulate a transmission signal including the encoded video data, and the input interface 122 may demodulate a received transmission signal, according to a communication standard, such as a wireless communication protocol. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network, such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from the source device 102 to the destination device 116.
[0025]
[0048] In some examples, the computer-readable medium 110 may include a storage device 112. The source device 102 may output the encoded data to the computer-readable medium 110 (e.g., storage device 112) from an output interface 108. Similarly, the destination device 116 may access the encoded data from the computer-readable medium 110 (e.g., storage device 112) via an input interface 122. The storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a DVD, a CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.
[0026]
[0049] In some examples, the computer-readable medium 110 may include a file server 114 or another intermediate storage device that may store the encoded video data generated by the source device 102. The source device 102 may output the encoded video data to the file server 114 or another intermediate storage device that may store the encoded video generated by the source device 102. The destination device 116 may access the stored video data from the file server 114 via streaming or download. The file server 114 may be any type of server device capable of storing the encoded video and transmitting the encoded video to the destination device 116. The file server 114 may represent a web server (e.g., for a website), a file transfer protocol (FTP) server, a content delivery network device, or a network attached storage (NAS) device. The destination device 116 may access the encoded video data from the file server 114 through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on the file server 114. The file server 114 and the input interface 122 may be configured to operate according to a streaming transmission protocol, a download transmission protocol, or a combination thereof.
[0027]
[0050] Output interface 108 and input interface 122 may represent a wireless transmitter / receiver, a modem, a wired networking component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 comprise wireless components, output interface 108 and input interface 122 may be configured to transfer data, such as video data, encoded according to a cellular communication standard, such as 4G, 4G-LTE (Long Term Evolution), LTE Advanced, 5G, or the like. In some examples where output interface 108 comprises a wireless transmitter, output interface 108 and input interface 122 may be configured to transfer data, such as video data, encoded according to other wireless standards, such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee™), the Bluetooth™ standard, or the like. In some examples, source device 102 and / or destination device 116 may include respective system-on-chip (SoC) devices. For example, source device 102 may include an SoC device for performing functionality provided by video encoder 200 and / or output interface 108, and destination device 116 may include an SoC device for performing functionality provided by video decoder 300 and / or input interface 122.
[0028]
[0051] The techniques of this disclosure may be applied to video coding to support any of a variety of multimedia applications, such as over-the-air television broadcast, cable television transmission, satellite television transmission, Internet streaming video transmission such as Dynamic Adaptive Streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications.
[0029]
[0052] An input interface 122 of the destination device 116 receives an encoded video bitstream from a computer-readable medium 110 (e.g., a communication medium, a storage device 112, a file server 114, or the like). The encoded video bitstream may include signaling information defined by the video encoder 200 and also used by the video decoder 300, such as syntax elements having values that describe characteristics and / or processing of video blocks or other coded units (e.g., slices, pictures, groups of pictures, sequences, or the like). A display device 118 displays decoded pictures of the decoded video data to a user. The display device 118 may represent any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.
[0030]
[0053] 1, in some examples, the video encoder 200 and the video decoder 300 may be integrated with an audio encoder and / or an audio decoder, respectively, and may include appropriate MUX-DEMUX units or other hardware and / or software to handle multiplexed streams including both audio and video in a common data stream. If applicable, the MUX-DEMUX units may comply with the ITU H.223 multiplexer protocol or other protocols such as the User Datagram Protocol (UDP).
[0031]
[0054] The video encoder 200 and the video decoder 300 may each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or combinations thereof. When the techniques are implemented partially in software, the device may store instructions for the software in a suitable, non-transitory computer-readable medium and may use one or more processors to execute the instructions in hardware to perform the techniques of this disclosure. Each of the video encoder 200 and the video decoder 300 may be included in one or more encoders or decoders, and either the encoder or the decoder may be integrated as part of a combined encoder / decoder (CODEC) in the respective device. The device including the video encoder 200 and / or the video decoder 300 may comprise an integrated circuit, a microprocessor, and / or a wireless communication device, such as a cellular telephone.
[0032]
[0055] Video encoder 200 and video decoder 300 may operate according to a video coding standard, such as ITU-T H.265, also referred to as High Efficiency Video Coding (HEVC), or extensions to High Efficiency Video Coding (HEVC), such as multiview and / or scalable video coding extensions. Alternatively, video encoder 200 and video decoder 300 may operate according to other proprietary or industry standards, such as ITU-T H.266, also referred to as Joint Exploration Test Model (JEM) or Versatile Video Coding (VVC). A recent draft of the VVC standard is described in "Versatile Video Coding (Draft 4)" in Bross et al., Joint Video Experts Team (JVET) of ITU-T SG16WP3 and ISO / IEC JTC1 / SC29 / WG11, 13th Meeting: Marrakesh, Massachusetts, January 9-18, 2019, JVET-M1001-v2 (hereinafter "VVC Draft 4"). However, the techniques of this disclosure are not limited to any particular coding standard.
[0033]
[0056] Generally, the video encoder 200 and the video decoder 300 may perform block-based coding of pictures. The term “block” generally refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used in the encoding and / or decoding process). For example, a block may include a two-dimensional matrix of samples of luminance data and / or chrominance data. Generally, the video encoder 200 and the video decoder 300 may code video data represented in a YUV (e.g., Y, Cb, Cr) format. That is, rather than coding red, green, and blue (RGB) data for samples of a picture, the video encoder 200 and the video decoder 300 may code luminance and chrominance components, where the chrominance components may include both red and blue hues of chrominance components. In some examples, the video encoder 200 converts received RGB formatted data to a YUV representation prior to encoding, and the video decoder 300 converts the YUV representation to an RGB format. Alternatively, pre-processing and post-processing units (not shown) may perform these transformations.
[0034]
[0057] This disclosure may generally refer to coding (e.g., encoding and decoding) a picture to include processes of encoding or decoding data for a picture. Similarly, this disclosure may refer to coding a block of a picture to include processes of encoding or decoding data for a block, e.g., predictive and / or residual coding. An encoded video bitstream generally includes a set of values for syntax elements that represent coding decisions (e.g., coding modes) and partitioning of a picture into blocks. Thus, references to coding a picture or a block should generally be understood as coding values for the syntax elements that form the picture or block.
[0035]
[0058] HEVC defines various blocks, including coding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video coder (such as the video encoder 200) partitions coding tree units (CTUs) into CUs according to a quad-tree structure. That is, the video coder partitions CTUs and CUs into four equal, non-overlapping squares, and each node of the quad-tree has either zero or four child nodes. A node without children may be referred to as a "leaf node", and a CU of such a leaf node may include one or more PUs and / or one or more TUs. The video coder may further partition PUs and TUs. For example, in HEVC, a residual quad-tree (RQT) represents a partition of TUs. In HEVC, a PU represents inter-predicted data, while a TU represents residual data. An intra-predicted CU includes intra-prediction information, such as an intra-mode indication. This disclosure generally uses the term partition to represent a block as a TU, PU, or CU, or may represent multiple blocks or a region including multiple blocks.
[0036]
[0059] As another example, the video encoder 200 and the video decoder 300 may be configured to operate according to JEM or VVC. According to JEM or VVC, a video coder (such as the video encoder 200) partitions a picture into multiple coding tree units (CTUs). The video encoder 200 may partition the CTUs according to a tree structure, such as a quaternary tree-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure eliminates the concept of multiple partition types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels, a first level partitioned according to a quaternary tree partition, and a second level partitioned according to a binary tree partition. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to coding units (CUs).
[0037]
[0060] In the MTT partition structure, blocks may be partitioned using one or more types of quaternary tree (QT) partitions, binary tree (BT) partitions, and ternary tree (TT) partitions. A ternary tree partition is a partition in which a block is divided into three sub-blocks. In some examples, a ternary tree partition divides a block into three sub-blocks without dividing the original block through the center. The partition types in MTT (e.g., QT, BT, and TT) may be symmetric or asymmetric.
[0038]
[0061] In some examples, video encoder 200 and video decoder 300 may use a single QTBT or MTT structure to represent each of the luminance and chrominance components, while in other examples, video encoder 200 and video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luminance component and another QTBT / MTT structure for both chrominance components (or two QTBT / MTT structures for each chrominance component).
[0039]
[0062] Video encoder 200 and video decoder 300 may be configured to use HEVC-specific quad-tree partitioning, QTBT partitioning, MTT partitioning, or other partitioning structures. For purposes of illustration, the description of the techniques of this disclosure is presented with respect to QTBT partitioning. However, it should be understood that the techniques of this disclosure may also be applied to video coders configured to use quad-tree partitioning or other types of partitioning as well.
[0040]
[0063] This disclosure may use "NxN" and "N by N" interchangeably to refer to the sample dimensions of a block (such as a CU or other video block) in terms of vertical and horizontal dimensions, e.g., 16x16 samples or 16 by 16 samples. Generally, a 16x16 CU has 16 samples in the vertical direction (y=16) and 16 samples in the horizontal direction (x=16). Similarly, an NxN CU generally has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU may be arranged in rows and columns. Furthermore, a CU does not necessarily have to have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU may comprise NxM samples, where M is not necessarily equal to N.
[0041]
[0064] Video encoder 200 encodes video data for a CU that represents prediction and / or residual information as well as other information. The prediction information indicates how the CU should be predicted to form a predictive block for the CU. The residual information generally represents sample-by-sample differences between samples of the CU before encoding and the predictive block.
[0042]
[0065] To predict a CU, the video encoder 200 may form a predictive block for the CU, typically through inter prediction or intra prediction. Inter prediction generally refers to predicting a CU from data of a previously coded picture, while intra prediction generally refers to predicting a CU from previously coded data of the same picture. To perform inter prediction, the video encoder 200 may generate a predictive block using one or more motion vectors. The video encoder 200 may generally perform a motion search to identify a reference block that closely matches the CU, e.g., with respect to the difference between the CU and the reference block. The video encoder 200 may calculate a difference metric using a sum of absolute differences (SAD), a sum of squared differences (SSD), a mean absolute difference (MAD), a mean squared difference (MSD), or other such difference calculation to determine whether the reference block closely matches the current CU. In some examples, the video encoder 200 may predict the current CU using unidirectional prediction or bidirectional prediction.
[0043]
[0066] Some instances of JEM and VVC also provide an affine motion compensation mode, which may be considered an inter prediction mode. In an affine motion compensation mode, video encoder 200 may determine two or more motion vectors that represent non-translational motion, such as zoom-in or zoom-out, rotation, perspective motion, or other irregular motion types.
[0044]
[0067] To perform intra prediction, the video encoder 200 may select an intra prediction mode to generate a prediction block. Some examples of JEM and VVC provide 67 intra prediction modes, including planar and DC modes, along with various directional modes. In general, the video encoder 200 selects an intra prediction mode that describes samples neighboring a current block (e.g., a block of a CU) that predict samples of the current block. Assuming that the video encoder 200 codes the CTUs and CUs in raster scan order (left to right, top to bottom), such samples may generally be above, to the left, or to the left of the current block in the same picture as the current block.
[0045]
[0068] The video encoder 200 encodes data representing a prediction mode for the current block. For example, for an inter prediction mode, the video encoder 200 may encode motion information for the corresponding mode along with data representing which of various available inter prediction modes is used. For unidirectional or bidirectional inter prediction, for example, the video encoder 200 may encode motion vectors using advanced motion vector prediction (AMVP) or merge mode. The video encoder 200 may encode motion vectors for an affine motion compensation mode using an analogous mode.
[0046]
[0069] Following prediction, such as intra-prediction or inter-prediction, of a block, the video encoder 200 may compute residual data for the block. The residual data, such as a residual block, represents sample-by-sample differences between the block and a prediction block for the block formed using a corresponding prediction mode. The video encoder 200 may apply one or more transforms to the residual block to generate transformed data in a transform domain instead of the sample domain. For example, the video encoder 200 may apply a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. Additionally, the video encoder 200 may apply a secondary transform subsequent to the first transform, such as a mode-dependent non-separable quadratic transform (MDNSST), a signal-dependent transform, a Karhunen-Loeve transform (KLT), or the like. The video encoder 200 generates transform coefficients following application of the one or more transforms.
[0047]
[0070] As described above, following any transformation to generate transform coefficients, the video encoder 200 may perform quantization of the transform coefficients. Quantization generally refers to a process in which transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients and provide further compression. By performing a quantization process, the video encoder 200 may reduce the bit depth associated with some or all of the transform coefficients. For example, the video encoder 200 may truncate an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder 200 may perform a bitwise right shift of the value to be quantized.
[0048]
[0071] Following quantization, the video encoder 200 may scan the transform coefficients to generate a one-dimensional vector from the two-dimensional matrix including the quantized transform coefficients. The scan may be designed to place higher energy (and therefore lower frequency) transform coefficients at the front of the vector and lower energy (and therefore higher frequency) transform coefficients at the rear of the vector. In some examples, the video encoder 200 may utilize a predefined scan order to scan the quantized transform coefficients to generate a serialized vector and then entropy encode the quantized transform coefficients of the vector. In other examples, the video encoder 200 may perform an adaptive scan. After scanning the quantized transform coefficients to form the one-dimensional vector, the video encoder 200 may entropy encode the one-dimensional vector, for example, according to context-adaptive binary arithmetic coding (CABAC). The video encoder 200 may also entropy encode values for syntax elements describing metadata related to the encoded video data for use by the video decoder 300 in decoding the video data.
[0049]
[0072] To perform CABAC, the video encoder 200 may assign a context in a context model to a symbol to be transmitted. The context may relate, for example, to whether neighboring values of the symbol are zero-valued or not. A probability determination may be based on the context assigned to the symbol.
[0050]
[0073] Video encoder 200 may further generate syntax data, such as block-based syntax data, picture-based syntax data, and sequence-based syntax data, for example, in a picture header, in a block header, in a slice header, or in other syntax data, such as a sequence parameter set (SPS), a picture parameter set (PPS), or a video parameter set (VPS), to video decoder 300. Video decoder 300 may similarly decode such syntax data to determine how to decode the corresponding video data.
[0051]
[0074] In this manner, the video encoder 200 may generate a bitstream containing the encoded video data, e.g., including syntax elements describing partitions of a picture into blocks (e.g., CUs) and prediction and / or residual information for the blocks. Finally, the video decoder 300 may receive the bitstream and decode the encoded video data.
[0052]
[0075] In general, video decoder 300 performs a process that is the inverse of that performed by video encoder 200 to decode encoded video data of the bitstream. For example, video decoder 300 may decode values for syntax elements of the bitstream using CABAC in a manner that is substantially similar, albeit inverse, to the CABAC encoding process of video encoder 200. The syntax elements may specify partition information for partitioning a picture into CTUs and partitioning each CTU according to a corresponding partition structure, such as a QTBT structure, to specify CUs of the CTUs. The syntax elements may further specify prediction and residual information for blocks (e.g., CUs) of video data.
[0053]
[0076] The residual information may be represented, for example, by quantized transform coefficients. The video decoder 300 may inverse quantize and inverse transform the quantized transform coefficients of the block to reconstruct a residual block for the block. The video decoder 300 may form a predictive block for the block using the signaled prediction mode (intra-prediction or inter-prediction) and associated prediction information (e.g., motion information for inter-prediction). The video decoder 300 may then combine the predictive block and the residual block (on a sample-by-sample basis) to reconstruct the original block. The video decoder 300 may perform additional processing, such as performing a deblocking process, to reduce visual artifacts along block boundaries.
[0054]
[0077] According to techniques of this disclosure, video encoder 200 and / or video decoder 300 may perform generalized intra prediction. For example, video encoder 200 and / or video decoder 300 may code an indication of performance of generalized prediction for a current coding block of video data, partition the current coding block into one or more predictive blocks based on the indication, define one or more predictive block groups (PBGs) based on the one or more predictive blocks, determine a coding order for the PBGs, and code the PBGs in the coding order.
[0055]
[0078] In some examples, a video coder (e.g., video encoder 200 and / or video decoder 300) may perform partitioning of a coding unit (CU) of video data using intra subpartitioning (ISP) to form a set of predictive blocks. For example, the video coder may perform vertical or horizontal partitioning of the CU. In another example, the video coder may perform partitioning of the CU as a combination of vertical and horizontal partitioning of the CU. For example, the video coder may perform vertical partitioning on some parts of the CU and horizontal partitioning on other parts of the CU. In either case, the video coder may partition the CU via various partitioning or partitioning techniques to form a set of predictive blocks.
[0056]
[0079] In some examples, the predictive block may include at least a first predictive block and a second predictive block. In some examples, the CU may include a single predictive block. The predictive block may include a narrow vertical block, a narrow horizontal block, or a combination of a vertical block and a horizontal block. It should be noted that in this context, the term "narrow" generally includes non-square blocks (e.g., rectangular blocks, etc.).
[0057]
[0080] In addition, the video coder may group more than one of the sets of predictive blocks into a first predictive block group (PBG). In some examples, the PBG may include at least two predictive blocks. For example, the PBG may include one or more vertical predictive blocks, one or more horizontal predictive blocks, or one or more combinations of vertical and horizontal predictive blocks. In another example, the PBG may include a single predictive block. Furthermore, a CU may include PBGs of various sizes. For example, a single CU may include multiple PBGs, with a first PBG formed by grouping one or more predictive blocks (e.g., vertical blocks, horizontal blocks, or both vertical and horizontal blocks) and a second PBG formed by grouping at least two predictive blocks (e.g., vertical blocks, horizontal blocks, or both vertical and horizontal blocks). In some examples, a single CU may additionally include a third PBG formed by grouping at least two predictive blocks (e.g., a vertical block, a horizontal block, or both a vertical block and a horizontal block), a fourth PBG formed by grouping at least three predictive blocks (e.g., a vertical block, a horizontal block, or both), and so on.
[0058]
[0081] In addition, one or more subsequent CUs may include the same or different arrangement of predictive blocks and / or PBGs. For example, a second CU may include the same or at least similar partitioning as that of one or more preceding CUs. In another example, a second CU may include a partitioning scheme different from that of one or more preceding CUs. For example, a video coder may partition a first CU using vertical partitioning, partition a second CU using horizontal partitioning, partition a third CU using a combination of horizontal and vertical partitioning, partition a fourth CU using a different type of horizontal partitioning (e.g., to form more or fewer narrow predictive blocks with respect to other partitioned predictive blocks), or some other combination or variation.
[0059]
[0082] Although this disclosure describes some examples with respect to vertical partitioning or, in some cases, with respect to horizontal partitioning, it will be understood that the techniques of this disclosure, as described, are not so limited. In an additional non-limiting example, for illustrative purposes, a video decoder may partition multiple CUs using the same partitioning scheme by using vertical partitioning to partition at least two CUs. The video coder may additionally group partitioned predictive blocks differently or the same for each partitioned CU. For example, the video coder may perform vertical partitioning in a first CU to form a group of at least three PBGs, while the video coder may perform a similar vertical partitioning in one or more subsequent CUs, but to form at least two PBGs. That is, even in cases where the video coder partitions each CU using, for example, vertical partitioning, the amount of PBGs for each CU may not necessarily be equal from one partitioned CU to the next partitioned CU. It should be understood that other partitioning and / or grouping options may be achieved by one or more of the various techniques or variations of techniques described in this disclosure, although for brevity not all options are explicitly depicted. In addition, the video coder may perform independent reconstruction of such PBGs, as described above and further below.
[0060]
[0083] The video coder may then reconstruct the samples of the predictive blocks included in the first PBG independently of the samples of other predictive blocks included in the first PBG. For example, the video coder may reconstruct the samples as part of a decoding loop. In other cases, the video decoder 300 may partition the CU to form a set of predictive blocks and group the predictive blocks into PBGs.
[0061]
[0084] This disclosure may generally relate to "signaling" certain information, such as syntax elements. The term "signaling" may generally relate to communication of values of syntax elements and / or other data used to decode encoded video data. That is, video encoder 200 may signal values for syntax elements in a bitstream. Generally, signaling refers to causing a value to occur in the bitstream. As mentioned above, source device 102 may forward the bitstream to destination device 116 in substantially real-time or not in real-time, such as might occur when storing syntax elements in storage device 112 for later retrieval by destination device 116.
[0062]
[0085] FIG. 2 is a diagram illustrating exemplary intra-prediction modes. An example of intra-prediction is wide-angle intra-prediction. In some examples, intra-prediction of a luma block includes 35 modes, including a planar prediction mode, a DC prediction mode, and 33 angular (or directional) prediction modes. For example, directional prediction for a square block may use a direction between −135 degrees and 45 degrees of a current block 250, as in the VVC Test Model 2 (VTM2) (see JVET-K1002, “Algorithm Description for Versatile Video Coding and Test Model 2 (VTM2),” 11th JVET Meeting, Ljubljana, SI, July 2018, by J. Chen, Y. Ye, and S. Kim), as illustrated in FIG. 2 in some cases. The 35 modes of intra-prediction are indexed as shown in Table 1 below.
[0063] [Table 1]
[0064]
[0086] In VTM2, the block structure used to identify the prediction block for intra prediction is not limited to a square (width w = height h). For example, a rectangular prediction block (w > h or w < h) can improve the coding efficiency based on the characteristics of the content.
[0065]
[0087] In such a rectangular block, restricting the intra prediction direction within -135 degrees to 45 degrees may result in a situation where a farther reference sample is used rather than a nearer reference sample for intra prediction. Such a design may affect the coding efficiency. It may be more beneficial to relax the range of the restriction so that a nearer reference sample (beyond the angle range of -135 to 45 degrees) is used for prediction. An example of such a case is shown in FIG. 3.
[0066]
[0088] FIG. 3 is a block diagram illustrating an example of an 8×4 rectangular block. In the example of FIG. 3, since the intra prediction direction is restricted to the range of -135 degrees to 45 degrees, the "nearer" reference sample (indicated by circle 3002) is not used, but the "farther" reference sample (indicated by circle 3006) may be used. That is, in the example of FIG. 3, some reference samples that are beyond -135 degrees but farther than those indicated by circle 3002 may be used, but some reference samples, although nearer than other samples, for example, nearer than the samples indicated by circle 3006, are not used.
[0067]
[0089] FIGS. 4A to 4C are block diagrams providing diagrams of the mode mapping process for modes outside the diagonal direction range. In the example of FIG. 4A, the square block CU402 does not require angular mode remapping. FIG. 4B illustrates the angular mode remapping for the horizontal non-square block CU404. FIG. 4C illustrates the angular mode remapping for the vertical non-square block CU406.
[0068]
[0090] 5 is a block diagram illustrating wide-angle intra prediction employed in VTM2. In some examples, wide-angle intra prediction includes angular intra prediction for square and non-square blocks, such as block 550.
[0069]
[0091] During the 12th JVET meeting, a modification of wide-angle intra prediction was adopted in the VVC Test Model 3 (VTM3), the details of which can be found in (i) L. Zhao, X. Zhao, S. Liu and X. Li, "CE3 related: unifying angular intra prediction for square and non-square blocks," 12th JVET meeting, Macau SAR, CN, October 2018, JVET-L0279; (ii) J. Chen, Y. Ye and S. Kim, "CE3 related: unifying angular intra prediction for square and non-square blocks," 12th JVET meeting, Macau SAR, CN, October 2018, JVET-L0279; ,“Versatile Video Coding and Test Model 3 (VTM3),” by B. Bross, J. Chen, and S. Liu, “Versatile Video Coding (Draft ...
[0070]
[0092] This adoption includes two modifications to unify angular intra prediction for square and non-square blocks. First, the angular prediction direction is modified to cover the diagonal directions for all block shapes. Second, as illustrated in Figures 4A-4C and described above, the angular direction may be kept within the range between the bottom-left diagonal and the top-right diagonal for all block aspect ratios (square and non-square). In addition, the number of reference samples in the top reference row and left reference column is limited to 2*width+1 and 2*height+1 for all block shapes. In the example of Figure 5, in addition to the 65 angular modes, wide angles (-1 to -10 and 67 to 76) are depicted.
[0071]
[0093] FIG. 6 is a block diagram illustrating the wider angles for prediction of block 650 employed in VTM3. VTM3 specifies 95 modes for any block size, but only 67 modes may be allowed. The exact modes allowed depend on the ratio of block width to block height. This is done by restricting the mode range for a certain block size. FIG. 6 provides an illustration of the wide angles (-1 to -14 and 67 to 80) in VTM3 beyond mode 2 and 66 for a total of 93 angular modes.
[0072]
[0094] 7 is an example mapping table providing an example mapping between predModeIntra and the angle parameter intraPredAngle in VTM3, further details of which are available from B. Bross, J. Chen and S. Liu, "Versatile Video Coding (Draft 3)," 12th JVET Meeting, Macau SAR, CN, October 2018, JVET-L1001. In the following, angular modes with positive intraPredAngle values are referred to as positive angular modes (mode index < 18 or > 50) and angular modes with negative intraPredAngle values are referred to as negative angular modes (mode index > 18 and < 50).
[0073]
[0095] The inverse angle parameter invAngle is derived based on intraPredAngle according to equation (1).
[0074]
number
[0075]
[0096] It should be noted that intraPredAngle values that are multiples of 32 (i.e., values of 0, 32, 64, 128, 256, and 512 in this example) correspond to predictions from non-fractional reference array samples, as is the case in the VTM3 specification. Table 2 below illustrates the diagonal modes that correspond to various block aspect ratios.
[0076] [Table 2]
[0077]
[0097] Intra-subpartition coding (ISP) is described in the remainder of this disclosure. Intra-subpartition coding (ISP) (S. De Luxan Hernandez, H. Schwarz, D. Marpe, T. Wiegand (HHI), "CE3: Line-Based Intra-Coding Mode", JVET-L0076) is a method of dividing or partitioning a coding block (e.g., a coding unit) into multiple sub-blocks. For example, a coding unit may be divided or partitioned into two sub-blocks, four sub-blocks, etc. In some instances, the sub-blocks may be referred to as prediction blocks. In some instances, each sub-block in a coding block is reconstructed in decoding order before reconstruction of the sub-block that follows it in decoding order. The Fourth Working Draft of the VVC (hereinafter "VVC In "WD4" and available at phenix.it-sudparis.eu / jvet / doc_end_user / documents / 13_Marrakech / wg11 / JVET-M1001-v2.zip, ISP may only be applied to luma coding blocks. In such examples, the reference samples for these ISP coded blocks are restricted to be from the reference line closest to the coding block (see MRLIdx=0 described below with reference to multiple reference line prediction).
[0078]
[0098] Figures 8 and 9 are block diagrams illustrating example partitioning of blocks for ISP coding. Figure 8 illustrates an example of partitioning of 4x8 and 8x4 blocks. Figure 9 illustrates an example of partitioning of all blocks except 4x8, 8x4, and 4x4 blocks.
[0079]
[0099] One bit is used to signal whether the coding block is divided into intra subpartitions, and a second bit is used to indicate the partition type. Exemplary partition types include horizontal partition, vertical partition, vertical and horizontal partition, etc. Based on the intra mode and partition type used, two different classes of processing orders, referred to as normal order and reverse processing order, may be used. In normal order, the first subpartition processed is the subpartition containing the top-left sample of the CU, and then continues downward (horizontal partition) or right (vertical partition). Meanwhile, the reverse processing order starts with the subpartition containing the bottom-left sample of the CU and continues upward (horizontal partition) or starts with the subpartition containing the top-right sample of the CU and continues leftward (vertical partition).
[0080]
[0100] In some examples, variations of ISP may use only normal processing order, such as in VVC WD4. Note that the terms "sub-block" and "sub-partition" may be used interchangeably herein. In either case, both terms refer to blocks obtained by partitioning a coding block using an ISP.
[0081]
[0101] Some syntax (eg, Tables 3 and 4) and semantics related to ISP in VVC WD4 are shown below, with italics indicating the relevant syntax.
[0082] [Table 3]
[0083] [Table 4]
[0084]
[0102] Semantics of coding units.
[0085]
[0103] intra_subpartitions_mode_flag[x0][y0] equal to 1 specifies that the current intra coding unit is partitioned into NumIntraSubPartitions[x0][y0] rectangular transform block subpartitions. intra_subpartitions_mode_flag[x0][y0] equal to 0 specifies that the current intra coding unit is not partitioned into rectangular transform block subpartitions.
[0086]
[0104] When intra_subpartitions_mode_flag[x0][y0] is not present, the value of intra_subpartitions_mode_flag is inferred to be equal to 0.
[0087]
[0105] In some examples, the intra subpartition split flag may be used to indicate the partition type. For example, intra_subpartitions_split_flag[x0][y0] may specify the partition type or a number of different partition types. In some examples, the intra subpartition split flag may indicate whether the intra subpartition partition type is horizontal, vertical, rectangular, mixed vertical and horizontal, or some other partition type. When intra_subpartitions_mode_flag[x0][y0] is not present, the video coder may infer that the flag is equal to 0.
[0088]
[0106] The variable IntraSubPartitionsSplitType specifies the type of split used for the current luma coding block, as shown below in Table 5. IntraSubPartitionsSplitType is derived as follows:
[0089]
[0107] If intra_subpartitions_mode_flag[x0][y0] is equal to 0, then IntraSubPartitionsSplitType is set equal to 0.
[0090]
[0108] Otherwise, IntraSubPartitionsSplitType is set equal to 1 + intra_subpartitions_split_flag[x0][y0].
[0091] [Table 5]
[0092]
[0109] The variable NumIntraSubPartitions specifies the number of transform block subpartitions into which an intra-luma coding block is divided. NumIntraSubPartitions is derived as follows:
[0093]
[0110] If IntraSubPartitionsSplitType is equal to ISP_NO_SPLIT, then NumIntraSubPartitions is set equal to 1.
[0094]
[0111] Otherwise, if one of the following conditions is true, then NumIntraSubPartitions is set equal to 2: (1) cbWidth is equal to 4 and cbHeight is equal to 8, or (2) cbWidth is equal to 8 and cbHeight is equal to 4. Otherwise, NumIntraSubPartitions is set equal to 4.
[0095]
[0112] In the following parts of this disclosure, multiple reference line prediction is described. For intra prediction of a block, samples in the neighborhood of the coding block are used. Typically, the reconstructed reference sample lines closest to the left and top boundaries of the coding block are used as reference samples for intra prediction. In some examples, such as in VVC WD4, the video coder may use other samples, such as samples in the neighborhood of the coding block or CU, as reference samples.
[0096]
[0113] 10 is a block diagram illustrating reference sample lines that may be used for intra prediction. For each coding block, an index is signaled indicating the reference line to be used.
[0097]
[0114] In some examples, such as in VVC WD4, the video coder may use only reference lines with MRLIdx equal to 0, 1, or 3. The video coder may use a truncated unary codeword to code an index to the reference line used to code the block (e.g., values 0, 1, and 2 to indicate lines with MRLIdx 0, 1, and 3, respectively). For used reference lines with MRLIdx>0, Planar and DC modes are not used.
[0098]
[0115] In some examples, intra subpartition (ISP) coding of a block (e.g., a coding unit (CU)) may result in a subblock (e.g., a prediction block) having a very small size in one or more dimensions. For example, when a video coder uses ISP coding to code a coding block of size 4×N (W×H), and vertical partitioning is used, the resulting subblock has size 1×N when N is greater than 8. In implementations that rely on row-wise storage of samples, narrower and narrower subblocks may result in memory accesses of more samples than the number of samples in the subblock. When a block is of size M×N, accessing M samples in each row from memory may often involve overhead, such as accessing more than M samples due to limitations in the minimum number of samples per access.
[0099]
[0116] This problem may be exacerbated when the number of rows is larger for narrow blocks due to the narrow shape of 1×N and 2×N blocks. For example, a 4×4 block may have only four rows. A 1×16 sub-block has the same number of samples as a 4×4 block, but has 16 rows to be accessed. The resulting overhead may be problematic in delay-critical cases, especially in the worst case where there are several small coding blocks in a high solution sequence. These sub-blocks may increase the delay to decode a block, since the reconstruction of a sub-block in a coding block may depend on the reconstruction of other sub-blocks in the coding block that precede it in the decoding order, which may be undesirable.
[0100]
[0117] Several solutions have been proposed to address this problem. However, each of the proposed solutions comes with drawbacks that may result in coding inefficiencies. One proposed solution is to disable vertical partitioning of 4×N and 8×N blocks resulting in blocks of size 1×N and 2×N, respectively. However, this proposed solution may reduce the efficiency of the ISP. Another proposed solution (i.e., S. De Luxan Hernandez, V. George, J. Ma, T. Nguyen, H. Schwarz, D. Marpe, T. Wiegand (HHI), "CE3: Intra Subpartition Coding Modes" JVET-M0102) is to restrict prediction of 1×N subblocks to VER prediction modes so that they do not depend on the reconstruction of other subblocks in the coding block. However, this proposed solution would limit the intra modes allowed for the coding block, resulting in coding inefficiencies.
[0101]
[0118] This disclosure proposes techniques for improving the design of subpartition coding of CUs. One or more of the techniques disclosed herein may be implemented separately or together in any combination.
[0102]
[0119] According to the first proposed technique, a video coder (e.g., video encoder 200 and / or video decoder 300) may perform generalized prediction of a sub-block (e.g., a predictive block). As one example, the video coder may code an indication (e.g., a syntax element) to perform generalized prediction for a coding block. In some examples, the indication to perform generalized prediction may be accompanied by an indication that the coding block is coded with ISP. In some examples, the indication to perform generalized prediction may apply only to blocks having certain characteristics, such as, for example, when the block width is within a threshold range, or when the block height is within a threshold range, or when the number of samples in the block is within a threshold range, or when the sum of the block width and the block height is within a threshold range, or when the minimum or maximum of the block width and the block height is within a threshold range. In some examples, a subset of intra modes from the set of intra modes allowed for the block may be defined for the block. In such an example, generalized prediction may be applied to a block only if the block is coded with an intra mode belonging to the subset. In some examples, the indication may be inferred by the characteristics / features of the block, the intra modes and characteristics of the neighboring blocks, i.e., no explicit signaling may be used. For example, generalized prediction may be applied when the block width is smaller than a certain threshold.
[0103]
[0120] As another example, based on the indication to perform generalized prediction, the video coder may divide a coding unit into one or more predictive blocks.
[0104]
[0121] As another example, a video coder may define one or more predictive block groups (PBGs). A PBG may include one or more predictive blocks. A prediction of any sample in a PBG may not be directly dependent on a predicted value of any other sample in the PBG. For example, the video coder may reconstruct samples of a predictive block included in a first PBG independently of samples of other predictive blocks included in the same PBG. In this way, a prediction of a sample in a first PBG does not depend on a predicted value of other samples in the first PBG.
[0105]
[0122] In some examples, each PBG contains exactly one predictive block. In some examples, each PBG contains n PB In such an example, the video coder may determine n prediction blocks (e.g., a prediction block amount) based on the width and height of the coding unit. PB In another example, the video coder may determine the value of n based on the number of samples in a coding unit, or based on the number of samples and the width and height of a coding unit. PB In some examples, the video coder may also use a partition type (e.g., horizontal, vertical, rectangular, mixed horizontal and vertical, etc.) and / or an intra mode (e.g., angular, etc.) to determine the value of n for a coding unit or coding block. PB In some examples, the video coder may specify a prediction block that is not rectangular in shape.
[0106]
[0123] In some examples, the restriction that "the prediction of any sample in the PBG may not depend on the predicted value of any other sample in the PBG" may be relaxed. For example, when applying certain filtering operations on the predicted block that may introduce dependencies between the predicted values of samples in the PBG, the restriction of prediction dependencies of samples in the PBG may still be considered valid as long as normal intra prediction (without post-prediction filtering operations) does not result in dependencies. For example, the restriction may not apply to dependencies due to position-dependent intra prediction combining (PDPC), boundary filtering, or other similar operations. In some examples, each predicted block in a coding block may be included in one and only one PBG. In some examples, this restriction may be stricter. For example, a restriction may be made that each predicted block in a coding block must be included in one and only one PBG.
[0107]
[0124] As another example, the video coder may specify a decoding order for the PBGs, and the decoding order may be defined such that samples in PBG P1 are independent of samples in PBG P2 when P1 precedes P2 in the decoding order.
[0108]
[0125] As another example, for each PBG, the video coder may specify a set of available reference samples (described below).
[0109]
[0126] As another example, the video coder may obtain predicted values for samples in the PBG based on reference samples available for the PBG and the intra mode used for prediction.
[0110]
[0127] According to a second proposed technique, the video coder may perform generalized planar prediction. As an example, the video coder may define a set of predictive blocks for a coding block. In some examples, the video coder may code one or more of the predictive blocks in a planar mode. The video coder may perform a decision to split the coding block into planar predictive blocks by one or more of the following:
[0111]
[0128] 1. An indication (e.g., a flag) in the bitstream that a block will be split into planar prediction blocks. In some examples, it may be indicated that the coding block will be coded in planar mode. In another example, the indication may be that the coding block will be coded using ISP.
[0112]
[0129] 2. Planar prediction blocks have separate mode values M PPB The block may be represented by M PPB When a block is indicated to be coded with a planar prediction block, the block is divided into planar prediction blocks.
[0113]
[0130] 3. The decision may be inferred based on certain block characteristics and other syntax elements in the bitstream, for example, a block that has a certain maximum number of samples and is coded with ISP may be split into planar prediction blocks.
[0114]
[0131] In some examples, the video coder may further include determining whether the number of planar prediction blocks, N, is increased by one or more of the following: PPB and split decision. N PPBN may be signaled in the bitstream (e.g., for all blocks or for blocks with certain characteristics) or may be derived based on block characteristics and other syntax elements in the bitstream. PPB may be set equal to the number of intra subpartition blocks for the coding block. PPB may be set equal to 4. The splitting direction or method may be determined by block characteristics or may be derived from syntax elements in the bitstream.
[0115]
[0132] In some examples, the blocks are arranged horizontally as PPB The decision to split horizontally may depend, for example, on whether a horizontal split direction is indicated.
[0116]
[0133] FIG. 11 is a conceptual diagram illustrating an example of how a coding block may be partitioned into four horizontal planar prediction blocks.
[0117]
[0134] In some examples, the video coder divides the blocks vertically into N PPB The video coder may determine whether to split vertically based on, for example, whether a vertical split direction is indicated.
[0118]
[0135] FIG. 12 is a conceptual diagram illustrating an example of how a coding block may be partitioned into four vertical planar prediction blocks.
[0119]
[0136] In some examples, the video coder divides the block horizontally and vertically to produce a total of N PPBThe video coder may determine whether to split horizontally and vertically based on one or more characteristics of the block (e.g., characteristics of the CU). In some examples, the video coder may determine whether to split horizontally and vertically when no split direction is indicated.
[0120]
[0137] In some examples, the block may share the same top-left sample as the coding block, N PPB The image may be divided into rectangular regions, each of which successively includes the previous one. Starting from the smallest block defined as the first planar prediction block, samples that belong to the next rectangular region but not to the previous rectangular region are said to constitute the subsequent planar prediction block. Figure 13 is a conceptual diagram illustrating an example of such a planar prediction block structure.
[0121]
[0138] In some examples, the video coder may also predict planar prediction blocks in a manner similar to the planar prediction mode.
[0122]
[0139] In some examples, the video coder may define a coding order for the predictive blocks. For example, the coding order may be defined by considering the following (a. through d.):
[0123]
[0140] a. Assign a representative sample for each prediction block, e.g., the top-left sample of the prediction block. In some cases (non-rectangular blocks), the top-left sample may not be defined, and the representative sample may be specified as the left-most sample in the top row of samples of the prediction block. It should be understood that the coding order may be specified by other definitions of the representative samples.
[0124]
[0141] b. Specify the scanning order of the representative samples (eg, raster scan, zigzag scan).
[0125]
[0142] c. A predictive block having a representative sample earlier in the scanning order is coded earlier than a predictive block having a representative sample later in the scanning order.
[0126]
[0143] A similar definition of coding order may be specified for predictive block groups.
[0127]
[0144] In some examples, a video coder may specify a planar predictive block group (Planar PBG) that includes one or more planar predictive blocks, similar to the concept of PBG described above.
[0128]
[0145] Boundaries between planar prediction blocks may be smoothed by applying filtering across prediction block boundaries within a coding block. One example of such filtering is low-pass filtering. A second example is the process performed in PDPC mode. A third example is a deblocking filter process. These processes may be applied as a second processing step after prediction or as a post-processing step.
[0129]
[0146] It should be understood that the above examples of planar prediction blocks are just a few examples to illustrate the concept. Samples in a coding block may be divided into spatial regions defined as planar prediction blocks, and one or more methods disclosed in this application may be applied to them.
[0130]
[0147] Furthermore, the term block (coding block, predicted block, etc.) as used herein may apply to a luma block or a chroma block (or both), or generally to a coding block of any component of a video.
[0131]
[0148] According to a third proposed technique, a video coder may determine the availability of reference samples. For each PBG, the available reference samples may be specified as follows:
[0132]
[0149] In some examples, the set of available reference samples is the same for all prediction blocks in a coding block. There may be multiple implementations of such a system. As one example, the reference sample derivation process is not changed, and changes are made to the prediction process. The set of available reference samples is derived once for a coding block. In some examples, changes may be made to the prediction process to ensure that reference samples are not directly adjacent to the prediction block or are not specified by MRLIdx for the prediction block.
[0133]
[0150] In some examples, the reference sample derivation is changed and no changes are made to the prediction process. In such examples, the available reference samples for the prediction block are derived from a set of available reference samples for the coding block. The derivation may result in a derived value of the reference sample for the prediction block at a position corresponding to the reference sample line used (i.e., based on the value of MRLIdx).
[0134]
[0151] In some examples, predicted samples from a preceding PBG in decoding order are marked as unavailable.
[0135]
[0152] In some examples, in addition to reference samples from neighboring coding blocks, processed values of a previously coded PBG of the same coding block may be used for intra prediction of the PBG. As one example, the processed values refer to reconstructed samples of the PBG. As another example, the processed values refer to reconstructed samples of the PBG before the loop filtering / post filtering operation. As another example, the processed values refer to predicted samples of the PBG.
[0136]
[0153] According to a fourth proposed technique, the video coder may perform transform coding. Prediction of samples in a block may be performed by one or more methods described herein. However, the techniques in this section may also be applied to coding blocks that are not predicted using generalized prediction as described above.
[0137]
[0154] In some examples, the video coder may apply the transform to residual samples in the PBG. A set of applicable shapes may be defined such that the video coder may apply the transform to all samples of the applicable shapes. For example, the applicable shapes may include square and / or rectangular shapes. In some examples, the transform coding of the block may be performed by considering the entire coding block as one block. In such examples, only one transform coding may be applied to samples in the coding block regardless of the number of predictive blocks defined therein. In some examples, the video coder may apply the transform coding to the PBG when the shape of the PBG is an applicable shape to which the transform may be applied. For example, the transform may be applied to samples of the PBG when the samples in the PBG form a rectangular region. In some examples, the video coder may apply the transform coding to one or more subsets of samples in the PBG such that the samples in each such subset form an applicable shape. As one example, the subsets of the PBG may be constrained to be mutually exclusive. As another example, the subsets of the PBG may be constrained to collectively include all samples of the PBG.
[0138]
[0155] In some examples, applying the transform may also include a transform skip decision on the block, whereby the sample values of the block are left unchanged, an identity transform is applied, or a scaled identity transform is applied. As one example, the video coder may code some predictive blocks in the PBG with a transform skip. In addition, the video coder may code some predictive blocks with other types of transforms. The decision to apply a transform kernel or a transform skip may be inferred (e.g., based on block characteristics, intra mode, neighboring blocks) or signaled in the bitstream.
[0139]
[0156] In some examples, a subblock partitioning scheme may be defined for transform coding different from the specified predictive block. A transform block group (TBG) may be defined using the transform subblocks similar to that defined for the predictive block. One or more of the techniques described above may be applied to the TBG. For example, when the intra subpartition results in subblocks of size 2×N (the predictive block may be of size 2×N), the transform block may be defined as 1×N and each transform block may be transform coded independently. In another example, the predictive block may be of size 2×N, the predictive block group may be of size 4×N, and the transform block may be of size 4×N. In another example, the predictive block may be of size 2×N, the predictive block group may be of size 4×N, and the transform block may be of size 2×N for an 8×N coding block.
[0140]
[0157] According to the fifth proposed technique, the video coder may not allow certain intra modes. For intra prediction of a block, several intra modes are defined (e.g., angular and non-angular). For a PBG, only a subset of the intra modes may be specified as allowed modes. In other words, certain intra modes may not be allowed for a PBG. The determination of the disallowed intra modes may be achieved by considering one or more of the following: (1) the partition direction, (2) the width, height, or a function of the width and height of the coding block, (3) the width, height, or a function of the width and height of the prediction block, or (4) the above block characteristics of the neighboring coding blocks and the intra modes used to code the neighboring coding blocks.
[0141]
[0158] It should be understood that the above description (including examples) is in no way limiting of the techniques involved and may be combined with other methods that may not be described herein. One or more decisions or indications may also depend on other conditions, including tile group type, partition type, characteristics of one or more neighboring blocks (width, height, intra mode, prediction type), etc.
[0142]
[0159] Below are two specific examples of the above techniques.
[0143]
[0160] In a first example, a predictive block group (PBG) is specified to have a minimum width of 4 samples. For example, a vertical partition of a coding block of size 4×N may result in sub-blocks of width 1 or 2, and a vertical partition of an 8×N coding block may result in sub-blocks of width 2 or 4.
[0144]
[0161] FIG. 14 is a block diagram illustrating how a coding block 1402 is divided into four predictive blocks (PB1-PB4). In an exemplary example, the coding block 1402 may be an 8×N coding block that a video coder codes with ISP and vertical partitioning. In this example, the predictive blocks PB1, PB2, PB3, and PB4 are of size 2×N. The predictive block groups PBG1 and PBG2 are defined with a minimum width (nW) of four samples. For example, PBG1 is a predictive block group of size 4×N, and PBG2 is a second predictive block group of size 4×N as well. The samples of PBG1 are predicted from neighboring reference samples. In some examples, the predicted values of samples in PBG1 may not depend on other sample values in PBG1. The samples of PBG2 are predicted from neighboring reference samples and from predicted samples of PBG1.
[0145]
[0162] In some examples, a transform block (e.g., a transform unit (TU)) may be defined to be of the same size as a predictive block. For example, the video coder may determine the size of a TU to be equal to the size of any one of the predictive blocks included in the PBG.
[0146]
[0163] In some examples, samples of PBG2 may be predicted from reference samples from neighboring coding blocks and from reconstructed samples belonging to the area of PBG1, or only from reference samples from neighboring coding blocks.
[0147]
[0164] FIG. 15 is a block diagram illustrating an example in which a video coder may code a coding block 1502 with an ISP and vertical partitioning. In an illustrative example, the coding block 1502 may be a 4×N coding block. In such an example, the predictive blocks (PB1-PB4) may be of size 1×N. In some examples, a PBG may be defined to have a width of at least four samples. In such an example, the video coder may group the four predictive blocks (PB1-PB4) into one predictive block group (e.g., PBG1). As such, PBG1 may be a PBG of size 4×N.
[0148]
[0165] In some cases, generalized prediction (specifying predictive blocks and predictive block groups as described herein) is applied only to a subset of blocks that satisfy one or more criteria, which may depend on block size, block shape, sub-block width and height, intra mode, etc. For example, the condition may apply only to coding blocks that give rise to sub-blocks with a width of 2 or less. In other cases, the condition may apply to sub-blocks with a total number of samples of 32 or less.
[0149]
[0166] In another example, samples of all predictive blocks of a coding block are predicted from reconstructed reference samples of neighboring coding blocks. In such an example, there is no prediction dependency within the predictive blocks of a coding unit. In some examples, the transform unit size may be equal to the size of the predictive blocks.
[0150]
[0167] FIG. 16 is a conceptual diagram illustrating an example in which an M×N coding block 1602 is divided into four predictive blocks (e.g., PB1-PB4). In an illustrative example, the predictive blocks (PB1-PB4) have a size of (M / 4)×N. A reference sample 1604 is used as a reference sample for the four predictive blocks. There is no dependency between the predicted samples of one predictive block and the others. For example, there is no dependency between the predicted samples of PB1 and the predicted samples of PB2, PB3, or PB4.
[0151]
[0168] In some cases, the four predicted sub-blocks may be predicted in parallel. In another example, the four sub-blocks (PB1 to PB4) may be considered together as one block for prediction. For example, the four sub-blocks (PB1 to PB4) may be considered together as one PBG for prediction.
[0152]
[0169] In some examples, the video coder may predict samples of some coding blocks using at least one predictive block group with a width of four samples, and some other coding units are predicted in constrained intra-mode. The following restrictions that apply to the coding of 4×N and 8×N coding blocks that are coded with ISP are added in the specification:
[0153]
[0170] For 4×N coding blocks (N>4) partitioned vertically in the ISP, only vertical mode is allowed for intra prediction. This will ensure that prediction of any subpartition (1×N subpartition in case of 4×N coding blocks, 2×N subpartition in case of 4×8 coding blocks, where N>8) is independent of other subpartitions in the coding block. In this way, the video coder may perform prediction of subpartitions in parallel or by considering all subpartitions as 4×N blocks for prediction purposes. For such coding blocks, intra mode may not be signaled.
[0154]
[0171] FIG. 17 is a conceptual diagram illustrating an example in which a video coder is allowed to perform only vertical intra (VER Intra) mode prediction for a 4×N coding block 1702 that the video coder codes with ISP and vertical partitioning. That is, FIG. 17 shows an example of how a vertical partition subpartition of a 4×N coding block 1702 may use only vertical intra mode for prediction. In an illustrative example, the coding block 1702 may have a height of at least 8 sample sizes (N>8). In such an example, the coding block 1702 may be divided into four 1×N subpartitions (e.g., subblocks, prediction blocks). In either case, the inverse transform operation is still performed at the 1×N subpartition level.
[0155]
[0172] For an 8×N coding block vertically partitioned with ISP, where N>4, prediction of samples is performed on a prediction block group (PBG) that is four samples wide. For example, when the coding block size is 8×N (N>4), the video coder may identify two PBGs, each having a size of 4×N. FIG. 18 shows an example of how an 8×N subpartition is partitioned into two prediction regions (PBG1 or region 1 and PBG2 or region 2) of size 4×N each, and four transforms of size 2×N each. In the example of FIG. 18, the video coder may code an 8×N (N>4) coding block 1802 with ISP and vertical partitioning. In such an example, the video coder may perform prediction on the 4×N prediction region (or PBG). In addition, the transform (SP) size may be 2×N. In some examples, the prediction of region 2 depends on the reconstruction of region 1. Thus, the prediction of any sample in the coding block 1802 depends only on reconstructed samples from adjacent coding blocks and the preceding 4×N prediction region.
[0156]
[0173] In FIG. 17, similar to FIG. 18, there are four line residuals of size 1×N, and two residual blocks of 2×N can be generated in one prediction region (prediction region 1 or 2) in parallel or together in one step. For example, since the vertical transform for all subpartitions in the ISP is the same, the vertical inverse transform operation of the two 2×N subpartitions in each prediction region may be performed by regarding the region as a 4×N block. The horizontal transform may also be performed in parallel for the 2×N subpartitions, either by separate processing or by regarding the two subpartitions as a single 4×N block and the horizontal inverse transform operation as a combination of two smaller horizontal inverse transform operations.
[0157]
[0174] Table 6 below summarizes the partition sizes for the prediction and transform sizes. In some examples, the prediction and transform sizes may be different for other coding block sizes. For example, the prediction and transform sizes may be those specified for the ISP in VVC WD4.
[0158] [Table 6]
[0159]
[0175] Changes to the relevant sections of the specification in the 5th Working Draft of the VVC (hereafter "VVC WD5", available at phenix.it-sudparis.eu / jvet / doc_end_user / documents / 14_Geneva / wg11 / JVET-N1001-v8.zip) are provided below with the changes marked. Deleted items are shown in strikethrough and added items are shown in bold italics.
[0160]
[0176] In some examples, the video encoder may use the following inputs to perform the coding process:
[0161]
[0177] 1. A sample location (xTb0, yTb0) that specifies the top-left sample of the current transform block with respect to the top-left sample of the current picture.
[0162]
[0178] 2. Variable nTbW that specifies the width of the current transformation block.
[0163]
[0179] 3. Variable nTbH, which specifies the height of the current transformation block.
[0164]
[0180] 4. Variable predModeIntra that specifies the intra prediction mode.
[0165]
[0181] 5. Variable cIdx that specifies the color components of the current block.
[0166]
[0182] The output of the process is a modified reconstructed picture before in-loop filtering.
[0167]
[0183] The maximum transform block size maxTbSize is derived using Equation 2 as follows: maxTbSize=(cIdx==0)?MaxTbSizeY:MaxTbSizeY / 2 (2).
[0168]
[0184] The luma sample locations are derived using Equation 3 as follows: (xTbY,yTbY)=(cIdx==0)?(xTb0,yTb0):(xTb0*2,yTb0*2) (3).
[0169]
[0185] Depending on maxTbSize the following applies:
[0170]
[0186] If IntraSubPartitionsSplitType is equal to NO_ISP_SPLIT and either nTbW is greater than maxTbSize or nTbH is greater than maxTbSize, then the following ordered steps (1-5) are applied.
[0171]
[0187] 1. The variables newTbW and newTbH are derived as follows: newTbW=(nTbW>maxTbSize)?(nTbW / 2):nTbW newTbH=(nTbH>maxTbSize)?(nTbH / 2):nTbH.
[0172]
[0188] 2. The general decoding process for intra blocks specified in this section is called with inputs the location (xTb0, yTb0), the transform block width nTbW which is set equal to newTbW, the height nTbH which is set equal to newTbH, the intra prediction mode predModeIntra, and the variable cIdx, and the output is the modified reconstructed picture before in-loop filtering.
[0173]
[0189] 3. If nTbW is greater than maxTbSize, the general decoding process for intra blocks specified in this section is called with inputs location (xTb0, yTb0) set to be equal to (xTb0+newTbW, yTb0), transform block width nTbW set to be equal to newTbW, height nTbH set to be equal to newTbH, intra prediction mode predModeIntra, and variable cIdx, and the output is the modified reconstructed picture before in-loop filtering.
[0174]
[0190] 4. If nTbH is greater than maxTbSize, the general decoding process for intra blocks specified in this section is called with inputs location (xTb0, yTb0) set to be equal to (xTb0, yTb0 + newTbH), transform block width nTbW set to be equal to newTbW, height nTbH set to be equal to newTbH, intra prediction mode predModeIntra, and variable cIdx, and the output is the modified reconstructed picture before in-loop filtering.
[0175]
[0191] 5. If nTbW is greater than maxTbSize and nTbH is greater than maxTbSize, the general decoding process for intra blocks specified in this section is called with inputs location (xTb0, yTb0) set to be equal to (xTb0+newTbW, yTb0+newTbH), transform block width nTbW set to be equal to newTbW, height nTbH set to be equal to newTbH, intra prediction mode predModeIntra, and variable cIdx, and the output is the modified reconstructed picture before in-loop filtering.
[0176]
[0192] Otherwise, the following ordered steps apply (1. and 2.): For example, when the video coder determines that IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT (e.g., when IntraSubPartitionsSplitType is equal to either ISP_VER_SPLIT or ISP_HOR_SPLIT), or when nTbW and nTbH are both less than or equal to maxTbSize, the video coder may derive the following variables:
[0177]
[0193] 1. In some examples, the video coder may derive multiple variables, including nW, nH, xPartIncrement, yPartIncrement, nPbW, and pbFactor. For example, the video coder may derive such variables using equations (6)-(11) as follows: nW=IntraSubPartitionsSplitType==ISP_VER_SPLIT?nTbW / NumIntraSubPartitions:nTbW (6) nH=IntraSubPartitionsSplitType==ISP_HOR_SPLIT?nTbH / NumIntraSubPartitions:nTbH (7) xPartIncrement=ISP_VER_SPLIT?1;0 (8) yPartIncrement=ISP_HOR_SPLIT?1;0 (9) nPbW=max(4,nW) (10) pbFactor=nPbW / nW (11).
[0178]
[0194] 2. Additionally, the variables xPartIdx and yPartIdx are set equal to 0.
[0179]
[0195] In such an example, the video coder may apply the following ordered steps (1-4) consecutively for i=0..NumIntraSubPartitions-1:
[0180]
[0196] 1. Variables xPartIdx, yPartIdx. xPartPbIdx is updated as follows: xPartIdx=xPartIdx+xPartIncrement (12) yPartIdx=yPartIdx+yPartIncrement (13) xPartPbIdx=xPartIdx%pbFactor (14).
[0181]
[0197] 2. In some examples, when xPartPbIdx is equal to 0, the video coder may invoke an intra sample prediction process as specified in 8.4.5.2. In such examples, the video coder may invoke the intra sample prediction process with the location (xTbCmp, yTbCmp) set to be equal to (xTb0+nW*xPartIdx, yTb0+nH*yPartIdx), the intra prediction mode predModeIntra, the transform block width nTbW and height nTbH set to be equal to nPbW and nH, the coding block width nCbW set to be equal to nTbW, the coding block height nCbH set to be equal to nTbH, and the variable cIdx as inputs, and the output is a (nPbW)×(nH) array predSamples (e.g., a prediction sample array).
[0182]
[0198] 3. The scaling and transformation process specified in section 8.7.2 is called with inputs luma location (xTbY, yTbY) set to be equal to (xTbY+nW*xPartIdx, yTbY+nH*yPartIdx), variable cIdx, and transform width nTbW and transform height nTbH set to be equal to nW and nH, and the output is a (nPbW)×(nH) array resSamples (e.g., a residual sample array). In one example, the output may be a (nW)×(nH) array resSamples.
[0183]
[0199] 4. The picture reconstruction process for color components specified in Section 8.7.5 is called with inputs the transform block location (xTbComp, yTbComp) which is set equal to (xTb0+nW*xPartIdx, yTb0+nH*yPartIdx), the transform block width nTbW and transform block height nTbH which are set equal to nW and nH, the variable cIdx, the (nW) by (nH) array predSamples[x][y], where x=xPartPbIdx*nW..(xPartPbIdx+1)*nW-1, y=0..nH-1, and the (nW) by (nH) array resSamples; the output is the modified reconstructed picture before in-loop filtering.
[0184]
[0200] In some examples, the following modifications may limit intra prediction modes to vertical intra modes for some block sizes: Example modifications are made to the signaling in section 7.3.7.5 of VVC WD5. Removed items are shown in strikethrough and added items are shown in italics.
[0185] [Table 7]
[0186]
[0201] Exemplary modifications are made to the derivation process for luma intra prediction modes in section 8.4.3 of VVC WD5, with removed items shown in strikethrough and added items shown in italics.
[0187]
[0202] For example, the video coder may perform a derivation process for a luma intra prediction mode, where inputs to this process include (1) a luma location (xCb, yCb) that specifies a top-left sample of the current luma coding block relative to a top-left luma sample of the current picture, (2) a variable cbWidth that specifies the width of the current coding block in luma samples, and / or (3) a variable cbHeight that specifies the height of the current coding block in luma samples.
[0188]
[0203] In this process, based on the above inputs, the video coder may derive a luma intra-prediction mode IntraPredModeY[xCb][yCb].
[0189]
[0204] Table 8 identifies the values and associated names for the intra prediction modes IntraPredModeY[xCb][yCb]. Note that the intra prediction modes INTRA_LT_CCLM, INTRA_L_CCLM, and INTRA_T_CCLM may be applicable only to chroma components.
[0190] [Table 8]
[0191]
[0205] IntraPredModeY[xCb][yCb] is derived as follows: if BdpcmFlag[xCb][yCb] is equal to 1 or intra_luma_not_planar_flag[xCb][yCb] is equal to 0, then IntraPredModeY[xCb][yCb] is set equal to INTRA_PLANAR. Otherwise, if IntraSubPartitionsSplitType is equal to ISP_VER_SPLIT and cbWidth is equal to 4, then IntraPredModeY[xCb][yCb] is set equal to INTRA_ANGULAR50. Otherwise (e.g., intra_luma_not_planar_flag[xCb][yCb] is equal to 1), the neighboring locations (xNbA, yNbA) and (xNbB, yNbB) are set equal to (xCb-1, yCb+cbHeight-1) and (xCb+cbWidth-1, yCb-1), respectively.
[0192]
[0206] 19A and 19B are conceptual diagrams illustrating an exemplary QTBT structure 130 and a corresponding coding tree unit (CTU) 132. The solid lines represent quaternary tree partitions, and the dotted lines represent binary tree partitions. At each partition (i.e., non-leaf) node of the binary tree, one flag is signaled to indicate which partition type is used (e.g., horizontal, vertical, etc.). In an exemplary and non-limiting example, a value of 0 may indicate a horizontal partition, and a value of 1 may indicate a vertical partition. In another example involving a quaternary tree partition, a quaternary tree node may not need to indicate a partition type, since it generally divides a block horizontally and vertically into four sub-blocks (e.g., prediction blocks). In such an example, the sub-blocks may be of equal size, or may be of equal size. Thus, video encoder 200 may encode, and video decoder 300 may decode, syntax elements (such as partitioning information) for the region tree level (i.e., first level) of QTBT structure 130 (i.e., solid lines) and syntax elements (such as partitioning information) for the prediction tree level (i.e., second level) of QTBT structure 130 (i.e., dashed lines). Video encoder 200 may encode, and video decoder 300 may decode, video data such as prediction data and transform data for CUs represented by terminal leaf nodes of QTBT structure 130.
[0193]
[0207] In general, the CTU 132 of Figure 19B may be associated with parameters that define the sizes of blocks corresponding to nodes of the QTBT structure 130 at the first and second levels. These parameters may include a CTU size (representing the size of the CTU 132 in the sample), a minimum quaternary tree size (MinQTSize, representing the minimum allowable quaternary tree leaf node size), a maximum bipartite tree size (MaxBTSize, representing the maximum allowable bipartite tree root node size), a maximum bipartite tree depth (MaxBTDepth, representing the maximum allowable bipartite tree depth), and a minimum bipartite tree size (MinBTSize, representing the minimum allowable bipartite tree leaf node size).
[0194]
[0208] The root node of the QTBT structure corresponding to the CTU may have four child nodes at the first level of the QTBT structure, each of which may be partitioned according to a quaternary tree partition. That is, the nodes at the first level may be leaf nodes (without child nodes) or may have four child nodes. The example QTBT structure 130 represents a node that includes a parent node and child nodes with solid lines for branches. If the nodes at the first level are not larger than the maximum allowed binary tree root node size (MaxBTSize), the nodes may be further partitioned by their respective binary trees. The binary tree partitioning of a node may be repeated until the nodes obtained by the partitioning reach the minimum allowed binary tree leaf node size (MinBTSize) or the maximum allowed binary tree depth (MaxBTDepth). The example QTBT structure 130 represents a node with dashed lines for branches. The binary tree leaf nodes are referred to as coding units (CUs), which are used for prediction (e.g., intra-picture or inter-picture prediction) and transformation without any further partitioning. As discussed above, a CU may also be referred to as a "video block," a "coding block," or a "block."
[0195]
[0209] In one example of a QTBT partitioning structure, the CTU size is set as 128×128 (luma samples and two corresponding 64×64 chroma samples), MinQTSize is set as 16×16, MaxBTSize is set as 64×64, MinBTSize is set as 4 (for both width and height), and MaxBTDepth is set as 4. First, quaternary tree partitioning is applied to the CTU to generate quaternary tree leaf nodes. The quaternary tree leaf nodes may have sizes from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the quaternary tree leaf node is 128×128, the node will not be further partitioned by the binary tree since the size exceeds MaxBTSize (i.e., 64×64 in this example). Otherwise, the quaternary tree leaf node will be further partitioned by the binary tree. Therefore, the quaternary tree leaf node is also the root node for the binary tree and has the binary tree depth as 0. When the binary tree depth reaches MaxBTDepth (4 in this example), no further splits are allowed. When a binary tree node has a width equal to MinBTSize (4 in this example), it means that no further vertical splits are allowed. Similarly, a binary tree node having a height equal to MinBTSize means that no further horizontal splits are allowed for that binary tree node. As mentioned above, the leaf nodes of the binary tree are called CUs and are further processed according to prediction and transformation without further partitioning.
[0196]
[0210] FIG. 20 is a block diagram illustrating an example video encoder 200 that may perform the techniques of this disclosure. FIG. 20 is provided for illustrative purposes and should not be considered a limitation of the techniques as broadly illustrated and described in this disclosure. For illustrative purposes, this disclosure describes video encoder 200 in the context of video coding standards, such as the developing HEVC video coding standard and the H.266 video coding standard. However, the techniques of this disclosure are not limited to these video coding standards and are generally applicable to video encoding and decoding.
[0197]
[0211] In the example of Figure 20, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DBP) 218, and an entropy encoding unit 220. Any or all of the video data memory 230, the mode selection unit 202, the residual generation unit 204, the transform processing unit 206, the quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a DBP 218, and an entropy encoding unit 220 may be implemented in one or more processors or processing circuits. Furthermore, the video encoder 200 may include additional or alternative processors or processing circuits to perform these or other functions.
[0198]
[0212] The video data memory 230 may store video data to be encoded by the components of the video encoder 200. The video encoder 200 may receive video data stored in the video data memory 230, for example, from the video source 104 (FIG. 1). The DPB 218 may function as a reference picture memory that stores reference video data for use in predicting subsequent video data by the video encoder 200. The video data memory 230 and the DPB 218 may be formed by any of a variety of memory devices, such as synchronous dynamic random access memory (DRAM), including DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The video data memory 230 and the DPB 218 may be provided by the same memory device or separate memory devices. In various examples, the video data memory 230 may be on-chip with other components of the video encoder 200, as shown, or may be off-chip relative to these components.
[0199]
[0213] In this disclosure, references to video data memory 230 should not be construed as limited to memory that is internal to video encoder 200, unless specifically described as such, or to memory that is external to video encoder 200, unless specifically described as such. Rather, references to video data memory 230 should be understood as a reference memory that stores video data that video encoder 200 receives for encoding (e.g., video data for a current block to be encoded). Memory 106 of FIG. 1 may also provide temporary storage of outputs from various units of video encoder 200.
[0200]
[0214] The various units in FIG. 20 are illustrated to aid in understanding the operations performed by the video encoder 200. The units may be implemented as fixed function circuits, programmable circuits, or a combination thereof. Fixed function circuits refer to circuits that provide specific functionality and are preconfigured for the operations they can perform. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations they can perform. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the software or firmware instructions. Although a fixed function circuit may execute software instructions (e.g., to receive parameters or output parameters), the types of operations the fixed function circuit performs are generally unchanged. In some examples, one or more of the units may be separate circuit blocks (fixed function or programmable), and in some examples, one or more of the units may be integrated circuits.
[0201]
[0215] The video encoder 200 may include an arithmetic logic unit (ALU), an elementary functional unit (EFU), digital circuitry, analog circuitry, and / or a programmable core formed from programmable circuitry. In examples in which the operations of the video encoder 200 are implemented using software executed by programmable circuitry, the memory 106 (FIG. 1) may store object code for the software that the video encoder 200 receives and executes, or another memory (not shown) within the video encoder 200 may store such instructions.
[0202]
[0216] The video data memory 230 is configured to store the received video data. The video encoder 200 may retrieve pictures of the video data from the video data memory 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data memory 230 may be raw video data to be encoded.
[0203]
[0217] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra prediction unit 226. The mode selection unit 202 may include additional functional units for performing video prediction according to other prediction modes. By way of example, the mode selection unit 202 may include a palette unit, an intra block copy unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, or the like.
[0204]
[0218] The mode selection unit 202 typically adjusts multiple encoding passes to test combinations of encoding parameters and results in rate-distortion values for such combinations. The encoding parameters may include partitioning of CTUs into CUs, prediction modes for CUs, transform types for residual data of CUs, quantization parameters for residual data of CUs, etc. The mode selection unit 202 may ultimately select a combination of encoding parameters that has a better rate-distortion value than the other tested combinations.
[0205]
[0219] Video encoder 200 may partition a picture retrieved from video data memory 230 into a series of CTUs and encapsulate one or more CTUs within a slice. Mode selection unit 202 may partition the CTUs of a picture according to a tree structure, such as the QTBT structure or the HEVC quad-tree structure described above. As described above, video encoder 200 may form one or more CUs from partitioning the CTUs according to the tree structure. Such CUs may also be generally referred to as "video blocks" or "blocks."
[0206]
[0220] In general, the mode selection unit 202 also controls its components (e.g., the motion estimation unit 222, the motion compensation unit 224, and the intra prediction unit 226) to generate a prediction block for a current block (e.g., a current CU, or, in HEVC, the overlapping portion of a PU and a TU). For inter prediction of a current block, the motion estimation unit 222 may perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previously coded pictures stored in the DPB 218). In particular, the motion estimation unit 222 may calculate a value representing how similar a potential reference block is to the current block, for example, according to a sum of absolute differences (SAD), a sum of squared differences (SSD), a mean absolute difference (MAD), a mean squared difference (MSD), or the like. The motion estimation unit 222 may generally perform these calculations using a sample-by-sample difference between the current block and the reference block under consideration. Motion estimation unit 222 may identify the reference block having the lowest value resulting from these calculations, which indicates the reference block that most closely matches the current block.
[0207]
[0221] The motion estimation unit 222 may form one or more motion vectors (MVs) that specify the location of a reference block in the reference picture relative to the location of the current block in the current picture. The motion estimation unit 222 may then provide the motion vectors to the motion compensation unit 224. For example, for unidirectional inter prediction, the motion estimation unit 222 may provide a single motion vector, while for bidirectional inter prediction, the motion estimation unit 222 may provide two motion vectors. The motion compensation unit 224 may then use the motion vectors to generate a prediction block. For example, the motion compensation unit 224 may use the motion vectors to retrieve data for the reference blocks. As another example, if the motion vectors have fractional sample precision, the motion compensation unit 224 may interpolate values for the prediction block according to one or more interpolation filters. Furthermore, for bidirectional inter prediction, the motion compensation unit 224 may retrieve data for the two reference blocks identified by the respective motion vectors and combine the retrieved data, such as through sample-wise averaging or weighted averaging.
[0208]
[0222] As another example, for intra prediction or intra predictive coding, intra prediction unit 226 may generate a predictive block from samples neighboring the current block. For example, for a directional mode, intra prediction unit 226 may generally mathematically combine values of neighboring samples and store these calculated values in a prescribed direction across the current block to generate a predictive block. As another example, for a DC mode, intra prediction unit 226 may calculate an average of neighboring samples for the current block and generate a predictive block to include this resulting average for each sample of the predictive block.
[0209]
[0223] The mode selection unit 202 provides the prediction block to a residual generation unit 204. The residual generation unit 204 receives a raw uncoded version of the current block from the video data memory 230 and receives the prediction block from the mode selection unit 202. The residual generation unit 204 calculates sample-by-sample differences between the current block and the prediction block. The resulting sample-by-sample differences define a residual block for the current block. In some examples, the residual generation unit 204 may also determine differences between sample values in the residual block to generate the residual block using residual pulse code modulation (RDPCM). In some examples, the residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.
[0210]
[0224] In an example where the mode selection unit 202 partitions a CU into PUs, each PU may be associated with a luma prediction unit and a corresponding chroma prediction unit. The video encoder 200 and the video decoder 300 may support PUs having various sizes. As indicated above, the size of a CU may refer to the size of a luma coding block of the CU, and the size of a PU may refer to the size of a luma prediction unit of the PU. Assuming that a size of a particular CU is 2N×2N, the video encoder 200 may support a PU size of 2N×2N or N×N for intra prediction, and a symmetric PU size of 2N×2N, 2N×N, N×2N, N×N, or the like for inter prediction. The video encoder 200 and the video decoder 300 may also support asymmetric partitioning for PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N.
[0211]
[0225] In examples where the mode select unit 202 does not further partition the CUs into PUs, each CU may be associated with a luma coding block and a corresponding chroma coding block. As described above, the size of a CU may refer to the size of the luma coding block of the CU. The video encoder 200 and the video decoder 300 may support CU sizes of 2N×2N, 2N×N, or N×2N.
[0212]
[0226] For other video coding techniques, such as intra block copy mode coding, affine mode coding, and linear model (LM) mode coding, in some examples, the mode select unit 202 generates a prediction block for the current block being encoded via a respective unit related to the coding technique. In some examples, such as palette mode coding, the mode select unit 202 may not generate a prediction block, but instead generate syntax elements that indicate how to reconstruct the block based on a selected palette. In such modes, the mode select unit 202 may provide these syntax elements to the entropy encoding unit 220 to be encoded.
[0213]
[0227] As described above, the residual generation unit 204 receives video data for a current block and a corresponding predictive block. The residual generation unit 204 then generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates sample-by-sample differences between the predictive block and the current block.
[0214]
[0228] Transform processing unit 206 applies one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transforms to the residual block to form the transform coefficient block. For example, transform processing unit 206 may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform to the residual block. In some examples, transform processing unit 206 may perform multiple transforms on the residual block, e.g., a linear transform and a secondary transform, such as a rotation transform. In some examples, transform processing unit 206 does not apply a transform to the residual block.
[0215]
[0229] The quantization unit 208 may quantize the transform coefficients in the transform coefficient block to generate a quantized transform coefficient block. The quantization unit 208 may quantize the transform coefficients of the transform coefficient block according to a quantization parameter (QP) value associated with the current block. The video encoder 200 (e.g., via the mode selection unit 202) may adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may result in a loss of information, and thus the quantized transform coefficients may have less precision than the original transform coefficients generated by transform processing unit 206.
[0216]
[0230] The inverse quantization unit 210 and the inverse transform processing unit 212 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficient block to reconstruct a residual block from the transform coefficient block. The reconstruction unit 214 may generate a reconstructed block that corresponds to the current block (potentially with some distortion) based on the reconstructed residual block and the predictive block generated by the mode selection unit 202. For example, the reconstruction unit 214 may add samples of the reconstructed residual block to corresponding samples from the predictive block generated by the mode selection unit 202 to generate the reconstructed block.
[0217]
[0231] Filter unit 216 may perform one or more filter operations on the reconstructed blocks. For example, filter unit 216 may perform a deblocking operation to reduce blockiness artifacts along edges of a CU. In some examples, operations of filter unit 216 may be skipped.
[0218]
[0232] The video encoder 200 stores the reconstructed blocks in the DPB 218. For example, in examples where the operation of the filter unit 216 is not required, the reconstruction unit 214 may store the reconstructed blocks in the DPB 218. In examples where the operation of the filter unit 216 is required, the filter unit 216 may store the reconstructed and filtered blocks in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 may retrieve reference pictures formed from the reconstructed (and potentially filtered) blocks from the DPB 218 to inter predict blocks of a picture to be subsequently encoded. In addition, the intra prediction unit 226 may intra predict other blocks in the current picture using the reconstructed blocks in the DPB 218 of the current picture.
[0219]
[0233] In general, the entropy encoding unit 220 may entropy encode syntax elements received from other functional components of the video encoder 200. For example, the entropy encoding unit 220 may entropy encode quantized transform coefficient blocks from the quantization unit 208. As another example, the entropy encoding unit 220 may entropy encode predictive syntax elements (e.g., motion information for inter prediction or intra mode information for intra prediction) from the mode selection unit 202. The entropy encoding unit 220 may perform one or more entropy encoding operations on syntax elements, which are other examples of video data, to generate entropy encoded data. For example, entropy encoding unit 220 may perform a context-adaptive variable length coding (CAVLC) operation, a CABAC operation, a variable-to-variable (V2V) length coding operation, a syntax-based context-adaptive binary arithmetic coding (SBAC) operation, a probability interval partitioning entropy (PIPE) coding operation, an exponential-Golomb encoding operation, or another type of entropy encoding operation on the data. In some examples, entropy encoding unit 220 may operate in a bypass mode in which syntax elements are not entropy encoded.
[0220]
[0234] The video encoder 200 may output a bitstream that includes entropy encoded syntax elements needed to reconstruct blocks of a slice or picture. In particular, the entropy encoding unit 220 may output the bitstream.
[0221]
[0235] The operations described above are described with respect to blocks. Such descriptions should be understood as operations on luma coding blocks and / or chroma coding blocks. As mentioned above, in some examples, the luma coding blocks and chroma coding blocks are luma and chroma components of a CU. In some examples, the luma coding blocks and chroma coding blocks are luma and chroma components of a PU.
[0222]
[0236] In some examples, operations performed with respect to luma coding blocks do not need to be repeated for chroma coding blocks. As one example, operations identifying motion vectors (MVs) and reference pictures for luma coding blocks do not need to be repeated to identify MVs and reference pictures for chroma blocks. Rather, the MVs for luma coding blocks may be scaled to determine MVs for chroma blocks, and the reference pictures may be the same. As another example, the intra prediction process may be the same for luma coding blocks and chroma coding blocks.
[0223]
[0237] The video encoder 200 may be configured to perform any of the predictive block grouping techniques described in this disclosure. For example, the intra prediction unit 226 may perform ISP coding to partition a coding unit (CU) into sub-blocks (e.g., predictive blocks). In addition, the intra prediction unit 226 may group the sub-blocks into predictive block groups. In this manner, the video encoder 200 may be considered to include one or more processors configured and implemented in a circuit to perform partitioning (e.g., vertical partitioning) of a coding unit (CU) of video data using intra sub-partitioning (ISP) to form a set of predictive blocks. The predictive blocks may include at least a first predictive block and a second predictive block. The video encoder 200 may then group more than one of the set of predictive blocks into a first predictive block group (PBG). In addition, the video encoder 200 may reconstruct samples of the predictive blocks included in the first PBG independently of samples of other predictive blocks included in the first PBG.
[0224]
[0238] FIG. 21 is a block diagram illustrating an example video decoder 300 that may utilize techniques of this disclosure. FIG. 21 is provided for illustrative purposes and is not limited to the techniques as broadly illustrated and described in this disclosure. For illustrative purposes, this disclosure describes a video decoder 300 in accordance with JEM, VVC, and HEVC techniques. However, the techniques of this disclosure may be performed by video coding devices configured for other video coding standards.
[0225]
[0239] In the example of Figure 21, the video decoder 300 includes a coded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, and a decoded picture buffer (DPB) 314. Any or all of the CPB memory 320, the entropy decoding unit 302, the prediction processing unit 304, the inverse quantization unit 306, the inverse transform processing unit 308, the reconstruction unit 310, the filter unit 312, and the DPB 314 may be implemented in one or more processors or processing circuits. Additionally, the video decoder 300 may include additional or alternative processors or processing circuits to perform these or other functions.
[0226]
[0240] Prediction processing unit 304 includes a motion compensation unit 316 and an intra prediction unit 318. Prediction processing unit 304 may include additional units for performing prediction according to other prediction modes. By way of example, prediction processing unit 304 may include a palette unit, an intra block copy unit (which may form part of motion compensation unit 316), an affine unit, a linear model (LM) unit, or the like. In other examples, video decoder 300 may include more, fewer, or different functional components.
[0227]
[0241] The CPB memory 320 may store video data, such as an encoded video bitstream, to be decoded by components of the video decoder 300. The video data stored in the CPB memory 320 may be obtained, for example, from the computer-readable medium 110 (FIG. 1). The CPB memory 320 may include a CPB that stores encoded video data (e.g., syntax elements) from the encoded video bitstream. The CPB memory 320 may also store video data other than syntax elements of a coded picture, such as temporary data representing output from various units of the video decoder 300. The DPB 314 generally stores a decoded picture that the video decoder 300 may output and / or use as reference video data when decoding subsequent data or pictures of the encoded video bitstream. The CPB memory 320 and the DPB 314 may be formed by any of a variety of memory devices, such as synchronous dynamic random access memory (DRAM), including DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The CPB memory 320 and the DPB 314 may be provided by the same memory device or separate memory devices. In various examples, the CPB memory 320 may be on-chip with other components of the video decoder 300 or off-chip relative to these components.
[0228]
[0242] Additionally or alternatively, in some examples, the video decoder 300 may retrieve the coded video data from memory 120 (FIG. 1). That is, memory 120 may store data as described above using CPB memory 320. Similarly, when some or all of the functionality of the video decoder 300 is implemented in software executed by processing circuitry of the video decoder 300, memory 120 may store instructions to be executed by the video decoder 300.
[0229]
[0243] The various units shown in FIG. 21 are illustrated to aid in understanding the operations performed by the video decoder 300. The units may be implemented as fixed function circuits, programmable circuits, or a combination thereof. As with FIG. 20, fixed function circuits refer to circuits that provide specific functionality and are preconfigured for the operations they can perform. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations they can perform. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the software or firmware instructions. Although a fixed function circuit may execute software instructions (e.g., to receive parameters or output parameters), the types of operations the fixed function circuit performs are generally immutable. In some examples, one or more of the units may be separate circuit blocks (fixed function or programmable), and in some examples, one or more of the units may be integrated circuits.
[0230]
[0244] The video decoder 300 may include a programmable core formed of ALUs, EFUs, digital circuits, analog circuits, and / or programmable circuits. In examples in which the operations of the video decoder 300 are performed by software executing on programmable circuits, on-chip or off-chip memory may store instructions of the software (e.g., object code) that the video decoder 300 receives and executes.
[0231]
[0245] The entropy decoding unit 302 may receive the encoded video data from the CPB and entropy decode the video data to recover the syntax elements. The prediction processing unit 304, the inverse quantization unit 306, the inverse transform processing unit 308, the reconstruction unit 310, and the filter unit 312 may generate decoded video data based on the syntax elements extracted from the bitstream.
[0232]
[0246] Generally, the video decoder 300 reconstructs a picture on a block-by-block basis. The video decoder 300 may perform a reconstruction operation on each block individually (the block currently being reconstructed, i.e., decoded, may be referred to as the "current block").
[0233]
[0247] The entropy decoding unit 302 may entropy decode transform information, such as a quantization parameter (QP) and / or a transform mode indication, along with syntax elements that define the quantized transform coefficients of the quantized transform coefficient block. The inverse quantization unit 306 may use the QP associated with the quantized transform coefficient block to determine the degree of quantization, and likewise the degree of inverse quantization that the inverse quantization unit 306 applies. The inverse quantization unit 306 may inverse quantize the quantized transform coefficients, for example, by performing a bitwise left shift operation. The inverse quantization unit 306 may thereby form a transform coefficient block including the transform coefficients.
[0234]
[0248] After the inverse quantization unit 306 forms a transform coefficient block, the inverse transform processing unit 308 may apply one or more inverse transforms to the transform coefficient block to generate a residual block related to the current block. For example, the inverse transform processing unit 308 may apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotational transform, an inverse transform, or another inverse transform to the transform coefficient block.
[0235]
[0249] Further, prediction processing unit 304 generates a prediction block according to the prediction information syntax element entropy decoded by entropy decoding unit 302. For example, if the prediction information syntax element indicates that the current block is inter predicted, motion compensation unit 316 may generate a prediction block. In this case, the prediction information syntax element may indicate a reference picture in DPB 314 from which the reference block should be retrieved, as well as a motion vector that identifies the location of the reference block in the reference picture relative to the location of the current block in the current picture. Motion compensation unit 316 may generally perform an inter prediction process in a manner substantially similar to that described with respect to motion compensation unit 224 (FIG. 20).
[0236]
[0250] As another example, if the prediction information syntax element indicates that the current block is intra predicted, the intra prediction unit 318 may generate a prediction block according to the intra prediction mode indicated by the prediction information syntax element. Again, the intra prediction unit 318 may generally perform an intra prediction process in a manner substantially similar to that described with respect to the intra prediction unit 226 (FIG. 20). The intra prediction unit 318 may retrieve data of neighboring samples for the current block from the DPB 314.
[0237]
[0251] The reconstruction unit 310 may use the predictive block and the residual block to reconstruct the current block. For example, the reconstruction unit 310 may add samples of the residual block to corresponding samples of the predictive block to reconstruct the current block.
[0238]
[0252] Filter unit 312 may perform one or more filter operations on the reconstructed blocks. For example, filter unit 312 may perform a deblocking operation to reduce blockiness artifacts along edges of the reconstructed blocks. The operations of filter unit 312 need not be performed in all instances.
[0239]
[0253] The video decoder 300 may store the reconstructed blocks in the DPB 314. For example, in examples where the operations of the filter unit 312 are not performed, the reconstruction unit 310 may store the reconstructed blocks in the DPB 314. In examples where the operations of the filter unit 312 are performed, the filter unit 312 may store the reconstructed and filtered blocks in the DPB 314. As described above, the DPB 314 may provide reference information, such as samples of a current picture for intra prediction and previously decoded pictures for subsequent motion compensation, to the prediction processing unit 304. Additionally, the video decoder 300 may output the decoded pictures from the DPB 314 for subsequent presentation on a display device, such as the display device 118 of FIG. 1.
[0240]
[0254] The video decoder 300 may be configured to perform any of the predictive block grouping techniques described in this disclosure. For example, the intra prediction unit 318 may perform ISP coding to partition a coding unit (CU) into sub-blocks (e.g., predictive blocks). In addition, the intra prediction unit 318 may group the sub-blocks into predictive block groups. In this manner, the video decoder 300 may be considered to include one or more processors configured and implemented in a circuit to perform partitioning (e.g., vertical partitioning) of a coding unit (CU) of video data using intra sub-partitioning (ISP) to form a set of predictive blocks. The predictive blocks may include at least a first predictive block and a second predictive block. The video decoder 300 may group more than one of the set of predictive blocks into a first predictive block group. In addition, the video decoder 300 may reconstruct samples of predictive blocks included in the first PBG independent of samples of other predictive blocks included in the first PBG.
[0241]
[0255] 22 is a flowchart illustrating an example of a method for encoding a current block. The current block may include a current CU. Although described with respect to video encoder 200 (FIGS. 1 and 20), it should be understood that other devices may be configured to perform a method similar to that of FIG.
[0242]
[0256] In this example, video encoder 200 may first predict a current block (350). For example, video encoder 200 may form a predictive block for the current block. Video encoder 200 may then calculate a residual block for the current block (352). In other words, video encoder 200 may calculate luma residual data and chroma residual data for the current block. To calculate the residual block, video encoder 200 may calculate a difference between an original uncoded block and the predictive block for the current block. Video encoder 200 may then transform and quantize coefficients of the residual block (354). Video encoder 200 may then scan the quantized transform coefficients of the residual block (356). During or following the scan, video encoder 200 may entropy encode the transform coefficients (358). For example, video encoder 200 may encode the transform coefficients using CAVLC or CABAC. The video encoder 200 may then output the entropy encoded data for the block (360). For example, the video encoder 200 may implement a reconstruction loop to reconstruct the coded video data using the predictive block grouping techniques described in this disclosure.
[0243]
[0257] 23 is a flow chart illustrating an example method for decoding a current block of video data. The current block may include a current CU. Although described with respect to video decoder 300 (FIGS. 1 and 21), it should be understood that other devices may be configured to perform a method similar to that of FIG.
[0244]
[0258] The video decoder 300 may receive entropy coded data for the current block, such as entropy coded prediction information and entropy coded data for transform coefficients of the residual block that corresponds to the current block (370). The video decoder 300 may entropy decode the entropy coded data to determine prediction information for the current block and reconstruct transform coefficients of the residual block (372). The video decoder 300 may predict the current block using, for example, the intra-prediction mode or inter-prediction mode indicated by the prediction information for the current block to compute a prediction block for the current block (374). The video decoder 300 may then inverse scan the reconstructed transform coefficients to generate a block of quantized transform coefficients (376). The video decoder 300 may then inverse quantize and inverse transform the coefficients to generate a residual block (378). The video decoder 300 may finally decode the current block by combining the prediction block and the residual block (380). The video decoder 300 may reconstruct the samples of the current block using the predictive block grouping techniques described in this disclosure.
[0245]
[0259] 24 is a flowchart illustrating an example technique for performing predictive block grouping on a current block of video data. The current block may include a current CU. Although described with respect to video encoder 200 (FIGS. 1 and 20) and / or video decoder 300 (FIGS. 1 and 21), it should be understood that other devices may be configured to perform a method similar to that of FIG. 24.
[0246]
[0260] In some examples, a video coder (e.g., video encoder 200 and / or video decoder 300) may perform a partition (e.g., vertical partition) of a coding unit (CU) of video data using intra subpartitioning (ISP). For example, the video coder may perform a vertical and / or horizontal partition of a first CU to form a set of predictive blocks (2402). In an illustrative example, the video coder may perform a vertical partition of a first CU to form a set of predictive blocks. In some cases, the set of predictive blocks may include all predictive blocks of the CU or only a subset of the predictive blocks of the CU. That is, the video coder may form a set of predictive blocks that does not include all predictive blocks of the CU. For example, the video coder may perform a partition of a first CU to form a first set of predictive blocks and one or more other sets of predictive blocks. The one or more other sets of predictive blocks may be mutually exclusive with each other and with the first set of predictive blocks. In some examples, the sets of predictive blocks may overlap each other to some extent, such that a portion of a CU may be partitioned into more than one predictive block. Although various techniques of this disclosure are described in terms of a video coder partitioning a CU into a set of predictive blocks, the techniques of this disclosure are not so limited, and a video coder may partition a CU into multiple sets of predictive blocks.
[0247]
[0261] In some examples, the predictive blocks of the set of predictive blocks may include at least a first predictive block and a second predictive block (e.g., at least two vertical predictive blocks). In some examples, the first predictive block may comprise a sample width of 2 or less. For example, the first predictive block may have a sample width of 1 or 2. In addition, the second predictive block may have a sample width of 1 or 2. In some examples, the first predictive block and the second predictive block may have the same width. For example, the first predictive block and the second predictive block may both have a sample width of 2. However, in some examples, the first predictive block and the second predictive block may have different widths, such that the first predictive block has a width of 2 and the second predictive block has a width of 1. In such examples, a third predictive block may be included in the first PBG, such that the first PBG may have a width of at least 4 sample sizes.
[0248]
[0262] In some examples, the video coder may group multiple predictive blocks from the set of predictive blocks into a first predictive block group (PBG) (2404). In examples, the video coder may group one or more predictive blocks from the first set of predictive blocks into the first PBG. For example, the video coder may group a first predictive block and a second predictive block from the first set of predictive blocks into the first PBG. In some examples, the first PBG may include only one predictive block from the first set of predictive blocks.
[0249]
[0263] In some examples, the video coder may group a first set of predictive blocks of a CU to form one or more PBGs. In such examples, the video coder may group PBGs similarly or differently with respect to one or more other sets of predictive blocks of the same CU. For example, the video coder may group at least one vertical predictive block from a first set of predictive blocks and at least one horizontal predictive block from the same set of predictive blocks into a first PBG. In such examples, the video coder may group at least one vertical predictive block from a second set of predictive blocks and at least one horizontal predictive block from the second set of predictive blocks into a second PBG. However, in some examples, the video coder may instead group only vertical predicted blocks from the second set of predictive blocks and / or only horizontal predicted blocks from the second set of predictive blocks into a second PBG, where the first set of predictive blocks includes only vertical sub-blocks, only horizontal sub-blocks, a mix of vertical and horizontal blocks, etc., and the second set of predictive blocks includes only vertical sub-blocks, only horizontal sub-blocks, a mix of vertical and horizontal blocks, etc. In some examples, the video coder may group only predictive blocks of a particular type into a PBG. In a non-limiting example involving CUs partitioned using multiple partition types (e.g., vertical and horizontal partitioning of a single CU), the video coder may group only predictive blocks that are the result of the vertical partitioning portion of the overall partitioning scheme into a PBG. As such, the video coder may precede grouping of predictive blocks that the video coder partitioned differently (e.g., using horizontal partitioning). In another example, the video coder may form a group that includes a vertical predicted block and one or more horizontal predicted blocks adjacent to the vertical predicted block. The video coder may additionally form a group including any remaining predictive blocks (e.g., non-adjacent predictive blocks) or may decide not to form a group for a predictive block in one or more sets of predictive blocks for the CU.In addition, the video coder may group only predictive blocks that are of a particular size (eg, a size that meets a predetermined size threshold).
[0250]
[0264] In some examples, the PBG may be specified to have a minimum dimension of nW samples. For example, the PBG may have a minimum width of 4 samples (e.g., nW=4). That is, the first PBG may have a width of at least 4 sample sizes. In such a case, if the dimension size of the subblock is smaller than nW, the adjacent one or more predictive blocks may be treated as one PBG of dimension size nW samples. The dimension size may apply to the height, width, or a function derived from the height and width (e.g., width*height).
[0251]
[0265] The video coder may reconstruct samples of the predictive block included in the first PBG independently of samples of other predictive blocks included in the first PBG (2406). When the video coder reconstructs a sample from one subblock to identify or reconstruct a sample in another subblock in the same PBG without accessing, reading, or performing other memory access procedures, the video coder may reconstruct samples of the predictive block independently of other predictive blocks in the PBG. As a corollary, when the video coder performs sequential operations, the video coder may reconstruct samples from one group that are dependent on reconstruction of samples from another group. That is, sequential operations involve first reconstructing one or more samples from the first group, and then reconstructing one or more samples from another group based on the samples from the first group. In some cases, independent reconstruction may involve a parallel procedure in which the video coder reconstructs samples from a first predictive block of a first PBG in parallel or substantially simultaneously (e.g., without any required time delay) with the video coder reconstructing samples from a second predictive block in the same first PBG.
[0252]
[0266] In an example where the first PBG includes a first predictive block and a second predictive block, the video coder may reconstruct samples of the first predictive block independently of samples of the second predictive block. In some examples, when a filtering operation is applied to a predictive block that may result in a dependency between predicted values of samples in the PBG, the restriction on the prediction dependency of samples in the PBG may still be considered valid, as long as normal intra prediction (without post-predictive filtering operations) does not result in the dependency. For example, the restriction may not apply to dependencies due to PDPC, boundary filtering, or other similar operations.
[0253]
[0267] In some examples, the video coder may form a second PBG such that a single CU includes at least two PBGs. For example, the video coder may group one or more predictive blocks into the first PBG. Similarly, the video coder may group a second set of predictive blocks into the second PBG. The predictive blocks of each PBG may include non-overlapping sets of predictive blocks. That is, the predictive blocks of one PBG may be mutually exclusive with the predictive blocks of another PBG. In either case, the video coder may reconstruct samples of predictive blocks included in the second PBG independently of samples of other predictive blocks included in the second PBG.
[0254]
[0268] In addition, the video coder may, in some examples, reconstruct samples of a predictive block included in a second PBG based on samples of the predictive block included in a first PBG. That is, reconstruction of samples in one PBG may depend on reconstruction of samples in another PBG. In such examples, the video coder may still maintain prediction independence of samples within a single PBG. In some examples, prediction of samples included in a second PBG may not be based on a predictive block included in a first PBG, but may instead be based on a predictive block included in another CU, such as a neighboring CU.
[0255]
[0269] In some examples, the video coder may divide a CU into one or more transform blocks. In such examples, the size of the transform block may be equal to the size of the first predictive block or any one of the predictive blocks of the CU. In some examples, the size of the transform block may be smaller than the size of the smallest predictive block included in the first PBG. Furthermore, the size of the transform block may be smaller than the size of the sum of the predictive blocks included in the first PBG. In another example, the transform block may have a size equal to the size of the PBG. In either case, the first transform block associated with the first PBG may have a size equal to the size of the first predictive block or the second predictive block included in the first PBG.
[0256]
[0270] In some examples, the video coder may generate a prediction sample array having a width equal to at least the aggregated width of the first predictive block and the second predictive block. For example, the video coder may output an array of size (nPbW)×(nH). In some examples, the video coder may generate a prediction sample array having a width equal to at least the aggregated width of the first predictive block and the second predictive block when reconstructing samples of the predictive block.
[0257]
[0271] In some examples, the video coder may determine the amount of predictive block groups based on a width of the coding unit and a width of the first predictive block. The video coder may code the CU according to the amount of predictive block groups such that the CU constitutes the amount of predictive block groups. The amount of predictive block groups may be referred to herein as pbFactor (e.g., predictive block group factor). In such cases, the video coder may calculate pbFactor by dividing the width of the CU by the width of the predictive block groups. The width of the PBG may be predefined to have a width of at least a sample size of 4.
[0258]
[0272] In some examples, the video coder may calculate pbFactor by dividing the subblock width (e.g., the predictive block width) by the width of the PBG (e.g., the minimum width of the PBG). For example, when the subblock has a width of 1 or 2, the video coder may calculate pbFactor by dividing the subblock width by the width of the PBG, such as by dividing by 4 when the PBG has a sample width of 4. In another example, when the width of the CU is 8 samples size and the predictive block group is 4 samples size, the video coder may calculate pbFactor to be equal to 2. That is, the number or amount of PBGs for a CU may be equal to two PBGs.
[0259]
[0273] In some examples, when the width of the predictive block is 4 or more, the predictive block width and the PBG width may have the same width. For example, the video coder may group a predictive block of width 4 into a PBG of width 4. That is, the video coder may group predictive blocks of width X into a single PBG of the same width, where X is equal to or greater than a predefined threshold number. In a non-limiting illustrative example, the predefined threshold number may be 4. In another illustrative example, the video coder may group a first predictive block of width 4 and a second predictive block of width 8 into a single PBG of width 8. In such an example, the video coder may include only a portion of the second predictive block in the PBG group, such as by further partitioning the second predictive block to achieve a particular PBG group size. It will be understood that while various techniques of this disclosure are described with width sizes in certain contexts, the techniques of this disclosure are not so limited and other constraints and / or considerations may apply to the grouping of predictive blocks into PBGs as well. For example, a PBG may have a particular height restriction (e.g., a height of 4 sample sizes), or in another example, a combination of height and width constraints and / or considerations may also apply to the grouping of predictive blocks into PBGs (e.g., a PBG may have a minimum H×W size of M×N, where M and N are not necessarily equal).
[0260]
[0274] As explained above, the illustrated example method of Figure 24 may be performed by one or more processors implemented in circuitry that may be included in a video encoder or, in some examples, may be included in a video decoder.
[0261]
[0275] Illustrative examples of the present disclosure include the following:
[0262]
[0276] Example 1: In a method for coding video data, the method includes performing partitioning of a coding unit (CU) of the video data using an intra subpartition (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block; grouping a plurality of predictive blocks from the set of predictive blocks into a first predictive block group (PBG); and reconstructing samples of the predictive blocks included in the first PBG independent of samples of other predictive blocks included in the first PBG.
[0263]
[0277] Example 2: The method of Example 1, wherein grouping a plurality of the sets of predictive blocks into a first PBG includes grouping a first plurality of the sets of predictive blocks into the first PBG, and the method further includes grouping a second plurality of the sets of predictive blocks into a second PBG, the second plurality of the sets of predictive blocks and the first plurality of the sets of predictive blocks including non-overlapping sets of predictive blocks, and reconstructing samples of the predictive blocks included in the second PBG independent of samples of other predictive blocks included in the second PBG.
[0264]
[0278] Example 3: The method of Example 2, wherein reconstructing samples of the predictive block included in the second PBG includes reconstructing samples of the predictive block included in the second PBG based on samples of the predictive block included in the first PBG.
[0265]
[0279] Example 4: The method of any of Examples 1-3, wherein the first PBG comprises a sample size range of at least four.
[0266]
[0280] Example 5: The method according to any of Examples 1 to 4, wherein the first transform block associated with the first PBG has a size equal to the size of the first predictive block.
[0267]
[0281] Example 6: A method described in any of Examples 1 to 5, wherein reconstructing samples of the predictive block includes generating a predictive sample array having a width at least equal to the aggregate width of the first predictive block and the second predictive block.
[0268]
[0282] Example 7: The method of any of Examples 1 to 6, wherein the first prediction block comprises a width of 2 samples or less.
[0269]
[0283] Example 8: The method of example 7, wherein the first prediction block and the second prediction block are of the same width.
[0270]
[0284] Example 9: A method described in any of Examples 1 to 8, further comprising: determining an amount of predictive block groups based on a width of the coding unit and a width of the first predictive block; and coding CUs according to the amount of predictive block groups, such that the CUs constitute the amount of predictive block groups.
[0271]
[0285] Example 10: The method of any of Examples 1 to 9, wherein the method is executed by one or more processors.
[0272]
[0286] Example 11: A method described in any of Examples 1 to 10, further comprising encoding, in the coded video bitstream, one or more syntax elements representing prediction information for a CU of the video data and one or more syntax elements representing residual data for the CU of the video data.
[0273]
[0287] Example 12: A method described in any of Examples 1 to 11, further comprising decoding from the coded video bitstream one or more syntax elements representing prediction information for a CU of the video data and one or more syntax elements representing residual data for the CU of the video data.
[0274]
[0288] Example 13: The method of any of Examples 1 to 12, wherein performing partitioning of the CU includes partitioning the CU using one or more of vertical partitioning, horizontal partitioning, or a combination of horizontal and vertical partitioning.
[0275]
[0289] Example 14: A device for coding video data, the device comprising one or more means for performing any of the methods of Examples 1 to 13. For example, the device of Example 14 includes a memory configured to store video data, and one or more processors implemented in a circuit, the one or more processors configured to perform partitioning of a coding unit (CU) of the video data using an intra subpartition (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block, group a plurality of predictive blocks from the set of predictive blocks into a first predictive block group (PBG), and reconstruct samples of the predictive blocks included in the first PBG independent of samples of other predictive blocks included in the first PBG.
[0276]
[0290] Example 15: A device as described in Example 14, wherein to group a plurality of the sets of predictive blocks into a first PBG, the one or more processors are configured to group a first plurality of the sets of predictive blocks into a first PBG, and the one or more processors are further configured to group a second plurality of the sets of predictive blocks into a second PBG, and reconstruct samples of the predictive blocks included in the second PBG independent of samples of other predictive blocks included in the second PBG, such that the second plurality of the sets of predictive blocks and the first plurality of the sets of predictive blocks include non-overlapping sets of predictive blocks.
[0277]
[0291] Example 16: A device as described in Example 15, wherein to reconstruct samples of a predictive block included in the second PBG, one or more processors are configured to reconstruct samples of a predictive block included in the second PBG based on samples of the predictive block included in the first PBG.
[0278]
[0292] Example 17: The device of any of Examples 14 to 16, wherein the first PBG comprises a sample size range of at least four.
[0279]
[0293] Example 18: The device according to any of Examples 14 to 17, wherein the first transform block related to the first PBG has a size equal to the size of the first prediction block.
[0280]
[0294] Example 19: A device described in any of Examples 14 to 18, wherein to reconstruct samples of a predictive block, one or more processors are configured to generate a predictive sample array having a width at least equal to the aggregate width of the first predictive block and the second predictive block.
[0281]
[0295] Example 20: The device of any of Examples 14 to 19, wherein the first prediction block comprises a sample width of 2 or less.
[0282]
[0296] Example 21: The device of example 20, wherein the first prediction block and the second prediction block are of the same width.
[0283]
[0297] Example 22: A device described in any of Examples 14 to 21, wherein the one or more processors are further configured to determine an amount of predictive block groups based on a width of the coding unit and a width of the first predictive block, and to code the CU according to the amount of predictive block groups, such that the CU configures the amount of predictive block groups.
[0284]
[0298] Example 23: The device of any of Examples 14-22, wherein the one or more processors are included in a video encoder.
[0285]
[0299] Example 24: The device of any of Examples 14 to 23, wherein the one or more processors are included in a video decoder.
[0286]
[0300] Example 25: The device of any of Examples 14 to 24, wherein to perform the partitioning of the CU, the one or more processors are configured to partition the CU using vertical partitioning.
[0287]
[0301] In some implementations, the above-described examples 1-13 and / or 14-25 can be realized using a computer-readable storage medium storing instructions that, when executed, cause one or more processors of a device to perform some or all of the various operations. For example, a computer-readable storage medium can be provided that stores instructions that, when executed, cause one or more processors of a device for decoding and / or encoding video data to perform a partitioning of a coding unit (CU) of the video data using an intra subpartition (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block, group a plurality of predictive blocks from the set of predictive blocks into a first predictive block group (PBG), and reconstruct samples of the predictive blocks included in the first PBG independently of samples of other predictive blocks included in the first PBG.
[0288]
[0302] In some implementations, examples 1-13 and / or 14-25 described above may be realized using an apparatus having one or more means for performing some or all of the various operations. For example, an apparatus for encoding video data may include means for performing vertical partitioning of a coding unit (CU) of the video data using intra subpartitioning (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block, means for grouping a plurality of predictive blocks from the set of predictive blocks into a first predictive block group (PBG), and means for reconstructing samples of the predictive blocks included in the first PBG independent of samples of other predictive blocks included in the first PBG.
[0289]
[0303] Example 26: In a method for coding video data, the method includes coding an indication of performance of generalized prediction for a current coding block of the video data, partitioning the current coding block into one or more predictive blocks based on the indication, defining one or more predictive block groups (PBGs) based on the one or more predictive blocks, determining a coding order of the PBGs, and coding the PBGs in the coding order.
[0290]
[0304] Example 27: The method of Example 26, wherein encoding the PBG includes obtaining predicted values for samples in the current PBG based on available reference samples for the current PBG and the intra mode used for prediction.
[0291]
[0305] Example 28: A method described in any of Examples 26 or 27, wherein dividing the current coding block into one or more predictive blocks includes dividing the current coding block into predictive blocks of size 4×N, wherein only vertical intra-prediction modes are allowed for predicting the predictive blocks, and wherein an indication of the intra-prediction mode for the predictive blocks is not signaled in a bitstream including an encoded representation of the video data.
[0292]
[0306] Example 29: The method of any of Examples 26 to 28, wherein the encoding includes decoding.
[0293]
[0307] Example 30: The method of any of Examples 26 to 28, wherein the coding comprises encoding.
[0294]
[0308] Example 31: A device for coding video data, the device comprising one or more means for performing any of the methods of Examples 26 to 30. For example, the device of Example 31 may include one or more processors configured to: code an indication of performance of generalized prediction for a current coding block of the video data; divide the current coding block into one or more predictive blocks based on the indication; define one or more predictive block groups (PBGs) based on the one or more predictive blocks; determine a coding order of the PBGs; and code the PBGs in the coding order.
[0295]
[0309] Example 32: The device of example 31, wherein the one or more means include one or more processors implemented in the circuit.
[0296]
[0310] Example 33: The device of any of Examples 31 or 32, further comprising a memory for storing video data.
[0297]
[0311] Example 34: The device of any of Examples 31 to 33, further comprising a display configured to display the decoded video data.
[0298]
[0312] Example 35: The device of any of Examples 31-34 comprising one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.
[0299]
[0313] Example 36: The device of any one of Examples 31 to 35, wherein the device comprises a video decoder.
[0300]
[0314] Example 37: The device of any one of Examples 31 to 36, wherein the device comprises a video encoder.
[0301]
[0315] In some implementations, examples 26-30 and / or 31-37 described above may be realized using a computer-readable storage medium storing instructions that, when executed, cause one or more processors of a device to perform some or all of the various operations. For example, a computer-readable storage medium may be provided that stores instructions that, when executed, cause one or more processors of a device that decodes and / or encodes video data to code an indication of performing a generalized prediction on a current coding block of video data, divide the current coding block into one or more predictive blocks based on the indication, define one or more predictive block groups (PBGs) based on the one or more predictive blocks, determine a coding order for the PBGs, and code the PBGs in the coding order.
[0302]
[0316] In some implementations, examples 26-30 and / or 31-37 described above may be realized using an apparatus having one or more means for performing some or all of the various operations. For example, an apparatus for encoding video data may include means for coding an indication of performance of generalized prediction for a current coding block of video data, means for partitioning the current coding block into one or more predictive blocks based on the indication, means for defining one or more predictive block groups (PBGs) based on the one or more predictive blocks, means for determining a coding order for the PBGs, and means for coding the PBGs in the coding order.
[0303]
[0317] It should be appreciated that, depending on the example, certain acts or events of any of the techniques described herein may be performed in a different sequence, may be added, merged, or may be omitted entirely (e.g., not all of the acts or events described may be required for the practice of the techniques). Furthermore, in some examples, acts or events may be performed simultaneously rather than sequentially, for example, through multithreading, interrupt processing, or multiple processors.
[0304]
[0318] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may also include computer-readable storage media, which corresponds to a tangible medium, such as a data storage medium, or a communication medium, which includes any medium that facilitates transfer of a computer program from one place to another, for example according to a communication protocol. Thus, computer-readable media may generally correspond to (1) a tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.
[0305]
[0319] By way of example and not limitation, such computer readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage media, magnetic disk storage media or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and accessed by a computer. Also, any connection is properly referred to as a computer readable medium. For example, if the instructions are transmitted from a website, from a server, or from other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. Disk and disc as used herein include compact discs (CDs), laser discs, optical disks, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, although typically disks reproduce data magnetically while discs reproduce data optically with a laser. Combinations of the above should also be included within the scope of computer readable media.
[0306]
[0320] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, as used herein, the terms "processor" and "processing circuitry" may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in dedicated hardware and / or software modules configured to encode and decode, or may be incorporated in a combined codec. Also, the techniques may be fully implemented in one or more circuits or logic elements.
[0307]
[0321] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs) or sets of ICs (e.g., chipsets). Various components, modules, or units are described herein to highlight functional aspects of devices configured to perform the disclosed techniques, but they do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors as described above, along with appropriate software and / or firmware.
[0308]
[0322] Various examples have been described. These and other examples are within the scope of the following claims. The invention as described in the claims of the present application as originally filed is set forth below. [C1] 1. A method for encoding video data, comprising: performing a partitioning of a coding unit (CU) of the video data using an intra subpartition (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block; grouping a plurality of prediction blocks from the set of prediction blocks into a first prediction block group (PBG); and reconstructing samples of a predictive block included in the first PBG independently of samples of other predictive blocks included in the first PBG. [C2] grouping the plurality of sets of predictive blocks into a first PBG includes grouping a first plurality of the sets of predictive blocks into the first PBG; The method comprises: grouping a second plurality of the sets of predictive blocks into a second PBG, the second plurality of the sets of predictive blocks and the first plurality of the sets of predictive blocks comprising non-overlapping sets of predictive blocks; The method of C1, further comprising reconstructing samples of the predictive blocks included in the second PBG independently of samples of other predictive blocks included in the second PBG. [C3] The method of C2, wherein reconstructing samples of the predictive block included in the second PBG includes reconstructing samples of the predictive block included in the second PBG based on samples of the predictive block included in the first PBG. [C4] The method of C1, wherein said first PBG comprises a sample size range of at least four. [C5] The method of C1, wherein a first transform block associated with the first PBG has a size equal to a size of the first predictive block. [C6] Reconstructing the samples of the predicted block The method of C1 comprising generating a predicted sample array having a width at least equal to an aggregate width of the first predicted block and the second predicted block. [C7] The method of C1, wherein the first prediction block comprises a width of 2 or less samples. [C8] The method of C7, wherein the first predictive block and the second predictive block are of the same width. [C9] determining a quantity of predictive block groups based on a width of the coding unit and a width of the first predictive block; The method of claim 1, further comprising: coding the CUs according to the amount of the predictive block group such that the CUs constitute an amount of the predictive block group. [C10] The method of C1, wherein the method is executed by one or more processors. [C11] The method of C1, further comprising encoding, in the coded video bitstream, one or more syntax elements representing prediction information for the CU of the video data and one or more syntax elements representing residual data for the CU of the video data. [C12] The method of C1, further comprising decoding from the coded video bitstream one or more syntax elements representing prediction information for the CU of the video data and one or more syntax elements representing residual data for the CU of the video data. [C13] The method of C1, wherein performing partitioning of the CU includes partitioning the CU using one or more of vertical partitioning, horizontal partitioning, or a combination of horizontal and vertical partitioning. [C14] 1. A device for encoding video data, comprising: a memory configured to store video data; one or more processors implemented in circuitry; The one or more processors: performing a partitioning of a coding unit (CU) of the video data using an intra subpartition (ISP) to form a set of predictive blocks, such that the predictive blocks include at least a first predictive block and a second predictive block; grouping a plurality of prediction blocks from the set of prediction blocks into a first prediction block group (PBG); A device configured to reconstruct samples of a predictive block included in the first PBG independently of samples of other predictive blocks included in the first PBG. [C15] to group a plurality of the sets of predictive blocks into a first PBG, the one or more processors are configured to group a first plurality of the sets of predictive blocks into the first PBG; the one or more processors: grouping a second plurality of the sets of predictive blocks into a second PBG, such that the second plurality of the sets of predictive blocks and the first plurality of the sets of predictive blocks comprise non-overlapping sets of predictive blocks; The device of C14, further configured to reconstruct samples of a predictive block included in the second PBG independently of samples of other predictive blocks included in the second PBG. [C16] The device of C15, wherein the one or more processors are configured to reconstruct samples of a predictive block included in the second PBG based on samples of a predictive block included in the first PBG, in order to reconstruct samples of a predictive block included in the second PBG. [C17] The device of C14, wherein the first PBG comprises a sample size range of at least four. [C18] The device of C14, wherein a first transform block associated with the first PBG has a size equal to a size of the first prediction block. [C19] To reconstruct samples of the prediction block, the one or more processors: The device of C14, configured to generate a predicted sample array having a width at least equal to an aggregate width of the first predicted block and the second predicted block. [C20] The device of C14, wherein the first prediction block comprises a width of 2 or less samples. [C21] The device of C20, wherein the first prediction block and the second prediction block are of the same width. [C22] the one or more processors: determining an amount of predictive block groups based on a width of the coding unit and a width of the first predictive block; The device of claim 14, further configured to code the CU according to an amount of the predictive block groups such that the CU constitutes an amount of the predictive block groups. [C23] The device of C14, wherein the one or more processors are included in a video encoder. [C24] The device of C14, wherein the one or more processors are included in a video decoder. [C25] The device of C14, wherein to perform the partitioning of the CU, the one or more processors are configured to partition the CU using vertical partitioning. [C26] A non-transitory computer readable storage medium having instructions stored thereon, comprising: The instructions, when executed, cause one or more processors to: performing a partitioning of a coding unit (CU) of the video data using an intra subpartition (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block; grouping a plurality of prediction blocks from the set of prediction blocks into a first prediction block group (PBG); A non-transitory computer-readable storage medium that causes samples of a predictive block included in the first PBG to be reconstructed independently of samples of other predictive blocks included in the first PBG. [C27] 1. An apparatus for encoding video data, comprising: means for performing a vertical partitioning of a coding unit (CU) of the video data using an intra subpartition (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block; means for grouping a plurality of prediction blocks from the set of prediction blocks into a first prediction block group (PBG); and means for reconstructing samples of a predictive block included in the first PBG independently of samples of other predictive blocks included in the first PBG.
Claims
1. 1. A method for encoding video data, comprising: performing a vertical partitioning of a coding unit (CU) of the video data using intra subpartitioning (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block, where ISP is an intra prediction mode that partitions the CU into a plurality of predictive blocks; grouping a plurality of prediction blocks from the set of prediction blocks into a first prediction block group (PBG), and a number of prediction blocks included in the first PBG is based on a width and height of the CU and / or a number of samples of the CU; reconstructing samples of a predictive block included in the first PBG by predicting the samples of the predictive block included in the first PBG using samples neighboring the CU, independent of samples of other predictive blocks included in the first PBG; A method for providing the above.
2. grouping the plurality of the set of predictive blocks into a first PBG comprises grouping a first plurality of the set of predictive blocks into the first PBG; The method comprises: grouping a second plurality of the sets of predictive blocks into a second PBG, the second plurality of the sets of predictive blocks and the first plurality of the sets of predictive blocks being non-overlapping; 2. The method of claim 1, further comprising: reconstructing samples of a predictive block included in the second PBG independent of samples of other predictive blocks included in the second PBG.
3. 3. The method of claim 2, wherein reconstructing samples of a predictive block included in the second PBG comprises reconstructing samples of a predictive block included in the second PBG based on samples of a predictive block included in the first PBG.
4. The method of claim 1 , wherein the first PBG has a sample size range of at least four.
5. The method of claim 1 , wherein a first transform block associated with the first PBG has a size equal to a size of the first predictive block.
6. Reconstructing the samples of the predicted block 2. The method of claim 1, comprising generating a predicted sample array having a width at least equal to an aggregate width of the first predicted block and the second predicted block.
7. 2. The method of claim 1, wherein the first prediction block comprises a width of 2 samples or less.
8. 8. The method of claim 7, wherein the first predictive block and the second predictive block are of the same width.
9. determining a quantity of predictive block groups based on a width of the coding unit and a width of the first predictive block; The method of claim 1 , further comprising: coding the CU according to an amount of the predictive block group such that the CU constitutes an amount of the predictive block group.
10. The method of claim 1 , wherein the method is performed by one or more processors.
11. 2. The method of claim 1 , further comprising encoding, in the coded video bitstream, one or more syntax elements representing prediction information for the CU of video data and one or more syntax elements representing residual data for the CU of video data.
12. 2. The method of claim 1 , further comprising: decoding, from the coded video bitstream, one or more syntax elements representing prediction information for the CU of video data and one or more syntax elements representing residual data for the CU of video data.
13. 1. A device for encoding video data, comprising: a memory configured to store video data; one or more processors implemented in a circuit; The one or more processors: performing a vertical partitioning of a coding unit (CU) of the video data using intra subpartitioning (ISP) to form a set of predictive blocks, the predictive blocks including at least a first predictive block and a second predictive block, where ISP is an intra prediction mode that partitions the CU into a plurality of predictive blocks; grouping a plurality of prediction blocks from the set of prediction blocks into a first prediction block group (PBG), and a number of prediction blocks included in the first PBG is based on a width and height of the CU and / or a number of samples of the CU; reconstructing samples of a predictive block included in the first PBG by predicting the samples of the predictive block included in the first PBG using samples neighboring the CU, independent of samples of other predictive blocks included in the first PBG; The device is configured to:
14. A non-transitory computer readable storage medium storing instructions which, when executed, cause a device according to claim 13 to perform the method according to any one of claims 1 to 12.
Citation Information
Patent Citations
Deblocking non-square blocks for video coding
JP2014525193A
VIDEO DECODING METHOD, VIDEO DECODING DEVICE, AND NON-TRANSITORY COMPUTER-READABLE STORAGE MEDIUM
JP2022518612A
Video coding using intra sub-partition coding mode
WO2020163535A1
