Method and apparatus for quadratic transformation using adaptive kernel option - Patents.com
Adaptive quadratic transformations with kernel options address inefficiencies in video coding by optimizing bit allocation based on intra-prediction modes and block characteristics, enhancing compression efficiency in video coding.
Patent Information
- Application Number
- JP2023528040
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-08-29
- Filing Date
- 2022-08-29
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2042-08-29
AI Technical Summary
Existing video coding techniques face inefficiencies in reducing redundancy and achieving optimal compression ratios due to the varying statistical likelihood of intra-prediction directions, especially with increasing numbers of possible directions in advanced video coding standards like H.265 and JEM/VVC/BMS, leading to suboptimal bit allocation and compression performance.
Implementing quadratic transformations with adaptive kernel options for video blocks, where the number and type of quadratic transform kernels are determined by the intra-prediction mode, block size, and primary transform type, allowing for entropy coding of kernel indices to optimize bit allocation based on statistical likelihood of prediction directions.
Enhances video compression efficiency by adaptively allocating bits to less likely intra-prediction directions, improving compression ratios and reducing redundancy in encoded video streams.
Smart Images

Figure 0007578330000015 
Figure 0007578330000016 
Figure 0007578330000017
Abstract
Description
[Technical field]
[0001] This disclosure describes a series of advanced video coding techniques. More specifically, the techniques disclosed include a quadratic transform with an adaptive kernel option.
[0002] This application claims the benefit of priority to U.S. Patent Application No. 17 / 897,815, filed August 29, 2022, which claims the benefit of priority to U.S. Provisional Application No. 63 / 238,635, filed August 30, 2021, both entitled "ETHOD AND APPARATUS FOR SECONDARY TRANSFORM WITH ADAPTIVE KERNEL OPTIONS." These prior patent applications are incorporated by reference in their entireties. [Background technology]
[0003] This background discussion provided herein is to generally present the context of the present disclosure. The work of the presently named inventors is not admitted expressly or impliedly as prior art to the present disclosure to the extent that that work is described in this background section, as well as in described aspects that may not otherwise qualify as prior art at the time of the filing of this application.
[0004] Video coding and decoding may be performed using inter-picture prediction with motion compensation. Uncompressed digital video may include a sequence of pictures, each having spatial dimensions of, for example, 1920x1080 luma samples and associated full or subsampled chrominance samples. The sequence of pictures may have a fixed or variable picture rate (alternatively referred to as frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bit rate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920x1080, a frame rate of 60 frames per second, and 4:2:0 chroma subsampling with 8 bits per pixel per color channel requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.
[0005] One goal of video coding and decoding may be the reduction of redundancy in an uncompressed input video signal through compression. Compression may help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than one order of magnitude. Both lossless and lossless compression, as well as combinations thereof, may be employed. Lossless compression refers to techniques where an exact copy of the original signal may be reconstructed from the compressed original signal through a decoding process. Lossy compression refers to a coding / decoding process where the original video information is not fully preserved during coding and cannot be fully recovered during decoding. When using lossy, the reconstructed signal may not be identical to the original signal. However, the distortion between the original signal and the reconstructed signal is made small enough to make the reconstructed signal useful for the intended application, even though there is some information loss. For video, lossy compression is widely used in many applications. The amount of distortion that is tolerated is application dependent. For example, a user of a given consumer video streaming application may tolerate higher distortion than a user of a movie or television broadcast application. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect different distortion tolerances: higher tolerable distortion generally allows for coding algorithms that result in higher loss and higher compression ratios.
[0006] Video encoders and decoders can utilize techniques from a number of broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.
[0007] Video codec techniques can include a technique known as intra-coding. In intra-coding, sample values are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially divided into blocks of samples. If all blocks of samples are coded in intra mode, the picture may be referred to as an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as the first picture in a coded video bitstream and video session or as still pictures. The samples of the block after intra-prediction can then be transformed into the frequency domain, and the transform coefficients so generated can be quantized before entropy coding. Intra-prediction represents a technique that minimizes sample values in the pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficients, the fewer bits are needed for a given quantization step size to represent the block after entropy coding.
[0008] Traditional intra-coding, e.g., as known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to code / decode blocks based on surrounding sample data and / or metadata, e.g., acquired during encoding and / or decoding of spatial neighbors and preceding in the decoding order of the intra-coded or decoded data. Such techniques are hereafter referred to as "intra prediction" techniques. Note that in at least some cases, intra-prediction uses reference data only from the current picture being reconstructed and not from other reference pictures.
[0009] There are many different forms of intra prediction. If one or more of such techniques are available in a given video coding technique, the technique being used may be referred to as an intra prediction mode. One or more intra prediction modes may be provided in a particular codec. In certain cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and intra coding parameters for a block of video may be included individually or collectively in a mode codeword. Which codeword to use for a given mode, sub-mode, and / or parameter combination may affect the coding efficiency gain through intra prediction, as may the entropy coding technique used to convert the codeword into a bitstream.
[0010] Certain modes of intra prediction were introduced in H.264, improved in H.265, and further refined in newer coding techniques such as Joint Search Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). In general, for intra prediction, a predictor block may be formed using neighboring sample values that become available. For example, available values for a particular set of neighboring samples along a certain direction and / or line may be copied into the predictor block. A reference to the direction in use may be coded in the bitstream or may itself be predicted.
[0011] Referring to FIG. 1A, shown at the bottom right is a subset of the nine predictor directions specified in the 33 possible intra-prediction directions of H.265 (corresponding to the 33 angle modes out of the 35 modes specified in H.265). The point where the arrows converge (101) represents the sample being predicted. The arrows represent the direction in which neighboring samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from neighboring samples to the top right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from neighboring samples to the bottom left of sample (101) at an angle of 22.5 degrees from the horizontal.
[0012] 1A, a square block (104) of 4×4 samples is shown at the top left (indicated by a dashed bold line). The square block (104) includes 16 samples, each labeled with an “S” including its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample of the block (104) in both the Y and X dimensions. Since the size of the block is 4×4 samples, S44 is at the bottom right. Also shown are exemplary reference samples that follow a similar numbering scheme. The reference samples are labeled with R and their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, predictive samples that neighbor the block being reconstructed are used.
[0013] Intra-picture prediction of a block 104 can start by copying reference sample values from adjacent samples according to a signaled prediction direction. For example, assume that a coded video bitstream includes a signal indicating a prediction direction of an arrow (102) for this block 104. That is, the sample is predicted from the prediction sample to the upper right at an angle of 45 degrees from the horizontal. In such a case, samples S41, S32, S23, S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.
[0014] In certain cases, particularly when the orientation is not evenly divisible by 45 degrees, the values of multiple reference samples may be combined, for example by interpolation, to calculate the reference sample.
[0015] As video coding technology develops, the number of possible directions is increasing. For example, in H.264 (2003), nine different directions are available for intra prediction. This increases to 33 in H.265 (2013), and at the time of this disclosure, JEM / VVC / BMS can support 65 child directions. Experimental studies have been performed to help identify the most suitable intra prediction directions, and certain techniques in entropy coding may be used to encode those most suitable directions with a small number of bits, accepting a certain bit penalty for the direction. Furthermore, the direction itself can sometimes be predicted from neighboring directions used in intra prediction of decoded neighboring blocks.
[0016] FIG. 1B shows a schematic diagram (180) depicting the 65 intra-prediction directions according to the JEM, illustrating the increase in the number of prediction directions in various encoding techniques developed over time.
[0017] The mapping method of bits representing intra-prediction directions to prediction directions in a coded video bitstream may vary from one video coding technique to another and may range, for example, from simple direct mapping of prediction directions to complex adaptation schemes involving intra-prediction modes, codewords, most likely modes, and similar techniques. However, in all cases, there may be intra-prediction directions that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, in a well-designed video coding technique, less likely directions may be represented by more bits than more likely directions.
[0018] Inter picture prediction, or inter prediction, may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or part thereof (reference picture) may be used for prediction of a newly reconstructed picture or picture part (e.g., block) after being spatially shifted in a direction indicated by a motion vector (hereinafter, MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or three dimensions, with the third dimension being an indication of the reference picture being used (similar to the time dimension).
[0019] In some video compression techniques, the current MV applicable to a given region of sample data can be predicted from other MVs, e.g., from other MVs related to other regions of sample data that are spatially adjacent to the region being reconstructed and that precede the current MV in decoding order. Doing so can substantially reduce the overall amount of data required to code the MV by relying on the removal of redundancy in correlated MVs, thereby increasing compression efficiency. MV prediction can work effectively, for example, when coding an input video signal derived from a camera (known as natural video), because there is a statistical probability that regions larger than the region to which a single MV applies move in a similar direction in the video sequence, and therefore, in some cases, can be predicted using similar motion vectors derived from MVs of neighboring regions. As a result, the actual MV for a given region is similar or identical to the MV predicted from the surrounding MVs. Such an MV can then be represented with fewer bits after entropy coding than would be used if the MV were directly coded instead of predicted from neighboring MVs. In some cases, the MV prediction may be an example of lossless compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, the MV prediction itself may be lossy, for example due to rounding errors in computing a predictor from some surrounding MVs.
[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, “High Efficiency Video Coding”, December 2016). Among the many MV prediction mechanisms specified by H.265, one that will be described later is a technique called “spatial merge”.
[0021] Specifically, referring to Figure 2, a current block (201) contains samples that the encoder found during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of coding its MV directly, the MV can be derived from metadata associated with one or more reference pictures, e.g., from the most recent reference picture (in decoding order), using MVs associated with any of the five surrounding samples denoted A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use predictors from the same reference picture as neighboring blocks use. Summary of the Invention [Problem to be solved by the invention]
[0022] Aspects of the present disclosure provide a method and apparatus for a quadratic transform with an adaptive kernel option.
[0023] In some implementations, a method for processing a video block in an encoded video stream is disclosed. The method may include analyzing and processing the encoded video stream to generate a set of secondary transform coefficients for the video block, an intra-prediction mode associated with the video block, and a kernel index into a group of secondary transform kernels. The method may further include identifying the group of secondary transform kernels based on the intra-prediction mode, and performing an inverse secondary transform of the set of secondary transform coefficients to generate primary transform coefficients for the video block based on a secondary transform kernel from the group of secondary transform kernels identified by the kernel index. A number of kernels in the group of secondary transform kernels depends on at least one of the intra-prediction mode associated with the video block, a size of the video block, or a primary transform type associated with the video block.
[0024] In some other implementations, another method of processing an intra-predicted video block is disclosed. The method may include transforming a residual block of an intra-predicted video block using at least one primary transform kernel to generate a set of primary transform coefficients, and selecting a current secondary transform kernel within a group of secondary transform kernels, where a number of kernels within the group of secondary transform kernels depends on at least one of an intra-prediction mode of the intra-predicted video block, a size of the intra-predicted video block, or a primary transform type of the at least one primary transform kernel. The method may further include determining a kernel index of the current secondary transform kernel within the group of secondary transform kernels, transforming the set of primary transform coefficients, where the current secondary transform kernel is used to generate a set of secondary transform coefficients, and encoding the kernel index and the set of secondary transform coefficients into an encoded video stream associated with the intra-predicted video block.
[0025] In any of the above implementations, the number of kernels in the group of secondary transform kernels depends on the primary transform type associated with the video block.
[0026] In any of the above implementations, the number of kernels in the group of secondary transform kernels is N if the primary transform type is ADST for both the vertical and horizontal dimensions of the video block, and is K otherwise, where N and K are different non-negative integers between 0 and 6.
[0027] In any of the above implementations, the number of kernels in the group of secondary transform kernels is N if the size of the video block is smaller than a predetermined threshold, and is K otherwise, where N and K are different non-negative integers between 0 and 6.
[0028] In any of the above implementations, the predetermined threshold is 8x8 or 8 for either the horizontal or vertical dimension of the video block.
[0029] In any of the above implementations, a number of kernels in the group of secondary transform kernels is N if the video block is characterized by any one of a predefined subset of intra-prediction modes, any one of a predefined subset of primary transform types, and any one of a predefined subset of block sizes, and is K otherwise; N and K are different non-negative integers between 0 and 6.
[0030] In any of the above implementations, the number of kernels in the group of secondary transform kernels is N if the video block is characterized by one of vertical, horizontal, smooth horizontal, and smooth vertical intra prediction modes, ADST_ADST primary transform modes in both the vertical and horizontal dimensions, and a block size smaller than a predefined size threshold, and is K otherwise.
[0031] In any of the above implementations, if the group of secondary transform kernels contains only N or K secondary transform kernels, the kernel index signaling scheme in the encoded video stream supports only N or up to N index options.
[0032] In any of the above implementations, when the number of kernels in the group of secondary transform kernels is N and the number of kernels in the group of secondary transform kernels is K, the kernel index is entropy coded in the encoded video stream using a different context model.
[0033] In any of the above implementations, when a number of kernels in the group of secondary transform kernels used for different video blocks in the encoded video stream are different, the entropy coding of the binary codeword of the kernel index to the group of secondary transform kernels having different numbers of kernels shares context modeling for at least one bin of the binary codeword.
[0034] In some other implementations, an apparatus for processing video information is disclosed. The apparatus may include circuitry configured to perform any of the implementations of the methods described above.
[0035] Aspects of the present disclosure also provide a non-transitory computer-readable medium having stored thereon instructions that, when executed by a computer, cause the computer to perform a method for decoding and / or encoding video. [Brief description of the drawings]
[0036] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1A] FIG. 1A shows a schematic diagram of an example subset of intra-prediction direction modes. [Figure 1B] FIG. 1B shows an illustrative diagram of an exemplary intra-prediction direction. [Diagram 2] FIG. 2 shows a schematic diagram of a current block and its surrounding spatial merging candidates for motion vector prediction in one example. [Diagram 3] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to one exemplary embodiment. [Figure 4]FIG. 4 illustrates a simplified block diagram of a communication system (400) according to one exemplary embodiment. [Diagram 5] FIG. 5 illustrates a simplified block diagram of a video decoder according to one exemplary embodiment. [Figure 6] FIG. 6 shows a simplified block diagram schematic of a video encoder according to one example embodiment. [Figure 7] FIG. 7 shows a block diagram of a video encoder according to another exemplary embodiment. [Figure 8] FIG. 8 shows a block diagram of a video decoder according to another exemplary embodiment. [Figure 9] FIG. 9 illustrates a coding block partitioning scheme according to an exemplary embodiment of the present disclosure. [Figure 10] FIG. 10 illustrates another coding block partitioning scheme according to an exemplary embodiment of the present disclosure. [Figure 11] FIG. 11 illustrates another coding block partitioning scheme in accordance with an exemplary embodiment of the present disclosure. [Figure 12] FIG. 12 shows an example of partitioning of base blocks into coding blocks according to one exemplary partitioning scheme. [Figure 13] FIG. 13 illustrates one exemplary ternary partitioning scheme. [Figure 14] FIG. 14 shows an exemplary quadtree binary tree coding block partitioning scheme. [Figure 15] FIG. 15 illustrates a scheme for partitioning a coding block into multiple transform blocks and the coding order of the transform blocks according to an example embodiment of this disclosure. [Figure 16] FIG. 16 illustrates a scheme for partitioning a coding block into multiple transform blocks and the coding order of the transform blocks according to an example embodiment of this disclosure. [Figure 17] FIG. 17 illustrates another scheme for partitioning a coding block into multiple transform blocks in accordance with an example embodiment of this disclosure. [Figure 18] FIG. 18 shows an example of a small angle in directional intra prediction. [Figure 19] FIG. 19 shows the nominal angle in directional intra prediction. [Figure 20] FIG. 20 shows the top, left and top-left positions for the PAETH mode of the block. [Figure 21] FIG. 21 shows an example of the recursive intra-filtering mode. [Figure 22] FIG. 22 shows an example of four-reference line intra-coding for a block. [Figure 23] FIG. 23 illustrates transform block partitioning and scanning of intra-predicted blocks according to an exemplary embodiment of this disclosure. [Figure 24] FIG. 24 illustrates transform block partitioning and scanning of inter-prediction blocks in accordance with an exemplary embodiment of this disclosure. [Diagram 25] FIG. 25 illustrates a low-frequency non-separable conversion process according to an exemplary embodiment of the present disclosure. [Figure 26] FIG. 26 illustrates a flow chart according to an exemplary embodiment of the present disclosure. [Figure 27] FIG. 27 illustrates a flow chart according to another exemplary embodiment of the present disclosure. [Figure 28] FIG. 28 illustrates a schematic diagram of a computer system in accordance with an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0037] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to one exemplary embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices capable of communicating with each other, e.g., via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) may perform a unidirectional transmission of data. For example, the terminal device (310) may code video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) can receive the coded video data from the network (350), decode the coded video data to recover the video pictures, and display the video pictures according to the recovered video data. One-way data transmission can be implemented in media provisioning applications, etc.
[0038] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) performing bidirectional transmission of coded video data, which may be implemented, for example, during a video conferencing application. For the bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) can code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) can also receive the coded video data transmitted by the other of the terminal devices (330) and (340), decode the coded video data to recover the video pictures, and display the video pictures on an accessible display device in accordance with the recovered video data.
[0039] In the example of FIG. 3, terminal devices 310, 320, 330, and 340 may be implemented as servers, personal computers, and smartphones, although the applicability of the underlying principles of the present disclosure is not so limited. Embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and / or the like. Network 350 represents any number or type of network that carries coded video data between terminal devices 310, 320, 330, and 340, including, for example, wired and / or wireless communication networks. Communication network 350 may exchange data in circuit-switched, packet-switched, and / or other types of channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network (350) may not be important to the operation of the present disclosure unless explicitly described herein.
[0040] 4 shows an arrangement of a video encoder and a video decoder in a video streaming environment as one example of an application for the disclosed technical subject matter. The disclosed technical subject matter is equally applicable to other video applications including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.
[0041] The video streaming system may include a video source (401), e.g., a video capture subsystem (413), which may include a digital camera, for generating an uncompressed video picture or stream of pictures (402). In one example, the video picture stream (402) includes samples recorded by the digital camera of the video source 401. The video picture stream (402), depicted as a thick line to emphasize its high amount of data when compared to the encoded video data (404) (or coded video bitstream), may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404), depicted as a thin line to emphasize its lower amount of data when compared to the stream of uncompressed video pictures (402)), can be stored on a streaming server (405) for future use or directly downstream to a video device (not shown). One or more streaming client subsystems, such as client subsystems (406) and (408) of FIG. 4, can then use the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may access a video server (405), for example within an electronic device (430). The video decoder (410) decodes an input copy of encoded video data (407) and generates an outgoing stream of video pictures (411) that is uncompressed and can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). The video decoder 410 may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) may be encoded according to a given video coding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC and other video coding standards.
[0042] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).
[0043] 5 shows a block diagram of a video decoder (510) according to any embodiment of the present disclosure below. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) in the example of FIG. 4.
[0044] The receiver (531) can receive one or more coded video sequences to be decoded by the video decoder (510). In the same or another embodiment, one coded video sequence can be decoded at a time, where the decoding of each coded video sequence is independent of the other coded video sequences. Each video sequence can be associated with multiple video frames or pictures. The coded video sequences can be received from a channel (501). The channel can be a hardware / software link to a storage device that stores the encoded video data or a streaming source that transmits the encoded video data. The receiver (531) can receive the encoded video data along with other data, such as coded audio data and / or ancillary data streams, which can be forwarded to their respective processing circuitry (not shown). The receiver (531) can separate the coded video sequences from the other data. To combat network jitter, a buffer memory (515) may be placed between the receiver (531) and the entropy decoder / parser (520) (hereafter, “parser (520)”). In certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, it may be external to and separate from the video decoder (510) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (510), for example, to combat network jitter, and there may be another buffer memory (515) inside the video decoder (510), for example, to handle playback timing. If the receiver (531) is receiving data from a store / forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may not be needed or may be small.For use in a best-effort packet network such as the Internet, a buffer memory (515) of sufficient size may be required and may be relatively large in size. Such a buffer memory may be implemented with an adaptive size and may be implemented, at least in part, in an operating system or similar element (not shown) external to the video decoder (510).
[0045] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (510) and potentially information for controlling a rendering device, such as a display (512) (e.g., a display screen), which may or may not be an integral part of the electronic device (530) but may be coupled to the electronic device (530) as shown in FIG. 5. The rendering device control information may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy-decode the coded video sequence received by the parser (520). The coded video sequence may be in accordance with entropy coding, a video coding technique or standard, and may be in accordance with a variety of principles, including variable length coding, Huffman coding, coding with or without context sensitivity, etc. The parser (520) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the subgroup. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (520) may also extract transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, motion vectors, etc. from the coded video sequence information.
[0046] The parser (520) can perform entropy decoding / syntax parsing operations on the video sequence received from the buffer memory (515) to generate symbols (521).
[0047] The reconstruction of the symbols (521) may involve a number of different processing or functional units, depending on the type of coded video picture or portion thereof (inter and intra picture, inter and intra block, etc.) and other factors. The units involved, and how they are involved, may be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following processing or functional units is not depicted for simplicity.
[0048] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually divided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these functional units will interact closely with each other and may be, at least partially, integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed technical subject matter, a conceptual division into functional units is adopted in the following disclosure.
[0049] The first unit may include a scalar / inverse transform unit (551), which may receive quantized transform coefficients and control information, including information indicating the type of inverse transform to use, block size, quantization coefficients / parameters, quantization scaling matrix, and location (lie) as a symbol (521) from the parser (520). The scalar / inverse transform unit (551) may output a block containing sample values, which may be input to an aggregator (555).
[0050] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may generate blocks of the same size and shape of the reconstructed block using surrounding block information already reconstructed and stored in the current picture buffer (558). The current picture buffer (558) may, for example, buffer the partially reconstructed current picture and / or the fully reconstructed current picture. In some implementations, the aggregator (555) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information as provided by the scaler / inverse transform unit (551).
[0051] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to an inter coded and potentially motion compensated block. In such a case, the motion compensated prediction unit 553 may access the reference picture memory 557 to retrieve samples used for inter picture prediction. After motion compensation of the samples fetched according to the symbols (521) related to the block, these samples may be added by the aggregator (555) to the output of the scalar / inverse transform unit (551) (the output of the unit 551 may be referred to as a residual sample or residual signal) to generate output sample information. The address in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by a motion vector available to the motion compensated prediction unit (553) in the form of a symbol (521), which may have, for example, an X, Y component (shift), and a reference picture component (time). Motion compensation may also include interpolation of sample values fetched from a reference picture memory (557) if sub-sample accurate motion vectors are used, and may also be associated with a motion vector prediction mechanism, etc.
[0052] The output samples of the aggregator (555) may be subject to various loop filtering techniques in the loop filter unit (556). Video compression techniques may include in-loop filter techniques that are controlled by parameters contained in the coded video sequence (also referred to as coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520). However, they may also include in-loop filter techniques that may be responsive to meta-information obtained during the decoding of a coded picture or previous part of the coded video sequence (in decoding order), as well as to previously reconstructed and loop filtered sample values. Several types of loop filters may be included as part of the loop filter unit 556 in various orders, as described in more detail below.
[0053] The output of the loop filter unit (556) may be a sample stream that may be output to a rendering device (512) and also stored in a reference picture memory (557) for use in future inter-picture prediction.
[0054] Once a given coded picture is fully reconstructed, it may be used as a reference picture for future inter-picture prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified (e.g., by the parser (520)) as a reference picture, the current picture buffer (558) may become part of the reference picture memory (557). The fresh current picture buffer may then be reallocated before beginning reconstruction of the next coded picture.
[0055] The video decoder (510) may perform decoding operations according to a predetermined video compression technique adopted in a standard, such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, a profile may select a given tool from all tools available in the video compression technique or standard as the only tool available for use under that profile. To comply with a standard, the complexity of the coded video sequence may be within a range defined by a level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.
[0056] In some exemplary embodiments, the receiver (531) can receive additional (redundant) data along with the coded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.
[0057] 6 shows a block diagram of a video encoder (603) according to one exemplary embodiment of the present disclosure. The video encoder (603) may be included within an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) may be used in place of the video encoder (403) in the example of FIG. 4.
[0058] The video encoder (603) can receive video samples from a video source (601) (not part of the electronic device (620) in the example of FIG. 6), which can capture video images to be encoded by the video encoder (603). In another example, the video source (601) can be implemented as part of the electronic device (620).
[0059] The video source (601) may provide a source video sequence that is coded by the video encoder (603) in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCb, RGB, XYZ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media distribution system, the video source (601) may be a storage device that can store previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image sensor information as a video sequence. The video data may be provided as a number of individual pictures or images that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, where each pixel may contain one or more samples, depending on the sampling structure, color space, etc. being used. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.
[0060] According to some example embodiments, the video encoder (603) can code and compress pictures of a source video sequence into a coded video sequence (643) in real-time or under any other time constraint required by an application. Achieving an appropriate coding rate constitutes one function of the controller (650). In some embodiments, the controller (650) can be functionally coupled to and control other functional units, as described below. This coupling is not shown for simplicity. Parameters set by the controller (650) can include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured with other appropriate functions related to the video encoder (603), optimized for a given system design.
[0061] In some example embodiments, the video encoder (603) may be configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop may include a source coder (630) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be coded and reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs symbols to generate sample data in a similar manner to the (remote) decoder, even though the embedded decoder 633 processes the coded video stream by the source coder 630 without entropy coding (because any compression between the symbols and the coded video bitstream in entropy coding may be lossless in the video compression techniques considered in the disclosed technical subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since decoding of the symbol stream produces bit-exact results independent of the decoder location (local or remote), the contents in the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the predictor of the encoder "sees" exactly the same sample values as the decoder would "see" when using the prediction during decoding as a reference picture sample. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is used to improve coding quality.
[0062] The operation of the “local” decoder (633) may be the same as a “remote” decoder, such as the video decoder (510), as already described in detail above in connection with FIG. 5. However, and also with brief reference to FIG. 5, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515), and the parser (520), may not need to be fully implemented in the local decoder (633) within the encoder.
[0063] An observation that can be made at this point is that any decoder technique, except analysis / entropy decoding, which may only be present in the decoder, may also need to be present, in substantially identical functional form, in the corresponding encoder. For this reason, the technical matters disclosed may at times focus on the decoder operations associated with the decoder portion of the encoder. Thus, a description of the encoder techniques may be omitted, since they are the inverse of the decoder techniques described generically. Only in certain areas or aspects is a more detailed description of the encoder provided below.
[0064] During operation in some example implementations, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with respect to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (632) codes differences (or residuals) in color channels between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.
[0065] The local video decoder (633) can decode the coded video data of pictures that may be designated as reference pictures based on the symbols generated by the source coder (630). The operation of the coding engine (632) can advantageously be a lossy process. If the coded video data can be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence can be a replica of the source video sequence, typically with some errors. The local video decoder (633) replicates the decoding process performed by the video decoder on the reference pictures and can cause the reconstructed reference pictures to be stored in a reference picture cache (634). In this way, the video encoder (603) can locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures acquired by a far-end (remote) video decoder.
[0066] The predictor (635) may perform a predictive search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidates for reference pixel blocks) or predefined metadata, such as reference picture motion vectors, block shapes, etc., which may serve as suitable prediction references for the new picture. The predictor (635) may operate on a sample block-by-sample block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).
[0067] The controller (650) can manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.
[0068] The output of all the above mentioned functional units may undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by lossless compression of the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.
[0069] The sender (640) may buffer the coded video sequence as it is generated by the entropy coder (645) in preparation for transmission over a communication channel (660). The communication channel may be a hardware / software link to a storage device that stores the coded video data. The sender (640) may merge the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and / or ancillary data streams (sources not shown).
[0070] The controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign a predefined coded picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:
[0071] An intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective applications and characteristics.
[0072] A predictive picture (P picture) may be one that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values of each block.
[0073] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple predicted pictures may use more than one reference picture and associated metadata for the reconstruction of a single block.
[0074] A source picture is usually spatially divided into multiple sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the respective picture of the block. For example, blocks of an I picture may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures. Source pictures or intermediate processed pictures may be subdivided into other types of blocks for other purposes. The partitioning of coding blocks and other types of blocks may or may not follow the same methodology, as described in more detail below.
[0075] The video encoder (603) may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In its operations, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. The coded video data may thus conform to a syntax specified by the video coding technique or standard being used.
[0076] In some example embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include other types of redundant data, such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.
[0077] A video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-picture prediction) exploits spatial correlation in a given picture, and inter-picture prediction exploits temporal or other correlation between pictures. For example, a particular picture being encoded / decoded, called a current picture, may be partitioned into blocks. If a block in the current picture resembles a reference block in a reference picture that was previously coded and is still buffered in the video, it may be coded by a vector called a motion vector. A motion vector points to a reference block in a reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.
[0078] In some exemplary embodiments, a bi-prediction technique may be used for inter-picture prediction. According to such a bi-prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture, both of which advance the current picture in the video in decoding order (but in display order, in the past or future, respectively). A block in the current picture may be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block may be predicted simultaneously by a combination of the first and second reference blocks.
[0079] Furthermore, to improve coding efficiency, merge mode techniques may be used in inter-picture prediction.
[0080] According to some exemplary embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed in units of blocks. For example, a picture in a sequence of video pictures is partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU may include three parallel coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels may be partitioned into one CU of 64×64 pixels, or four CUs of 32×32 pixels. One or more of the 32×32 blocks may be further partitioned into four CUs of 16×16 pixels. In some example embodiments, each CU may be analyzed during encoding to determine the prediction type of the CU among various prediction types, such as inter prediction type or intra prediction type. The CU may be divided into one or more prediction units (PUs) depending on the temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. The division of the CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. The luma or chroma PB may include a matrix of sample values (e.g., luma values), such as, for example, 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, etc.
[0081] 7 shows a diagram of a video encoder (703) according to another exemplary embodiment of the present disclosure. The video encoder (703) is configured to receive a processed block of sample values (e.g., a predictive block) in a current video picture in a sequence of video pictures and to encode the processed block into a coded picture that is part of a coded video sequence. The exemplary video encoder (703) can be used in place of the video encoder (403) in the example of FIG. 4.
[0082] For example, the video encoder (703) receives a matrix of sample values of a processing block, such as a prediction block of 8×8 samples. The video encoder (703) then determines whether the processing block is best coded using intra mode, inter mode, or bi-prediction mode, for example, using rate-distortion optimization (RDO). If it is determined that the processing block is coded in intra mode, the video encoder 703 may use intra prediction techniques to encode the processing block into a coded picture, and if it is determined that the processing block is coded in inter mode or bi-prediction mode, the video encoder 703 may use inter prediction techniques or bi-prediction techniques, respectively, to encode the processing block into a coded picture. In some exemplary embodiments, the merge mode may be used as a sub-mode of inter-picture prediction, in which motion vectors are derived from one or more motion vector predictors, without the benefit of coded motion vector components outside the predictors. In some other exemplary embodiments, there may be a motion vector component applicable to the subject block. Accordingly, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module, to determine the partition mode of a processing block.
[0083] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in the example configuration of FIG. 7.
[0084] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in earlier and later pictures in display order), generate inter-prediction information (e.g., a description of redundant information according to an inter-encoding technique, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that is decoded based on encoded video information using a decoding unit 633 embedded in the example encoder 620 of FIG. 6 (shown as a residual decoder 728 of FIG. 7, as described in more detail below).
[0085] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block with previously coded blocks in the same picture, and generate transformed and quantized coefficients, and possibly also intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). The intra encoder (722) can calculate intra prediction results (e.g., prediction blocks) based on the intra prediction information and reference blocks in the same picture.
[0086] The generic controller (721) can be configured to determine generic control data and control other components of the video encoder (703) based on the generic control data. In one example, the generic controller (721) determines a prediction mode of a block and provides a control signal to a switch (726) based on the prediction mode. For example, if the prediction mode is an intra mode, the generic controller 721 controls the switch 726 to select an intra mode result for use by the residual calculator 723 and controls the entropy encoder 725 to select intra prediction information and include the intra prediction information in the bitstream. And, if the prediction mode of the block is an inter mode, the generic controller 721 controls the switch 726 to select an inter prediction result for use by the residual calculator 723 and controls the entropy encoder 725 to select inter prediction information and include the inter prediction information in the bitstream.
[0087] The residual calculator (723) may be configured to calculate a difference (residual data) between a received block and a prediction result for a block selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder (724) may be configured to transform the residual data from a spatial domain to a frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be used by the intra-encoder (722) and the inter-encoder (730) as appropriate. For example, the inter-encoder (730) may generate decoded blocks based on the decoded residual data and the inter-prediction information, and the intra-encoder (722) may generate decoded blocks based on the decoded residual data and the intra-prediction information. The decoded blocks may be appropriately processed to generate decoded pictures, and the decoded pictures may be buffered in a memory circuit (not shown) and used as reference pictures.
[0088] The entropy encoder (725) may be configured to format a bitstream to include the encoded block and perform entropy coding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. When coding a block in a merged sub-mode of either an inter-mode or a bi-prediction mode, the residual information may not be present.
[0089] 8 shows a diagram of an exemplary video decoder (810) in accordance with another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and to decode the coded pictures to generate reconstructed pictures. In one embodiment, the video decoder (810) may be used in place of the video decoder (410) in the embodiment of FIG. 4.
[0090] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) coupled together as shown in the example configuration of FIG. 8.
[0091] The entropy decoder (871) can be configured to reconstruct from a coded picture certain symbols that represent syntax elements of which the coded picture is composed. Such symbols can include, for example, the mode in which the block is coded (e.g., intra mode, inter mode, bi-predictive mode, merged submode, or other submode), prediction information (e.g., intra prediction information or inter prediction information) that can identify certain samples or metadata used for prediction by the intra decoder (872) or the inter decoder (880), residual information, for example in the form of quantized transform coefficients, etc. As an example, if the prediction mode is an inter or bi-predictive mode, the inter prediction information is provided to the inter decoder (880), and if the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can undergo inverse quantization and is provided to the residual decoder (873).
[0092] The inter decoder (880) may be configured to receive the inter prediction information and generate inter prediction results based on the inter prediction information.
[0093] The intra decoder (872) can be configured to receive the intra prediction information and to generate a prediction result based on the intra prediction information.
[0094] The residual decoder (873) may be configured to perform inverse quantization to extract inverse quantized transform coefficients, and then process the de-quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize certain control information (including Quantizer Parameters (QP)) that may be provided by the entropy decoder (871) (a data path is not shown, as there may only be low-volume control information).
[0095] The reconstruction module (874) may be configured to combine, in the spatial domain, the residual as output by the residual decoder (873) and the prediction result (possibly as output by an inter- or intra-prediction module) to form a reconstructed block that forms part of the reconstructed picture as part of the reconstructed video. It should be noted that other suitable operations, such as deblocking operations, may also be performed to improve visual quality.
[0096] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In some exemplary embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.
[0097] Turning to block partitioning for coding and decoding, a general partitioning can start from a base block and follow a predefined set of rules, a specific pattern, a partition tree, or any partition structure or scheme. The partitioning is hierarchical and recursive. After dividing or partitioning the base block according to the partitioning procedure described below or other procedure examples, or a combination thereof, a final set of partitions or coding blocks can be obtained. Each of these partitions can be at any of various partition levels in the partition hierarchy and can be of various shapes. Each partition can be referred to as a coding block (CB). In the various partitioning implementation examples described further below, each resulting CB can be of any of the allowed sizes and partitioning levels. Such partitions are referred to as coding blocks because they are called such because they may form a unit around which some basic coding / decoding decisions are made and coding / decoding parameters are optimized, determined, and signaled in the encoded video bitstream. The top or deepest level of the final partition represents the depth of the coding block partitioning structure of the tree. The coding block may be a luma coding block or a chroma coding block. The CB tree structure for each color may be referred to as a coding block tree (CBT).
[0098] The coding blocks of all color channels may be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels may be collectively referred to as a coding tree unit (CTU). The partitioning pattern or structure for various color channels within a CTU may or may not be the same.
[0099] In some implementations, the partitioning tree scheme or structure used for the luma channel and the chroma channel may not need to be the same. Unless otherwise specified, the luma channel and the chroma channel may have separate coding tree structures or patterns. Furthermore, whether the luma channel and the chroma channel use the same or different coding partition tree structures and the coding partition tree structure actually used may depend on whether the slice being coded is a P, B, or I slice. For example, for an I slice, the chroma channel and the luma channel may have separate coding partition tree structures or coding partition tree structure modes, while for a P or B slice, the luma channel and the chroma channel may share the same coding partition tree structure. When separate coding partition tree structures or modes are applied, the luma channel may be partitioned into CBs by one coding partition tree structure, and the chroma channel may be partitioned into chroma CBs by another coding partition tree structure.
[0100] In some implementations, a predefined partitioning pattern may be applied to the base block. As shown in FIG. 9, in an example of a 4-way partition tree, starting from a first predefined level (e.g., 64×64 block level or other size as the basic block size), the base block may be partitioned hierarchically down to a predefined lowest level (e.g., 4×4 level). For example, the base block may be subject to four predefined partition options or patterns shown at 902, 904, 906, and 908, and a partition designated as R may allow recursive partitioning, where the same partition option shown in FIG. 9 may be repeated at a lower scale down to the lowest level (e.g., 4×4 level). In some implementations, additional restrictions may be applied to the partitioning scheme of FIG. 9. In the implementation of FIG. 9, rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) are allowed, but not recursively. On the other hand, square partitions may be recursive. Partitioning with recursion following FIG. 9 generates a final set of coding blocks, if necessary. The coding tree depth can be further defined to indicate the depth of the division from the root node or root block. For example, the coding tree depth can be set to 0 for a root node or root block of 64×64 blocks, and the coding tree depth can be increased by 1 after one further division of the root block following FIG. 9. The maximum or deepest level from the 64×64 base block to the smallest partition of 4×4 is 4 (starting from level 0) in the above scheme. Such a partitioning scheme can be applied to one or more color channels. Each color channel can be partitioned independently according to the scheme of FIG. 9 (e.g., the division pattern or option between the predefined patterns can be determined independently for each color channel at each hierarchical level).Alternatively, two or more color channels may share the same hierarchical pattern tree of FIG. 9 (e.g., the same splitting pattern or option may be selected from among predefined patterns for two or more color channels at each hierarchical level).
[0101] FIG. 10 illustrates another example predefined partitioning pattern that enables recursive partitioning to form a partitioning tree. As illustrated in FIG. 10, ten exemplary partitioning structures or patterns may be predefined. A root block may start at a predefined level (e.g., a 128×128 level, or a base block at a 64×64 level). The exemplary partitioning structure of FIG. 10 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. Note that the partition type with three subpartitions illustrated in the second row of FIG. 10 may be referred to as a “T-type” partition. The “T-type” partitions 1002, 1004, 1006, and 1008 may be referred to as Left T-Type, Top T-Type, Right T-Type, and Bottom T-Type. In some exemplary implementations, none of the rectangular partitions of FIG. 10 are allowed to be further subdivided. A coding tree depth may be further defined to indicate the partition depth from a root node or root block. For example, for a root node or root block, e.g., a 128x128 block, the coding tree depth may be set to 0, and after the root block is further partitioned following FIG. 10, the coding tree depth is increased by 1. In some implementations, only all-square partitions of 1010 may be allowed for recursive partitioning to the next level of the partitioning tree according to the pattern of FIG. 10. In other words, for square partitions having T-shaped patterns 1002, 1004, 1006, and 1006, recursive partitioning may not be allowed. The partitioning procedure following FIG. 10 with recursion generates a final set of coding blocks, if necessary. Such a scheme may be applied to one or more color channels. In some implementations, more flexibility may be added to the use of partitions below the 8x8 level. For example, 2x2 chroma inter prediction may be used in certain cases.
[0102] In some other implementations of coding block partitioning, a quadtree structure may be used to split a base block or an intermediate block into quadtree partitions. Such quadtree splitting may be applied hierarchically and recursively to any square partition. Whether the base block, or the intermediate block, or the partition is further quadtree splitting may be adapted to various local characteristics of the base block or the intermediate block / partition. The quadtree partitioning at the picture boundary may be further adapted. For example, an implicit quadtree splitting may be performed at the picture boundary, so that the block keeps the quadtree splitting until its size fits the picture boundary.
[0103] In some other implementations, hierarchical binary partitioning from the base block may be used. For such schemes, the base block or mid-level block may be partitioned into two partitions. The binary partitioning may be either horizontal or vertical. For example, horizontal binary partitioning may divide the base block or mid-block into equal left and right partitions. Similarly, vertical binary partitioning may divide the base block or mid-block into equal top and bottom partitions. Such binary partitioning may be hierarchical and recursive. At each of the base block or mid-block, a decision may be made whether to continue the binary partitioning scheme and, if the scheme continues further, whether to use horizontal or vertical binary partitioning. In some implementations, further partitioning may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, further partitioning may stop once a predefined division level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be limited. For example, the aspect ratio of the partitions cannot be smaller than 1:4 (or larger than 4:1). Thus, the vertical strip partition, with a vertical to horizontal aspect ratio of 4:1, is simply further binary partitioned vertically into an upper partition and a lower partition, each with a vertical to horizontal aspect ratio of 2:1.
[0104] In yet some other examples, a ternary partitioning scheme can be used to split the base block or any intermediate blocks, as shown in FIG. 13. The ternary pattern can be implemented vertically, as shown in 1302 of FIG. 13, or horizontally, as shown in 1304 of FIG. 13. The exemplary split ratio in FIG. 13 is shown as 1:2:1, either vertically or horizontally, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a ternary partitioning scheme can be used to complement a quadtree or binary partitioning structure. Such a ternary partitioning can capture objects located at block centers in one contiguous partition, whereas quadtrees and binaries always split along block centers, thus splitting objects into separate partitions. In some implementations, the width and height of the exemplary triple tree partitions are always powers of two to avoid additional transformations.
[0105] The above partitioning schemes can be combined in any way at different partitioning levels. As an example, the above quadtree and binary partitioning schemes can be combined to partition the base block into a quadtree-binary (QTBT) structure. In such a scheme, the base block or intermediate blocks / partitions can be either quadtree partitioned or binary, according to a set of predefined conditions, if specified. A specific example is shown in FIG. 14. In the example of FIG. 14, the base block is a first quadtree partitioned into four partitions, as shown at 1402, 1404, 1406, and 1408. From then on, each of the resulting partitions is either quadtree partitioned into four further partitions (as in 1408), binary partitioned into two further partitions at the next level (e.g., both symmetric, either horizontally or vertically, as in 1402 or 1406), or not partitioned (as in 1404). Binary or quadtree partitioning may be allowed recursively for square partitions, as shown in the general example partitioning pattern in 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree partitioning and dashed lines represent binary partitioning. A flag can be used for each binary partition node (non-leaf binary partition) to indicate whether the binary partitioning is horizontal or vertical. For example, as shown in 1420, flag "0" can represent horizontal binary partitioning and flag "1" can represent vertical binary partitioning, consistent with the partitioning structure in 1410. In the case of quadtree partitioning, there is no need to indicate the type of partitioning, since quadtree partitioning always splits a block or partition horizontally and vertically to generate four sub-blocks / partitions of equal size. In some implementations, flag "1" can represent horizontal binary partitioning and flag "0" can represent vertical binary partitioning.
[0106] In some implementations of QTBT, the rule sets for quadtree and binary splitting are represented by the following predefined parameters and their associated corresponding functions: obtain. -CTU size: size of the root node of the quadtree (size of the base block) -MinQTSize: The minimum allowed size of a quadtree leaf node. -MaxBTSize: Maximum allowed size of a binary tree root node -MaxBTDepth: The maximum allowed depth of a binary tree. -MinBTSize: The minimum allowed size of a binary tree leaf node. In some implementation examples of the QTBT partitioning structure, the CTU size may be set as 128×128 luma samples with corresponding two 64×64 blocks of chroma samples (when used considering the chroma subsampling example), MinQTSize may be set as 16×16, MaxBTSize may be set as 64×64, MinBTSize (both width and height) may be set as 4×4, and MaxBTDepth may be set as 4. First, quad-tree partitioning may be applied to the CTU to generate quad-tree leaf nodes. The quad-tree leaf nodes may have a size from the minimum allowed size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the node is 128×128, it will not be split by the binary tree first because its size exceeds MaxBTSize (i.e., 64×64). Otherwise, the node that does not exceed MaxBTSize may be split by the binary tree. In the example of FIG. 14, the base block is 128×128. A basic block can only be quadtree partitioned according to a predefined set of rules. The basic block has a partitioning depth of 0. Each of the resulting 4 partitions is 64x64, does not exceed MaxBTSize, and can be further partitioned into quadtrees or binary trees at level 1. The process continues. Once the depth of the binary tree reaches MaxBTDepth (i.e., 4), no further partitions are considered. Once a binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal partitions are considered. Similarly, once a binary tree node has a height equal to MinBTSize, no further vertical partitions are considered.
[0107] In some implementations, the above QTBT schemes may be configured to support the flexibility of luma and chroma having the same or separate QTBT structures. For example, for P and B slices, the luma and chroma CTBs in one CTU may share the same QTBT structure. However, for an I slice, the luma CTB may be partitioned into CBs by a QTBT structure, and the chroma CTB may be partitioned into chroma CTBs by another QTBT structure. This means that a CU may be used to refer to different color channels in an I slice. For example, an I slice may consist of a coding block of a luma component or a coding block of two chroma components, and a CU in a P or B slice may consist of coding blocks of all three color components.
[0108] In some other implementations, the QTBT scheme may be supplemented with the ternary scheme mentioned above. Such implementations may be referred to as multi-type tree (MTT) structures. For example, in addition to the binary split of nodes, one of the ternary split patterns in Figure 13 may be selected. In some implementations, only square nodes may be subject to ternary split. An additional flag may be used to indicate whether the ternary split is horizontal or vertical.
[0109] The design of two-level or multi-level trees, such as the QTBT implementation and the QTBT implementation complemented by ternary splitting, can be motivated primarily by reduced complexity. In theory, the complexity of traversing a tree is T D where T denotes the number of split types and D is the depth of the tree. Tradeoffs can be made by using multiple types (T) while reducing the depth (D).
[0110] In some implementations, the CB may be further partitioned. For example, the CB may be further partitioned into multiple prediction blocks (PBs) for the purpose of intra-frame or inter-frame prediction during the coding and decoding process. In other words, the CB may be further partitioned into different sub-partitions, where individual prediction decisions / configurations may be made. In parallel, the CB may be further partitioned into multiple transform blocks for the purpose of expressing the level at which the transform or inverse transform of the video data is performed. The partitioning scheme of the CB into PBs and TBs may be the same or may not be. As an example, each partitioning scheme may be performed using its own procedure, for example, based on various characteristics of the video data. The partitioning schemes of the PBs and TBs may be independent in some exemplary implementations. The partitioning schemes and boundaries of the PBs and TBs may be correlated in some other exemplary implementations. In some implementations, for example, the TBs may be partitioned after the PB partitioning, and in particular, each PB may be further partitioned into one or more TBs after the next partition of the coding block is determined. For example, in some implementations, the PB may be divided into 1, 2, 4, or some other number of TBs.
[0111] In some implementations, the luma and chroma channels may be processed differently to partition the base blocks into coding blocks and further into predictive and / or transform blocks. For example, in some implementations, partitioning of coding blocks into predictive and / or transform blocks may be allowed for the luma channel, while partitioning of such coding blocks into predictive and / or transform blocks may not be allowed for the chroma channels. In such implementations, transform and / or prediction of luma blocks may therefore be performed only at the coding block level. In another example, the minimum transform block size of the luma and chroma channels may be different. For example, the coding blocks of the luma channel may be allowed to be partitioned into smaller transform and / or predictive blocks than the chroma channels. In yet another example, the maximum depth of partitioning of coding blocks into transform and / or predictive blocks may be different between the luma and chroma channels. For example, the coding blocks of the luma channel may be allowed to be partitioned into deeper transform and / or predictive blocks than the chroma channels. In a particular example, a luma coding block may be partitioned into transform blocks of multiple sizes, which may be represented by recursive partitions down to a maximum of two levels, and transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, and transform block sizes from 4×4 to 64×64 may be allowed. However, for chroma blocks, only the largest possible transform block specified for the luma block may be allowed.
[0112] In some example implementations for partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partitioning may depend on whether the PB is intra-coded or inter-coded.
[0113] Partitioning of coding blocks (or predictive blocks) into transform blocks may be implemented in various exemplary schemes, including, but not limited to, quadtree partitioning and predefined pattern partitioning, recursively or non-recursively, and with additional consideration of transform blocks at coding or predictive block boundaries. In general, the resulting transform blocks may be at different partition levels, may not be the same size, and may not need to be square in shape (e.g., they may be rectangular with an allowed size and aspect ratio). Further examples are described in more detail below in connection with Figures 15, 16, and 17.
[0114] However, in some other implementations, the CB obtained through any of the above partitioning schemes can be used as a basic or smallest coding block for prediction and / or transformation. In other words, no further partitioning is performed for inter / intra prediction and / or transformation purposes. For example, the CB obtained from the above QTBT scheme can be directly used as a unit for performing prediction. Specifically, such a QTBT structure removes the concept of multiple partition types, i.e., removes the separation of CU, PU, and TU, and supports more flexibility for CU / CB partition shapes as described above. In such a QTBT block structure, the CU / CB can have either a square or rectangular shape. The leaf nodes of such a QTBT are used as units of prediction and transformation processing without further partitioning. This means that in such an exemplary QTBT coding block structure, the block sizes of CU, PU, and TU are the same.
[0115] The various CB partitioning schemes described above, and further partitioning of the CB into PB and / or TB (including no PB / TB partitioning), can be combined in any manner. The following specific implementations are provided as non-limiting examples.
[0116] Specific exemplary implementations of partitioning of coding blocks and transform blocks are described below. In such exemplary implementations, a base block may be partitioned into coding blocks using the recursive quadtree partitioning described above, or a predefined partitioning pattern (such as those in FIG. 9 and FIG. 10). At each level, whether to continue further quadtree partitioning of a particular partition may be determined by local video data characteristics. The resulting CBs may be of different sizes and at different quadtree partitioning levels. The decision of whether to code a picture region using inter-picture (temporal) or intra-picture (spatial) prediction may be made at the CB level (or at the CU level, for all three color channels). Each CB may be further partitioned into one, two, four, or other number of PBs according to a predefined PB partitioning type. Within one PB, the same prediction process may be applied, and related information may be transmitted to the decoder on a PB basis. After obtaining the residual block by applying the prediction process based on the partitioning type of the PB, the CB may be partitioned into TBs according to another quadtree structure similar to the coding tree of the CB. In this particular implementation, the CB or TB may be, but need not necessarily be, limited to a square. Furthermore, in this particular implementation, the PB may be square or rectangular in shape for inter prediction, and may only be square for intra prediction. A coding block may be divided, for example, into four square TBs. Each TB may be further divided recursively (using quad-tree partitioning) into smaller TBs, referred to as Residual Quad-Tree (RQT).
[0117] Another exemplary implementation for partitioning a base block into CB and other PBs and / or TBs is further described below. For example, rather than using multiple partition unit types such as those shown in FIG. 9 or FIG. 10, a quadtree with nested multi-type trees using binary and ternary partition segmentation structures (e.g., QTBT or QTBT with ternary partition as described above) can be used. The separation of the concepts of CB, PB, and TB (i.e., partitioning CB into PB and / or TB, and partitioning PB into TB) may be abandoned unless required for CBs with sizes that are too large for the maximum transform length, which may require further partitioning. This exemplary partitioning scheme may be designed to support more flexibility for the partitioning shape of CBs, such that both prediction and transformation can be performed at the CB level without further partitioning. In such a coding tree structure, the CB may have either a square or rectangular shape. Specifically, the coding tree block (CTB) is first partitioned by a quadtree structure. The quadtree leaf nodes can then be further partitioned by nested multitype tree structures. One example of a nested multitype tree structure using binary or ternary splitting is shown in FIG. 11. Specifically, the exemplary multitype tree structure of FIG. 11 includes four split types, referred to as vertical binary split (SPLIT_BT_VER) (1102), horizontal binary split (SPLIT_BT_HOR) (1104), vertical ternary split (SPLIT_TT_VER) (1106), and horizontal ternary split (SPLIT_TT_HOR) (1108). CB then corresponds to the leaf of the multitype tree. In this exemplary implementation, if CB is not too large for the maximum transform length, this segmentation is used for both prediction and transform processing without any further partitioning.This means that in most cases, CB, PB, and TB have the same block size in a quadtree with nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is smaller than the width or height of the color components of CB. In some implementations, in addition to the binary or ternary split, the nested pattern of FIG. 11 may further include a quadtree split.
[0118] One particular example of a quadtree with nested multi-type tree coding block structure of block partitions for one CTB is shown in FIG. 12 (including options of quadtree partitioning, binary partitioning, or ternary partitioning). More specifically, FIG. 12 shows a base block 1200 being quadtree partitioned into four square partitions 1202, 1204, 1206, and 1208. A decision is made for each quadtree partition partition to further use the multi-type tree structure and quadtree of FIG. 11 for further partitioning. In the example of FIG. 12, partition 1204 is not further partitioned. Partitions 1202 and 1208 each employ a different quadtree partitioning. For partition 1202, the top-left, top-right, bottom-left, and bottom-right partitions of the second level quadtree partition employ a third level partition that is a quadtree, horizontal binary partition 1104 of FIG. 11, no partition, and horizontal ternary partition 1108 of FIG. 11, respectively. Partition 1208 employs another quadtree partition, and the top-left, top-right, bottom-left, and bottom-right partitions of the second level quadtree partition employ a third level partition that is a vertical ternary partition 1106 of FIG. 11, no partition, and horizontal binary partition 1104 of FIG. 11, respectively. The two subpartitions of the third level top-left partition of 1208 are further partitioned according to the horizontal binary partition 1104 and horizontal ternary partition 1108 of FIG. 11. Partition 1206 adopts a second level splitting pattern following vertical binary split 1102 of Figure 11 into two partitions, and is further split at a third level according to horizontal ternary split 1108 and vertical binary split 1102 of Figure 11. A fourth level split is further applied to one of them according to horizontal binary split 1104 of Figure 11.
[0119] In the particular example above, the maximum luma transform size may be 64×64, and the maximum supported chroma transform size may differ from the luma, e.g., 32×32. Although the CB in the above example of Figure 12 is generally not further divided into smaller PBs and / or TBs, if the width or height of the luma coding block or chroma coding block is larger than the maximum transform width or height, the luma coding block or chroma coding block is automatically divided in the horizontal and / or vertical directions to meet the transform size limitation in that direction.
[0120] In the specific example of partitioning the base blocks into the above CBs, and as described above, the coding tree scheme can support the ability for luma and chroma to have separate block tree structures. For example, for P and B slices, the luma CTB and chroma CTB in one CTU can share the same coding tree structure. For example, for an I slice, luma and chroma may have separate coding block tree structures. When separate block tree structures are applied, the luma CTB is partitioned into luma CBs by one coding tree structure, and the chroma CTB is partitioned into chroma CBs by another coding tree structure. This means that a CU in an I slice may consist of a coding block of a luma component or a coding block of two chroma components, and a CU in a P or B slice is always composed of coding blocks of all three color components unless the video is monochrome.
[0121] If a coding block is further divided into multiple transform blocks, the transform blocks therein may be ordered in the bitstream according to various orders or scanning methods. Exemplary implementations for dividing a coding block or a prediction block into transform blocks and the coding order of the transform blocks are described in further detail below. In some exemplary implementations, as described above, the transform partitioning can support multiple shapes of transform blocks, e.g., 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transform block sizes ranging from, e.g., 4×4 to 64×64. In some implementations, when the coding block is 64×64 or smaller, the transform block partitioning is only applicable to the luma component, and for chroma blocks, the transform block size is the same as the size of the coding block. Otherwise, if the width or height of the coding block is greater than 64, both the luma and chroma coding blocks may be implicitly partitioned into multiple min(W,64)×min(H,64) and min(W,32)×min(H,32) transform blocks, respectively.
[0122] In some example implementations of transform block partitioning, for both intra- and inter-coding blocks, the coding block may be further partitioned into multiple transform blocks with a partitioning depth up to a predefined number of levels (e.g., two levels). The depth and size of the transform block partitioning may be related. For some example implementations, the mapping from the transform size of current depth to the transform size of next depth is shown in Table 1. [Table 1] Table 1: Conversion Partition Size Settings
[0123] Based on the example mapping of Table 1, for a 1:1 square block, the next level transform partition may generate four 1:1 square sub-transform blocks. The transform partition may stop at, for example, 4×4. Thus, the transform size at the current depth of 4×4 corresponds to the same size at the next depth of 4×4. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform partition may generate two 1:1 square sub-transform blocks, while for a 1:4 / 4:1 non-square block, the next level transform partition may generate two 1:2 / 2:1 sub-transform blocks.
[0124] In some example implementations, for the luma components of intra-coded blocks, additional restrictions may be applied regarding the partitioning of transform blocks. For example, for each level of transform partitioning, all sub-transform blocks may be restricted to have equal size. For example, for a 32×16 coding block, the level 1 transform partitioning creates two 16×16 sub-transform blocks, and the level 2 transform partitioning creates eight 8×8 sub-transform blocks. In other words, the second level partitioning must be applied to all first level sub-blocks to keep the transform units equal in size. One example of transform block partitioning for an intra-coded square block following Table 1 is shown in FIG. 15 with the coding order indicated by the arrows. Specifically, 1502 shows a square coding block. The first level partitioning according to Table 1 into four equal-sized transform blocks is shown in 1504 with the coding order indicated by the arrows. The second level partitioning according to Table 1 into 16 equal-sized transform blocks is shown in 1506 with the coding order indicated by the arrows.
[0125] In some example implementations, the above restrictions on intra-coding may not apply for the luma components of an inter-coded block. For example, after the first level of transform partitioning, any one of the sub-transform blocks may be further partitioned independently at another level. The resulting transform blocks may or may not therefore be of the same size. An example partitioning of an inter-coded block into transform blocks using a coding order is shown in FIG. 16. In the example of FIG. 16, an inter-coded block 1602 is partitioned into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is partitioned into four transform blocks of equal size. Then, only one (but not all) of the four transform blocks is further partitioned into four sub-transform blocks, resulting in a total of seven transform blocks having two different sizes, as shown at 1604. An example coding order of these seven transform blocks is indicated by arrows at 1604 in FIG. 16.
[0126] In some example implementations, for chroma components, some additional restrictions on transform blocks may be applied: for example, for chroma components, the transform block size may be the same size as the coding block size, but not smaller than a predefined size, e.g., 8×8.
[0127] In some other example implementations, for coding blocks with either width (W) or height (H) greater than 64, both luma and chroma coding blocks may be implicitly partitioned into multiples of min(W,64)×min(H,64) and min(W,32)×min(H,32) transform units, respectively, where in this disclosure, “min(a,b)” may return the smaller value between a and b.
[0128] Figure 17 further illustrates an alternative exemplary scheme for partitioning a coding block or a prediction block into transform blocks. As shown in Figure 17, instead of using recursive transform partitioning, a predefined set of partitioning types may be applied to the coding block according to the transform type of the coding block. In the particular example shown in Figure 17, one of six exemplary partitioning types may be applied to partition the coding block into various numbers of transform blocks. Such a scheme for generating a partitioning of a transform block may be applied to either the coding block or the prediction block.
[0129] More specifically, the partitioning scheme of FIG. 17 provides up to six exemplary partitioning types for any given transform type (transform type refers to the type of primary transform, e.g., ADST, etc.) as shown in FIG. 15. In this scheme, every coding block or predictive block may be assigned a transform partitioning type, e.g., based on a rate-distortion cost. In one example, the transform partitioning type assigned to a coding block or predictive block may be determined based on the transform type of the coding block or predictive block. A particular transform partitioning type may correspond to a transform block partition size and pattern (or partitioning type), as shown by the six transform partitioning types shown in FIG. 17. The correspondence between various transform types and various transform partitioning types may be predefined. One exemplary correspondence is shown below, with capitalized labels indicating transform partitioning types that may be assigned to a coding block or predictive block based on a rate-distortion cost. ·PARTITION_NONE: Allocate the transformation size equal to the block size. ·PARTITION_SPLIT: Allocates a transformation size that is 1 / 2 the width of the block size and 1 / 2 the height of the block size. ·PARTITION_HORZ: Allocates a transformation size that is the same width as the block size and half the height of the block size. ·PARTITION_VERT: Allocates a transformation size that is half the width of the block size and the same height as the block size. ·PARTITION_HORZ4: Allocates a transformation size that is the same width as the block size and 1 / 4 of the height of the block size. ·PARTITION_VERT4: Allocates a transformation size that is 1 / 4 of the width of the block size and the same height as the block size.
[0130] In the above example, all transform partitioning types as shown in Figure 17 include uniform transform sizes for the partitioned transform blocks. This is merely an example and not a limitation. In some other implementations, mixed transform block sizes may be used for transform blocks partitioned with a particular partitioning type (or pattern).
[0131] A video block (PB or CB, also referred to as PB when not further divided into multiple predictive blocks) may be predicted in various ways rather than being directly encoded, thereby exploiting various correlations and redundancies in the video data to improve compression efficiency. Accordingly, such prediction may be performed in various modes. For example, a video block may be predicted via intra-prediction or inter-prediction. In particular, in an inter-prediction mode, a video block may be predicted via either single-reference or compound-reference inter-prediction, by one or more other reference blocks or an inter-prediction block from one or more other frames. For an inter-prediction implementation, a reference block may be specified by its frame identifier (the temporal location of the reference block) and a motion vector indicating a spatial offset between a current block being encoded or decoded and the reference block (the spatial location of the reference block). The reference frame identification and the motion vector may be signaled in the bitstream. The motion vector as a spatial block offset may be signaled directly or may itself be predicted by another reference motion vector or a predictive motion vector. For example, the current motion vector may be predicted directly by a reference motion vector (e.g., of a candidate neighboring block), or may be predicted by a combination of the reference motion vector and the motion vector difference (MVD) between the current motion vector and the reference motion vector. The latter may be referred to as merge mode with motion vector difference (MMVD). The reference motion vector is identified in the bitstream, for example, as a pointer to a spatially neighboring block or a temporally neighboring but spatially collocated current block.
[0132] Returning to the intra prediction process, therein, samples within a block (e.g., luma or chroma prediction block, or coding block if not further divided into prediction blocks) are predicted by samples from neighbors, next neighbors, or other lines, or a combination thereof, to generate a prediction block. The residual between the actual block being coded and the prediction block may then be processed by a transform followed by quantization. Various intra prediction modes may be available, and parameters related to intra mode selection and other parameters may be signaled in the bitstream. The various intra prediction modes may relate, for example, to the line position for predicting the sample, the direction along which the prediction sample is selected from the line to predict, and other special intra prediction modes.
[0133] For example, the set of intra prediction modes (also interchangeably referred to as "intra modes") may include a predefined number of directional intra prediction modes. As discussed above in connection with the implementation of FIG. 1, these intra prediction modes may correspond to a predefined number of directions in which an out-of-block sample is selected as a prediction of a sample predicted within a particular block. In another particular implementation, eight main direction modes, corresponding to angles from 45 degrees to 207 degrees with respect to the horizontal axis, may be supported and predefined.
[0134] In some other implementations of intra prediction, the directional intra modes may be further extended to angles set with finer granularity to further exploit the more diverse spatial redundancy in the directional texture. For example, the above eight-angle implementation may be configured to provide eight nominal angles, designated V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, as shown in FIG. 19, and a predefined number (e.g., 7) of finer angles may be added to each nominal angle. With such extension, a larger total number of directional angles (e.g., 56 in this example) may be available for intra prediction, corresponding to the same number of predefined directional intra modes. The prediction angle may be expressed as the nominal intra angle plus an angle delta. For the particular example above with 7 finer angle directions for each nominal angle, the angle delta can be a step size of 3 degrees multiplied by -3 to 3. Several angle schemes can be used, as shown in Figure 18 with 65 different predicted angles.
[0135] In some implementations, as an alternative or in addition to the above directional intra modes, a predefined number of non-directional intra prediction modes may also be predefined and made available. For example, five non-directional intra modes, referred to as smooth intra prediction modes, may be specified. These non-directional intra mode prediction modes may be specifically referred to as DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H intra modes. Prediction of samples of a particular block under these example non-directional modes is illustrated in FIG. 20. As an example, FIG. 20 illustrates a 4×4 block 2002 predicted by samples from an upper neighboring line and / or a left neighboring line. A particular sample 2010 of the block 2002 may correspond to a sample 2004 directly above the sample 2010 of the upper neighboring line of the block 2002, a sample 2006 above and to the left of the sample 2010 as the intersection of the upper neighboring line and the left neighboring line, and a sample 2008 directly to the left of the sample 2010 of the left neighboring line of the block 2002. In an example of a DC intra prediction mode, the average of the left and top neighboring samples 2008 and 2004 may be used as a predictor of sample 2010. In an example of a PAETH intra prediction mode, the top, left, and top left reference samples 2004, 2008, and 2006 may be fetched, and then the closest value of these three reference samples to (top+left-top left) may be set as the predictor of sample 2010. In an example of a SMOOTH_V intra prediction mode, sample 2010 may be predicted by a quadratic vertical interpolation of the top left neighboring sample 2000 and the left neighboring sample 2008. In an example of a SMOOTH_H intra prediction mode, sample 2010 may be predicted by a quadratic horizontal interpolation of the top left neighboring sample 2006 and the top neighboring sample 2004. In an example of a SMOOTH intra prediction mode, sample 2010 may be predicted by an average of the quadratic vertical and horizontal interpolations. The above non-directional intra mode implementation is given merely as a non-limiting example.Other adjacent lines, and other non-directional selections of samples, and methods of combining prediction samples to predict a particular sample within a prediction block are also contemplated.
[0136] The selection of a particular intra-prediction mode by the encoder from the above directional or non-directional modes at various coding levels (picture, slice, block, unit, etc.) may be signaled in the bitstream. In some implementations, the exemplary eight nominal directional modes may be signaled first, along with five non-angular smooth modes (a total of 13 options). Then, if the signal mode is one of the eight nominal angle intra modes, an index is further signaled to indicate the selected angle delta to the corresponding signaled nominal angle. In some other implementations, all intra-prediction modes may be indexed together for signaling (e.g., 56 directional modes plus five non-directional modes to generate 61 intra-prediction modes).
[0137] In some implementations, the exemplary 56 or other number of directional intra-prediction modes may be implemented with a unified directional predictor that projects each sample of a block to the location of a reference subsample and then interpolates the reference samples with a 2-tap bilinear filter.
[0138] In some implementations, additional filter modes, called FILTER INTRA modes, may be designed to capture the decaying spatial correlation with the reference on the edge. In these modes, prediction samples within a block may be used in addition to out-of-block samples as intra prediction reference samples for some patches within the block. These modes may, for example, be predefined and made available for intra prediction of at least the luma block (or only the luma block). A predefined number (e.g., 5) of filter intra modes may be predesigned, each of which may be represented by a set of n-tap filters (e.g., 7-tap filters) reflecting, for example, the correlation between a sample in a 4×2 patch and its n neighboring samples. In other words, the weighting coefficients of the n-tap filters may depend on the position. Taking an 8×8 block, a 4×2 patch, and 7-tap filtering as an example, the 8×8 block 2002 may be divided into eight 4×2 patches, as shown in FIG. 21. These patches are denoted as B0, B1, B2, B3, B4, B5, B6, and B7 in FIG. 21. For each patch, seven neighbors, denoted as R0 to R7 in FIG. 21, can be used to predict samples in the current patch. For patch B0, all neighbors may already be reconstructed. However, for other patches, some neighbors may be in the current block and therefore not reconstructed, so the predicted values of the nearest neighbors are then used as reference. For example, all neighbors of patch B7 shown in FIG. 21 are not reconstructed, so the predicted samples of the neighbors are used instead.
[0139] In some implementations of intra prediction, one color component may be predicted using one or more other color components. A color component may be any one of the components in YCrCb, RGB, XYZ color space, etc. For example, prediction of a chroma component (e.g., a chroma block) from a luma component (e.g., a luma reference sample), referred to as luma-to-chroma or CfL, may be implemented. In some implementations, most cross-color predictions are only allowed from luma to chroma. For example, a chroma sample in a chroma block may be modeled as a linear function of the coincident reconstructed luma sample. CfL prediction may be implemented as follows:
number
[0140] Here, L AC where α denotes the AC contribution of the luma component, α denotes a parameter of the linear model, and DC denotes the DC contribution of the chroma component. The AC components are obtained, for example, for each sample of the block, while the DC components are obtained for the entire block. Specifically, the reconstructed luma samples may be subsampled to the chroma resolution, and then the average luma value (the luma DC) may be subtracted from each luma value to form the luma AC contribution. The luma AC contribution may then predict the chroma component AC value, which is used in the linear mode of equation (1). To approximate or predict the chroma AC components from the luma AC contribution, instead of requiring the decoder to calculate the scaling parameters, in one example of a CfL implementation, the parameters α may be determined based on the original chroma samples and signaled in the bitstream. This reduces the decoder complexity and provides a more accurate prediction. As for the DC contribution of the chroma components, in some implementations, it may be calculated using an intra DC mode within the chroma components.
[0141] In some implementations of reference lines in intra prediction, multi-line intra prediction may be used. In these implementations, one or more reference lines are available for selection in intra prediction, and the encoder determines and signals which reference line is used to generate the intra prediction. The reference line index may be signaled before the intra prediction mode, and if a non-zero reference line index is signaled, only the most probable prediction mode is allowed. Referring to FIG. 22, an example of four reference lines is shown (reference line 0 to reference line 3), where each reference line is composed of six segments, i.e., segments A to F (denoted 2202-2212), with a top-left reference sample. Furthermore, segments A and F may be padded with the nearest samples from segments B and E, respectively. Multi-reference line intra prediction may be used with directional and non-directional intra prediction.
[0142] A transform of the residual of either the intra-predicted block or the inter-predicted block is then implemented, followed by quantization of the transform coefficients. For the purpose of performing the transform, both intra-coded blocks and inter-coded blocks can be further divided into multiple transform blocks (the term "unit" is typically used to denote a collection of three color channels, but may sometimes be used interchangeably as "transform unit"; for example, a "coding unit" would include a luma coding block and a chroma coding block) before the transform. In some implementations, a maximum division depth of a coded block (or predictive block) can be specified (the term "coded block" may be used interchangeably with "coding block"). For example, such division cannot exceed two levels. The division of a predictive block into transform blocks may be handled differently between intra-predicted blocks and inter-predicted blocks. In some implementations, however, such division may be similar between intra-predicted blocks and inter-predicted blocks.
[0143] In some implementations, and for intra-coded blocks, the transform partitions can be done in a way that all transform blocks have the same size and the transform blocks are coded in raster scan order. An example of such a transform block partitioning of an intra-coded block is shown in Figure 23. Specifically, Figure 23 shows that a coding block 2302 is partitioned into 16 transform blocks of the same block size via a mid-level quadtree partitioning 2304, as shown at 2306. An example raster scan order for coding is shown by the ordered arrows in Figure 23.
[0144] In some implementations, and for inter-coded blocks, the partitioning of transform units may be done in a recursive manner with a partitioning depth up to a predefined number of levels (e.g., two levels). The partitioning can stop or continue recursively at any subpartition and at any level, as shown in FIG. 23. In particular, FIG. 24 shows an example where block 2402 is partitioned into four quadtree subblocks 2404, and one of the subblocks is further partitioned into four second-level transform blocks, while the partitioning of the other subblock stops after the first level, generating a total of seven transform blocks of two different sizes. An example of a raster scan order for coding is further illustrated by the ordered arrows in FIG. 24. While FIG. 24 shows an implementation of a quadtree partitioning of square transform blocks up to two levels, in some generation implementations, the transform partitioning may be a 1:1 partitioning ranging from 4×4 to 64×64. (square), 1:2 / 2:1, and 1:4 / 4:1 transform block shapes and sizes can be supported. In some implementations, if the coding block is 64x64 or smaller, the transform block partitioning may be applied to the luma component only (in other words, the chroma transform block will be the same as the coding block under that condition). Otherwise, if the width or height of the coding block is greater than 64, both the luma coding block and the chroma coding block may be implicitly partitioned into multiples of min(W,64)xmin(H,64) and min(W,32)xmin(H,32), respectively.
[0145] Each of the above transform blocks may then be subject to a linear transform, which essentially moves the residuals in the transform block from the spatial domain to the frequency domain. In some implementations of the actual linear transform, multiple transform sizes (ranging from 4 points to 64 points for each of the two dimensions) and transform shapes (square, rectangular with aspect ratios of 2:1 / 1:2, and 4:1 / 1:4) may be allowed to support the partitions of the above exemplary enhanced coding blocks.
[0146] Turning to the actual linear transform, in some implementations, the two-dimensional (2-D) transform process may include the use of a hybrid transform kernel (e.g., may be composed of a different 1-D transform for each dimension of the coded residual transform block). Examples of one-dimensional (1-D) transform kernels may include, but are not limited to, a) 4-point, 8-point, 16-point, 32-point, 64-point DCT-2, b) 4-point, 8-point, 16-point asymmetric DST (DST-4, DST-7) and their flipped versions, and c) 4-point, 8-point, 16-point, 32-point identity transform. The selection of the transform kernel used for each dimension may be based on a rate-distortion (RD) criterion. For example, the basis functions of the DCT-2 and asymmetric DST that may be implemented are listed in Table 2. [Table 2] Table 2: Exemplary linear transform basis functions (DCT-2, DST-4, DST-7 for N-point input)
[0147] In some implementations, the availability of hybrid transform kernels for a particular primary transform implementation may be based on transform block size and prediction mode. An example of the dependency is shown in Table 3. The table shows example AV1 hybrid transform kernels and their availability based on prediction mode and block size, where → and ↓ represent horizontal and vertical dimensions. The "R" and "X" (check mark) indicate the availability of the kernel for that block size and prediction mode. For chroma components, the selection of the transform type may be performed in an implicit manner. For example, for intra prediction residuals, the transform type may be selected according to the intra prediction mode, as specified in Table 4. For inter prediction residuals, the transform type of the chroma block may be selected according to the transform type selection of the co-located luma block. Thus, for chroma components, there is no signaling of the transform type in the bitstream. [Table 3] Table 3: AV1 hybrid transform kernels and their availability based on prediction mode and block size. [Table 4] Table 4: Transform type selection for chroma component intra prediction residuals
[0148] In some implementations, a secondary transform may be performed on the primary transform coefficients. For example, a low-frequency non-separable transform (LFNST), known as a reduced secondary transform, may be applied between the forward primary transform and the quantization (at the encoder) and between the inverse quantization and the inverse primary transform (at the decoder side) to further decorrelate the primary transform coefficients, as shown in FIG. 25. In essence, the LFNST may obtain a portion of the primary transform coefficients, e.g., a low-frequency portion (thus being "reduced" from the complete set of primary transform coefficients of the transform block), to proceed to the secondary transform. In one example of LFNST, a 4×4 non-separable transform or an 8×8 non-separable transform may be applied depending on the transform block size. For example, a 4×4 LFNST may be applied to a small transform block (e.g., min(width, height)<8), while an 8×8 LFNST may be applied to a large transform block (e.g., (width, height)>8). For example, if an 8x8 transform block is subjected to a 4x4 LFNST, then only the low frequency 4x4 portion of the 8x8 primary transform coefficients undergo a further secondary transform.
[0149] As specifically shown in FIG. 25, a transform block may be 8×8 (or 16×16). Thus, a forward primary transform 2502 of the transform block generates an 8×8 (or 16×16) primary transform coefficient matrix 2504, where each square unit represents a 2×2 (or 4×4) portion. For example, the input to the forward LFNST may not be the entire 8×8 (or 16×16) primary transform coefficients. For example, a 4×4 (or 8×8) LFNST may be used for the secondary transform. Thus, as shown in the shaded portion (top left) 2506, only the 4×4 (or 8×8) low frequency primary transform coeffects of the primary transform coefficient matrix 2504 may be used as input to the LFNST. The remaining portion of the primary transform coefficient matrix may not be subject to the secondary transform. Thus, after the secondary transform, the portion of the primary transform coefficients that are subject to the LFNST become secondary transform coefficients, while the remaining portion not subject to the LFNST (e.g., the unshaded portion of matrix 2504) retains the corresponding primary transform coefficients. In some implementations, the remaining portion not subject to the secondary transform may be set to all zero coefficients.
[0150] An example for application of a non-separable transform used in LFNST is shown below: To apply the exemplary 4×4 LFNST, a 4×4 input block X (representing a 4×4 low frequency portion of a block of primary transform coefficients, e.g., shaded portion 2506 of linear transform matrix 1404 in FIG. 25) can be expressed as follows:
number
[0151] This 2-D input matrix is first sorted into the vector
number
number
[0152] The non-separable transform of the 4×4 LFNST is then [Number] can be calculated as. [Number] indicates the output transform coefficient vector, and T is a 16×16 transform matrix. The resulting 16×1 coefficient vector [Number] is then inverse-scanned as a 4×4 block using the scan order of that block (e.g., horizontal, vertical, or diagonal). Coefficients with small indices can be placed at smaller scan indices of the 4×4 coefficient block. In this way, the redundancy of the primary transform coefficients X can be further exploited via the second transform T. Thereby, additional compression enhancement is provided.
[0153] The above exemplary LFNST is based on a direct matrix multiplication approach for applying non-separable transforms so as to be implemented in a single pass without multiple iterations. In some further implementation examples, the dimensions of the non-separable transform matrix (T) of the exemplary 4×4 LFNST can be further reduced to minimize the computational complexity and the memory space requirements for storing the transform coefficients. Such an implementation can be referred to as a reduced non-separate transform (RST). More specifically, the main idea of the RST is to map an N (where N is 4×4 = 16 in the above example, but may be equal to 64 for an 8×8 block) -dimensional vector to an R -dimensional vector in another space, where N / R (R < N) represents the dimension reduction factor. Thus, instead of an N×N transform matrix, the RST matrix becomes an R×N matrix as follows. [Number] (Formula 4)
[0154] Here, the R rows of the transformation matrix are a reduced R basis for the N-dimensional space. The transformation thus transforms an N-dimensional input vector into a reduced R-dimensional output vector. Thus, and as shown in Figure 25, the secondary transform coefficients 2508 transformed from the primary coefficients 2506 are reduced in dimension by a factor or N / R. The three squares around 2508 in Figure 25 may be padded with zeros.
[0155] The inverse transform matrix of the RTS may be the transpose of its forward transform. For an exemplary 8×8 LFNST (described more divergently here as opposed to the 4×4 LFNST described above), an exemplary reduction factor of 4 may be applied, and thus the 64×64 direct non-separable transform matrix is correspondingly reduced to a 16×64 direct matrix. Furthermore, in some implementations, some, but not all, of the input primary coefficients may be linearized into the input vectors of the LFNST. For example, only a portion of the exemplary 8×8 input primary transform coefficients may be linearized into the X vector described above. For a specific example, of the four 4×4 quadrants of the 8×8 primary transform coefficient matrix, the bottom right (high frequency coefficients) may be omitted, and only the other three quadrants are linearized into a 48×1 vector using a predefined scan order rather than a 64×1 vector. In such implementations, the non-separable transform matrix may be further reduced from 16×64 to 16×48.
[0156] Thus, an example of a reduced 48×16 inverse RST matrix can be used at the decoder side to generate the top left, top right, and bottom left 4×4 quadrants of the 8×8 core (primary) transform coefficients. Specifically, if a further reduced 16×48 RST matrix is applied instead of the 16×64 RST in the same transform set configuration, the non-separable secondary transform takes as input 48 matrix elements vectorized from the three 4×4 quadrant blocks of the 8×8 primary coefficient block, excluding the bottom right 4×4 block. In such an implementation, the omitted bottom right 4×4 primary transform coefficient would be ignored in the secondary transform. This further reduced transform converts the 48×1 vector into a 16×1 output vector, which is inversely scanned into a 4×4 matrix to fill 2508 in FIG. 25. The three squares of secondary transform coefficients surrounding 2508 can be zero-filled.
[0157] With the help of such dimension reduction in the RST, the memory usage for storing all LFNST matrices is reduced. In the above example, the memory usage can be reduced, for example, from 10KB to 8KB, with a fairly small performance loss compared to the implementation without dimension reduction.
[0158] In some implementations, to reduce complexity, LFNST may be further restricted to be applied only if all coefficients other than the portion of the primary transform coefficients subject to LFNST (e.g., outside of 2506 portion of 1404 in FIG. 25) are non-significant. Thus, when LFNST is applied, all primary-only transform coefficients (e.g., the unshaded portion of primary coefficient matrix 2504 in FIG. 25) may be close to zero. Such a restriction allows conditioning of the LFNST index signaling at the last-significant position, and thus avoids additional coefficient scanning that may be required to check for valid coefficients at certain positions if this restriction is not applied. In some implementations, the worst-case processing of LFNST (in terms of multiplications per pixel) may limit the separate transforms for 4×4 and 8×8 blocks to 8×16 and 8×48 transforms, respectively. In such cases, when LFNST is applied, for other sizes less than 16, the last valid scan position needs to be less than 8. For blocks of shape 4xN and Nx4 with N>8, the above restriction means that LFNST is now applied only once to the top-left 4x4 region. Since only the first-order coefficients are all zero when LFNST is applied, the number of operations required for the first-order transform in such cases is reduced. From the encoder's perspective, quantization of coefficients can be simplified when the LFNST transform is tested. Rate-distortion optimized quantization (RDO) needs to be performed on at most the first 16 coefficients (in scan order), and the remaining coefficients may be forced to zero.
[0159] In some implementations, the available RST kernels may be specified as a number of transform sets, with each transform set containing many non-separable transform matrices. For example, there may be a total of four transform sets and two non-separable transform matrices (kernels) for each transform set used in LFNST. These kernels may be pre-trained offline and are therefore data driven. The offline trained transform kernels may be stored in memory or hard-coded in the encoding or decoding device for use during encoding / decoding. The selection of the transform set during the encoding or decoding process may be determined by the intra-prediction mode. The mapping from intra-prediction mode to transform sets may be predefined. An example of such a predefined mapping is shown in Table 5. For example, as shown in Table 5, for the current block (i.e., 81<=predModeIntra<=83), if one of the three Cross-Component Linear Model (CCLM) modes (INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used, transform set 0 may be selected for the current chroma block. For each transform set, the selected non-separable secondary transform candidate may be further specified by an explicitly signaled LFNST index. For example, the index may be signaled in the bitstream after the transform coefficients, once per intra CU. [Table 5] Table 5: Conversion Selection Table
[0160] Since LFNST is restricted to be applicable only when all coefficients outside the first coefficient subgroup or portion are non-significant in the above implementation example, the LFNST index coding depends on the position of the last significant coefficient. In addition, the LFNST index may be context coded, but does not depend on the intra prediction mode, and only the first bin may be context coded. Furthermore, LFNST may be applied for intra CUs in both intra slices and inter slices, and for both luma and chroma. If dual tree is enabled, the luma and chroma LFNST indexes may be signaled separately. In the case of inter slices (dual tree disabled), a single LFNST index may be signaled and used for both luma and chroma.
[0161] In some implementations, LFNST may be disabled and the RST index may not be signaled when an Intra Sub-Partitioning (ISP) mode is selected, since applying RST to all feasible partition blocks is likely to result in little performance improvement. Furthermore, disabling RST for ISP prediction residuals may reduce encoding complexity. In some further implementations, LFNST may also be disabled and the RST index may not be signaled when a Multiple Linear Regression Intra Prediction (MIP) mode is selected.
[0162] Considering that due to existing maximum transform size limitations (e.g., 64×64), large CUs larger than 64×64 (or other predefined size representing the maximum transform block size) are implicitly split (e.g., TU tiling), LFNST index lookup could increase data buffering by 4 times for a given number of decode pipeline stages. Thus, in some implementations, the maximum size allowed for LFNST may be limited, for example, to 64×64. In some implementations, LFNST may be enabled only as a primary transform using DCT2.
[0163] In some other implementations, an intra-secondary transform (IST) is provided for the luma component by defining 12 sets of secondary transforms, for example with 3 kernels in each set. An intra-mode dependent index may be used for transform set selection. Kernel selection within a set may be based on signaled syntax elements. IST may be enabled when either DCT2 or ADST is used as both horizontal and vertical primary transforms.
[0164] In some implementations, a 4x4 non-separable transform or an 8x8 non-separable transform can be selected according to the block size. If min(tx_width,tx_height,tx_width,tx_height)<8, a 4x4 IST can be selected. For larger blocks, an 8x8 IST can be used, where tx_width and tx_height correspond to the width and height of the transform block, respectively. The input to the IST can be low-frequency linear transform coefficients in zigzag scan order.
[0165] In some other implementations, without limitation, the number of kernel sets may be other than 12. For example, there may be 14 secondary transform kernels. The mapping between the secondary kernel sets and the various intra-prediction modes may be pre-determined or pre-configured. For example, a particular intra-prediction mode may be mapped to one of the 12 or 14 sets. Each kernel set may include a number of secondary kernels other than 3. For example, there may be 6 secondary kernels in each secondary transform kernel set. The number of kernels in each set may be determined based on a trade-off between the overhead associated with increasing the number of secondary kernels and the potential additional coding gain. In some implementations, the number of kernels in each kernel set may be 3 to 6. The coding gain over having 6 optional secondary transform kernels in each kernel set may be sufficiently small from a practical and statistical standpoint compared to the increased coding overhead.
[0166] In conventional implementations of either separable or non-separable secondary transforms (to better capture directional texture patterns), however, the number of secondary transform kernel options (or candidates) available in each secondary kernel set, pre-determined by the intra prediction mode to kernel set mapping (corresponding to some combination of intra prediction mode, primary transform type, block size), may be fixed and not adaptive. Some statistical studies on kernel usage have shown that for a given combination of intra prediction mode, primary transform type, block size, and other parameters, only a reduction in the number of kernel options (or candidates) of the secondary transform is necessary to capture directional information. Such a reduction in kernel options (or candidates) can reduce the signaling overhead of the proposed scheme. Various implementation examples below provide a scheme of adaptive secondary kernel options according to a combination of intra prediction mode, primary transform type, block size, and other potentially related parameters.
[0167] These implementations may be used individually or in combination in any order. Furthermore, each implementation or combination of implementations may be applied to an encoder in determining a secondary transform kernel and signaling the selection of the kernel set and the secondary kernel from the kernel set at various coding levels (block, macroblock, slice, image, frame, etc.). Each implementation or combination of implementations may also be applied to a decoder for identifying the secondary transform kernel set and the selected secondary transform kernel from the kernel set for each transform block based on various signaling information in the encoded bitstream. The encoder and decoder employing the disclosed method may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-transitory computer-readable medium. In the following, the term block may be interpreted as a transform block (TU). However, the basic principles disclosed below apply to other levels of video blocks, including but not limited to prediction blocks, coding blocks, or coding units, i.e., CUs. Furthermore, in the following disclosure, the term block size is used to refer to any one or combination of block width, block height, block aspect ratio, block area size, the larger of block width and height, the smaller of block width and height, and equal (square) block width and height. Furthermore, if an intra prediction mode is not a smooth mode and depends on prediction samples according to a given prediction direction, the intra prediction mode may be referred to as an angle mode or a direction mode.
[0168] In some common implementations, the number of secondary transform kernels available for selection can be adaptive according to any one or combination of coding characteristics, including but not limited to intra-prediction mode, primary transform type, and block size. For a particular block (such as a transform block), the final candidate selected among the corresponding secondary transform kernel set is signaled using a secondary transform kernel index to the corresponding kernel set indicated by stxIdx in the encoded bitstream (alternatively referred to as a "kernel index" or "index"). In some implementations, one of the possible stxIdx values can indicate that no secondary transform is performed and thus no secondary transform kernel is selected. In some other implementations, whether a particular block undergoes a secondary transform can be indicated by a separate signaling, and thus stxIdx can be used and signaled only if an actual kernel in the corresponding secondary kernel set is used for the particular block.
[0169] In some implementations, the mapping between various combinations of these coding characteristics (e.g., intra-prediction modes, primary transform types, and block sizes) and secondary transform kernel sets may be pre-determined. The secondary kernels in each kernel set may be pre-trained. They may be pre-defined and known to the encoder and decoder, and therefore may not need to be signaled. In some other implementations, they may be signaled in the bitstream. Although the secondary transform kernel sets may differ from each other in terms of the number of secondary transform kernels available for selection, the actual trained secondary transform kernels among the various secondary transform kernel sets may or may not overlap. The term "overlap" is used to indicate that two different secondary transform kernel sets, having the same or different number of optional kernels, may share at least one pre-trained secondary transform kernel.
[0170] In some specific implementation examples, for the secondary transform for encoding a block or the inverse secondary transform in decoding the block, an adaptive number of transform kernels (or candidates) may be selected in coding and decoding the block depending on the intra prediction mode associated with the block. As a non-limiting example, when the intra prediction mode is any one of vertical mode (V_PRED), horizontal mode (H_PRED), smooth horizontal mode (SMOOOTH_H_PRED), or smooth vertical mode (SMOOTH_V_PRED), the corresponding secondary transform kernel set may include N transform kernels (or candidates) available for selection. Meanwhile, for all other intra prediction modes, the corresponding secondary transform kernel set may include K transform kernels (or candidates) available for selection. Here, N and K are different non-negative integers. For example, N and K may be any one of non-negative integers including, but not limited to, 0, 1, 2, 3, 4, 5, and 6. The actual secondary transform kernels in each secondary transform kernel set may be pre-trained. They may be predefined or signaled in the bitstream. In some implementations, for each intra-coded block, a specific secondary transform kernel is selected and used from the corresponding secondary transform kernel set, and the selection of the specific kernel may be signaled by stxIdx in the bitstream. Although the above example provides a secondary transform kernel set having an adaptive number of two optional kernels, the underlying principles may be extended to kernel sets having different adaptive numbers of three or more optional kernels.
[0171] In some other specific implementations, depending on the primary transform type of a particular block, the number of transform kernels (or candidates) available for selection in the corresponding secondary transform kernel set may be adaptive. For example, if the primary transform type is ADST_ADST (ADST in both vertical and horizontal dimensions), only N transform kernels (or candidates) are available in the corresponding secondary transform kernel set. Meanwhile, for all other primary transform types, K transform kernels (or candidates) are available, where N and K may be different non-negative integer values. For example, N and K may be any one of non-negative integers including, but not limited to, 0, 1, 2, 3, 4, 5, 6. The actual secondary transform kernels in each secondary transform kernel set may be pre-trained. They may be pre-defined or may be signaled in the bitstream. In some implementations, for each intra-coded block, a particular secondary transform kernel is selected and used from the corresponding secondary transform kernel set, and the selection of the particular kernel may be signaled by stxIdx in the bitstream. While the above example provides a quadratic transformation kernel set having an adaptive number of two optional kernels, the underlying principles can be extended to kernel sets having different adaptive numbers of three or more optional kernels.
[0172] In some other implementations, depending on the block size, the number of transform kernels (or candidates) available for selection in the corresponding secondary transform kernel set may be adaptive. For example, when the transform block size is less than a predetermined block size threshold, only N transform kernels (or candidates) are available. On the other hand, for a transform block size equal to or greater than the block size threshold, K transform kernels (or candidates) are available, where N and K may be different non-negative integer values. For example, N and K may be any one of non-negative integers including, but not limited to, 0, 1, 2, 3, 4, 5, 6. The block size may be any one of a block size in a width dimension, a block size in a height dimension, a block region size, a larger of the width dimension and the height dimension, a smaller of the width dimension and the height dimension, and the like. For example, the block size threshold may be specified as 8 in height. As another example, the block size threshold may be specified as 8 in width. As another example, the block size threshold may be specified as 8 in width and height. As another example, the block size threshold may be specified as an area (e.g., 64 pixels). As another example, the block size threshold may be specified as W×H (e.g., 8×8). Such a block size threshold may mean that block widths smaller than W are below the threshold, or blocks with heights smaller than H are below the threshold, or blocks with widths smaller than W and heights smaller than H are below the threshold, or blocks with widths smaller than W or heights smaller than H are below the threshold, or blocks with areas smaller than W×H pixels are below the threshold. Furthermore, the actual secondary transform kernels in each secondary transform kernel set may be pre-trained. They may be predefined or may be signaled in the bitstream. In some implementations, for each intra-coded block, a specific secondary transform kernel is selected and used from the corresponding secondary transform kernel set, and the selection of the specific kernel may be signaled by stxIdx in the bitstream.While the above example provides a quadratic transformation kernel set having an adaptive number of two optional kernels, the underlying principles can be extended to kernel sets having different adaptive numbers of three or more optional kernels.
[0173] In some other implementations, for any one or some combinations of the various coding parameters above, an adaptive number of transform kernels (or candidates) may be available for selection. For example, when the intra prediction mode is one of vertical mode (V_PRED), horizontal mode (H_PRED), smooth horizontal mode (SMOOTH_H_PRED), or smooth vertical mode (SMOOTH_V_PRED), the primary transform type is ADST_ADST, and the transform block size is equal to or greater than (or smaller than) the block size threshold, only N transform kernels (or candidates) are available, while for all other combinations, K transform kernels (or candidates) are available, where N and K may be different non-negative integer values. For example, N and K may be any one of non-negative integers, including, but not limited to, 0, 1, 2, 3, 4, 5, and 6. The terms block size and size threshold may convey similar meanings as in the other implementations above. Although this example provides a quadratic transformation kernel set having an adaptive number of two optional kernels, the underlying principles can be extended to kernel sets having different adaptive numbers of optional kernels, such as three or more. Schemes for adaptively allocating the number of available kernels according to any combination of these coding parameters are also contemplated.
[0174] In some other implementations, for any of the above examples, if only N (or K) transform kernels (or candidates) are available for a block with a corresponding secondary transform kernel set, then the stxIdx signaling may need to support only up to N (or K) transform kernels (or candidates). Such a reduction in possible stxIdx for a particular coding parameter or combination thereof (e.g., intra-prediction mode, primary transform mode, block size, etc.) can help improve bitstream signaling overhead.
[0175] In some other implementations, for any of the above examples, if different numbers of secondary transform kernels (or candidates) are used for different kernel sets, different context modeling is applied for entropy coding stxIdx. In other words, stxIdx used for indexing between kernel sets having a first number of candidate kernels may be entropy coded based on a first context model or a first probability distribution. Meanwhile, stxIdx used for indexing between kernel sets having a second number of candidate kernels different from the first number may be entropy coded based on a second context model different from the first context model. This may be based on the possibility that the probability distribution of stxIdx for the two different numbers of kernel sets is different.
[0176] In some other implementations, for any of the above examples, when different numbers of secondary transform kernels (or candidates) are adaptively used for different secondary kernel sets, stxIdx may be first binarized, and then the context modeling of the first M bins may be shared among the stxIdx of kernel sets having different numbers of candidate secondary transform kernels.
[0177] For a specific example, when N=3 for the first kernel set and K=4 for the second kernel set, stxIdx may be binarized as in Table 6. The context modeling used for entropy coding of the first two bins may be shared for the codeword used when N=3 and K=4. [Table 6] Table 6
[0178] FIG. 26 shows an example flowchart 2600 that follows the principles underlying the implementation above. The example flow starts at S2601. At S2610, an encoded video stream is analyzed and processed to generate a set of secondary transform coefficients of a video block, an intra-prediction mode associated with the video block, and a kernel index into a group of secondary transform kernels. At S2620, a group of secondary transform kernels based on the intra-prediction mode is identified. At S2630, an inverse secondary transform of the set of secondary transform coefficients is performed to generate primary transform coefficients of the video block based on a secondary transform kernel from the group of secondary transform kernels identified by the kernel index, where the number of kernels in the group of secondary transform kernels depends on at least one of an intra-prediction mode associated with the video block, a size of the video block, or a primary transform type associated with the video block. The process ends at S2699.
[0179] FIG. 27 shows a flowchart 2700 of another exemplary method following the principles underlying the implementation above. The flow of the exemplary method starts at S2701. At S2710, a residual block of an intra-predicted video block is transformed using at least one primary transform kernel to generate a set of primary transform coefficients. At S2720, a current secondary transform kernel is selected from among a group of secondary transform kernels, where the number of kernels in the group of secondary transform kernels depends on at least one of an intra-prediction mode of the intra-predicted video block, a size of the intra-predicted video block, or a primary transform type of the at least one primary transform kernel. At S2730, a kernel index of a current secondary transform kernel in the group of secondary transform kernels is determined. At S2740, the set of primary transform coefficients is transformed to generate a set of secondary transform coefficients using the current secondary transform kernel. At S2750, the kernel index and the set of secondary transform coefficients are encoded into an encoded video stream associated with the intra-predicted video block. The process ends at S2799.
[0180] The embodiments of the present disclosure may be used separately or in any order in combination. Furthermore, each of the methods (or embodiments), the encoder, and the decoder may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to luma blocks or chroma blocks.
[0181] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 28 illustrates a computer system (2800) suitable for implementing certain embodiments of the disclosed subject matter.
[0182] Computer software may be coded using any suitable machine code or computer language. It may be assembled, compiled, linked, or otherwise processed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., may be subject to similar mechanisms for creating code that contains instructions that may be executed directly, or through an interpreter, microcode execution, etc.
[0183] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things, and the like.
[0184] 28 for the computer system (2800) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of the computer system (2800).
[0185] The computer system 2800 may include certain human interface input devices that may be responsive to input by one or more human users through, for example, tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, clapping), visual input (such as gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media that do not necessarily involve direct human conscious input, such as audio (such as speech, music, ambient sounds), images (such as scanned images, photographic images acquired from a still image camera), and video (such as two-dimensional video, three-dimensional video including stereoscopic vision).
[0186] The input human interface devices may include one or more of a keyboard (2801), a mouse (2802), a trackpad (2803), a touch screen (2810), a data glove (not shown), a joystick (2805), a microphone (2806), a scanner (2807), and a camera (2808) (only one of each is depicted).
[0187] The computer system 2800 may also include certain human interface output devices that can stimulate one or more of the senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., a touch screen (2810), data gloves (not shown), or joystick (2805), although there are also haptic feedback devices that do not function as input devices), audio output devices (such as speakers (2809), headphones (not shown)), visual output devices (such as screens (2810), including CRT screens, LCD screens, plasma screens, each with or without touch screen input capability, and each with or without haptic feedback capability - some of which may output two dimensional visual output or three or more dimensional output via means such as stereographic output, virtual reality glasses (not shown), holographic displays and smoke tanks (not shown), and printers (not shown).
[0188] The computer system (2800) may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2820) with associated media such as CD / DVD (2821), thumb-drives (2822), removable hard drives or solid state drives (2823), traditional magnetic media such as tape and floppy disks (not shown), specialized ROM / ASIC / PLD based devices such as security dongles (not shown), and the like.
[0189] Those skilled in the art should also understand that the term "computer readable media" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transient signals.
[0190] The computer system (2800) may also include an interface (2854) to one or more communication networks (2855). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, TV wide area or wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial including CAN bus, and the like. Certain networks typically require an external network interface adapter (such as, for example, a USB port on the computer system (2800)) to be connected to a certain general-purpose data port or peripheral bus (2849). Others are typically integrated into the core of the computer system (2800) by attachment to a system bus, as described below (such as, for example, an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2800) can communicate with other entities. Such communications may be unidirectional, receive-only (e.g., broadcast television), unidirectional transmit-only (e.g., a CAN bus to a given CAN bus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Predetermined protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.
[0191] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (2840) of the computer system (2800).
[0192] The cores (2840) may include one or more central processing units (CPUs) (2841), graphics processing units (GPUs) (2842), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (2843), hardware accelerators for certain tasks (2844), graphics adapters (2850), etc. These devices may be connected via a system bus (2848), along with read only memory (ROM) (2845), random access memory (2846), internal mass storage such as an internal non-user accessible hard drive, SSD, etc. (2847). In some computer systems, the system bus (2848) is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (2848) or may be connected via a peripheral bus (2849). In one example, a screen (2810) may be connected to the graphics adapter (2850). Peripheral bus architectures include PCI, USB, and the like.
[0193] The CPU (2841), GPU (2842), FPGA (2843), and accelerator (2844) can execute predetermined instructions, which in combination can constitute the above-mentioned computer code. The computer code can be stored in ROM (2845) or RAM (2846). Transient data can also be stored in RAM (2846), while permanent data can be stored in, for example, internal mass storage (2847). Rapid storage and retrieval in any of the memory devices can be made possible through the use of cache memory, which can be closely associated with one or more of the CPU (2841), GPU (2842), mass storage (2847), ROM (2845), RAM (2846), etc.
[0194] The computer-readable media can have computer code thereon for performing various computer-implemented operations. The media and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.
[0195] As a non-limiting example, a computer system having the architecture (2800), and in particular the cores (2840), can provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible, computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage as described above, as well as storage associated with the cores (2840) of a non-transitory nature, such as core-internal mass storage (2847) or ROM (2845). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the cores (2840). The computer-readable media may include one or more memory devices or chips according to particular needs. The software can cause the cores (2840), and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute certain processes described herein, or certain portions of certain processes, including defining data structures stored in RAM (2846) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system can provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerators (2844) that can operate in place of or in conjunction with software to execute certain processes or certain portions of certain processes described herein. References to software include logic, and vice versa, as appropriate. References to computer-readable media can include circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, as appropriate. The present disclosure encompasses any appropriate combination of hardware and software.
[0196] While this disclosure has described several exemplary embodiments, there exist modifications, permutations, and various substitute equivalents which are within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise numerous systems and methods which, although not explicitly shown or described in this disclosure, embody the principles of the present disclosure and are therefore within its spirit and scope.
[0197] Appendix A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplemental Extended Information VUI: Video Usability Information GOP: Group of Pictures TU: conversion unit PU: Prediction unit CTU: Coding Tree Unit CTB: coding tree block PBs: Predicted blocks HRD: Hypothetical Reference Decoder SNR: Signal to Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD:Compact Disc DVD: Digital Video Disc ROM: Read-only memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Array SSD: Solid State Drive IC: Integrated Circuit HDR: High Dynamic Range SDR: Standard Dynamic Range JVET: Joint Video Exploration Team MPM: Most Probable Mode WAIP: Wide-angle Intra Prediction CU: coding unit PU: Prediction unit TU: conversion unit CTU: Coding Tree Unit PDPC: Position-dependent prediction combination ISP: Intra Subpartition SPS: Sequence Parameter Settings PPS: Picture Parameter Set APS: Adaptive Parameter Set VPS: Video Parameter Set DPS: Decoding Parameter Set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC-ALF: Cross-component adaptive loop filter CDEF: Constrained directional enhancement filter CCSO: Cross component sample offset LSO: Local Sample Offset LR: Loop recovery filter AV1:AOMedia Video 1 AV2:AOMedia Video 2 LFNST: Low Frequency Nonseparable Transform IST: Intra-quadratic transformation
Claims
1. 1. A method for processing video blocks in an encoded video stream, comprising: parsing and processing the encoded video stream; a set of secondary transform coefficients of said video block; an intra-prediction mode associated with the video block; and Kernel index into a group of quadratic transformation kernels, generating a identifying the group of secondary transform kernels based on the intra prediction mode; performing an inverse secondary transform of said set of secondary transform coefficients; generating primary transform coefficients for the video block based on a secondary transform kernel from the group of secondary transform kernels identified by the kernel index. and The number of kernels in said group of secondary transformation kernels is the intra-prediction mode associated with the video block; the size of the video block, or a linear transform type associated with the video block; is dependent on at least one of a number of kernels in the group of secondary transform kernels depends on the primary transform type associated with the video block. method.
2. The number of kernels in said group of secondary transformation kernels is N if the primary transform type is ADST for both vertical and horizontal dimensions of the video block, and K otherwise; N and K are distinct non-negative integers between 0 and 6; The method of claim 1.
3. The number of kernels in said group of secondary transformation kernels is N if the size of the video block is less than a predetermined threshold, and K otherwise; N and K are distinct non-negative integers between 0 and 6; The method of claim 1.
4. the predetermined threshold is 8×8 or 8 for either the horizontal or vertical dimension of the video block. The method according to claim 3.
5. The number of kernels in said group of secondary transformation kernels is N if the video block is characterized by any one of a predefined subset of intra prediction modes, any one of a predefined subset of linear transform types, and any one of a predefined subset of block sizes, and K otherwise; N and K are distinct non-negative integers between 0 and 6; The method of claim 1.
6. The number of kernels in said group of secondary transformation kernels is N if the video block is characterized by one of vertical, horizontal, smooth horizontal, and smooth vertical intra prediction modes, ADST_ADST linear transform modes in both vertical and horizontal dimensions, and a block size smaller than a predefined size threshold; and otherwise, K. The method according to claim 5.
7. if the group of secondary transform kernels includes only N or K secondary transform kernels, the kernel index signaling scheme in the encoded video stream supports only N or up to N index options. The method according to claim 5.
8. where a number of kernels in the group of secondary transform kernels is N and a number of kernels in the group of secondary transform kernels is K, the kernel indexes are entropy coded in the encoded video stream using different context models. The method according to claim 5.
9. and when a number of kernels in the group of secondary transform kernels used for different video blocks in the encoded video stream are different, entropy coding of the binary codeword of the kernel index into the group of secondary transform kernels having different numbers of kernels shares context modeling for at least one bin of the binary codeword. The method of claim 1.
10. 1. A method for processing intra-predicted video blocks, comprising the steps of: transforming a residual block of the intra-predicted video block using at least one linear transform kernel to generate a set of linear transform coefficients; selecting a current secondary transformation kernel within a group of secondary transformation kernels, the number of kernels within said group of secondary transformation kernels being: an intra-prediction mode for the intra-predicted video block; the size of the intra-predicted video block; or a linear transformation type of the at least one linear transformation kernel; and determining a kernel index of the current secondary transformation kernel within the group of secondary transformation kernels; transforming the set of primary transform coefficients using the current secondary transform kernel to generate a set of secondary transform coefficients; encoding the kernel index and the set of secondary transform coefficients into an encoded video stream associated with the intra-predicted video block; Including, a number of kernels in the group of secondary transform kernels depends on the primary transform type associated with the intra-predicted video block. method.
11. The number of kernels in said group of secondary transformation kernels is N if the primary transform type is ADST for both vertical and horizontal dimensions of the intra-predicted video block, and K if not bi-dimensional; N and K are distinct non-negative integers between 0 and 6; The method of claim 10.
12. The number of kernels in said group of secondary transformation kernels is N if the size of the intra-predicted video block is less than a predetermined threshold, and K otherwise; N and K are distinct non-negative integers between 0 and 6. The method of claim 10.
13. the predetermined size threshold is 8×8 or 8 for either a horizontal or vertical dimension of the intra-predicted video block. The method of claim 12.
14. The number of kernels in said group of secondary transformation kernels is N if the intra-predicted video block is characterized by any one of a predefined subset of intra-prediction modes, any one of a predefined subset of linear transform types, and any one of a predefined subset of block sizes, and K otherwise; N and K are distinct non-negative integers between 0 and 6. The method of claim 10.
15. The number of kernels in said group of secondary transformation kernels is N if the intra-predicted video block is characterized by one of vertical, horizontal, smooth horizontal, and smooth vertical intra-prediction modes, ADST_ADST linear transform modes in both vertical and horizontal dimensions, and a block size smaller than a predefined size threshold; and otherwise, K. The method of claim 14.
16. if the group of secondary transform kernels includes only N or K secondary transform kernels, the kernel index signaling scheme in the encoded video stream supports only N or up to N index options. The method of claim 14.
17. where a number of kernels in the group of secondary transform kernels is N and a number of kernels in the group of secondary transform kernels is K, the kernel indexes are entropy coded in the encoded video stream using different context models. The method of claim 14.
18. and when a number of kernels in the group of secondary transform kernels used for different video blocks in the encoded video stream are different, entropy coding of the binary codeword of the kernel index into the group of secondary transform kernels having different numbers of kernels shares context modeling for at least one bin of the binary codeword. The method of claim 10.
19. a memory for storing a set of instructions; A processor for executing said instructions to perform the method of any one of claims 1 to 18; 23. A video processing device comprising:
20. A computer program comprising a set of computer-executable instructions, The instructions, when executed by a processor of a video processing apparatus, cause the video processing apparatus to perform the method of any one of claims 1 to 18. Computer program.
Citation Information
Patent Citations
Image decoding method, image encoding method, image decoding device, image encoding device, and recording medium
JP2019517221A
Secondary transform based video coding method and apparatus
JP2022511860A
Method, apparatus and system for encoding and decoding blocks of video samples
JP2022548448A
Image coding method based on secondary transform, and device therefor
WO2020116961A1
Method, apparatus and system for encoding and decoding a block of video samples
WO2021051156A1
