Methods, apparatus, media, and programs for video decoding

Abstract

The present invention provides a method, apparatus, medium, and program for enabling implicit multiple translation selection (MTS) for intracoded blocks. [Solution] The method involves a first high-level syntax (HLS) element indicating whether explicit MTS is enabled or disabled for an intracoded block, and a second HLS element indicating whether explicit MTS is enabled or disabled for an intercoded block. If the first HLS element indicates that explicit MTS is disabled for an intracoded block, and the second HLS element indicates that explicit MTS is enabled for an intercoded block, then implicit MTS is enabled for the intracoded block.

Description

[Technical Field]

[0001] Embedding by reference This application claims the benefit of U.S. Patent Application No. 16 / 878,390, “Method and Apparatus for Video Coding,” filed on 19 May 2020, which in turn claims the benefit of U.S. Provisional Application No. 62 / 860,149, “High-level Syntax Control on Enabling Implicit Transform Selection,” filed on 11 June 2019. The disclosures of the prior applications are incorporated herein by reference in their entirety.

[0002] This disclosure describes embodiments generally related to video coding. [Background technology]

[0003] The background art provided herein is intended to provide a general context for this disclosure. The inventors' research, to the extent described in this background art section, and any aspects of the description that may not be considered prior art at the time of filing, are not expressly or implicitly considered prior art to this disclosure.

[0004] Video coding and decoding can be performed using inter-image prediction with motion compensation. Uncompressed digital video can contain a series of images, each image having spatial dimensions of, for example, 1920×1080 luminance samples and associated chrominance samples. The series of images can have, for example, 60 images per second or a fixed or variable image rate (also informally called frame rate) of 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at a 60 Hz frame rate) requires a bandwidth of nearly 1.5 Gbit / s. Using such video for one hour requires more than 600 GByte of storage space.

[0005] One purpose of video coding and decoding may be to reduce the redundancy of the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage space requirements, sometimes by more than two orders of magnitude. Both lossless and lossy compression, and combinations thereof, may be used. Lossless compression refers to a technique that allows an exact copy of the original signal to be reconstructed from the compressed original signal. With lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal is useful for the intended purpose. In the case of video, lossy compression is widely adopted. The amount of distortion that can be tolerated varies depending on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect that the higher the tolerable / acceptable distortion, the higher the compression ratio that can be obtained.

[0006] Video encoders and decoders can utilize techniques from several broad categories, such as motion compensation, transformation, quantization, and entropy coding.

[0007] Video codec techniques may include a technique known as intra coding. In intra coding, sample values ​​are represented without referencing other data from samples or previously reconstructed reference images. In some video codecs, the image is spatially subdivided into blocks of samples. If all blocks of samples are coded in intra mode, the image can be an intra image. Intra images and their derivatives, such as independent decoder refresh images, can be used to reset the decoder state and thus can be used as the first image in the coded video bitstream and video session, or as a still image. Samples in intra blocks may be transformed, and the transformation coefficients may be quantized before entropy coding. Intra prediction may be a technique to minimize the sample values ​​in the pre-transformation region. In some cases, smaller post-transformation DC values ​​and smaller AC coefficients result in fewer bits being required at a given quantization step size to represent the block after entropy coding.

[0008] Traditional intra-coding, such as that known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include, for example, techniques that try from surrounding sample data and / or metadata obtained during encoding / decoding of spatially adjacent blocks of data that precede in the decoding order. Such techniques will hereafter be referred to as “intra-prediction” techniques. It should be noted that, in at least some cases, intra-prediction uses only reference data from the current image being reconstructed, and not reference data from the reference image.

[0009] Intra-prediction can take many different forms. When two or more of these techniques can be used in a given video coding technique, the techniques used can be coded in intra-prediction mode. In some cases, a mode may have submodes and / or parameters, which may be coded individually or included in a mode codeword. Which codeword is used for a given mode / submode / parameter combination may affect the gain in coding efficiency via intra-prediction, and therefore the entropy coding technique used to convert the codeword into a bitstream may also have an effect.

[0010] Certain modes of intra-prediction were introduced in H.264, improved in H.265, and further refined with newer coding techniques such as Joint Search Models (JEM), Versatile Video Coding (VVC), and Benchmark Sets (BMS). Predictor blocks can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of neighboring samples are copied to the predictor block according to direction. References to the direction in use can be coded within the bitstream or predicted themselves.

[0011] Motion compensation can be a lossy compression technique and may relate to a technique used to predict a newly reconstructed image or portion of an image after blocks of sample data from a previously reconstructed image or a portion of it (reference image) have been spatially shifted in the direction indicated by a motion vector (MV). In some cases, the reference image may be the same as the image currently being reconstructed. The MV can have two dimensions, X and Y, or three dimensions, the third of which is a representation of the reference image in use (the latter of which can indirectly be the time dimension).

[0012] In some video compression techniques, a motion vector (MV) applicable to a particular region of sample data can be predicted from other MVs, for example, from MVs related to another region of sample data that is spatially adjacent to the region being reconstructed and precedes that MV in the decoding order. This substantially reduces the amount of data required to code the MV, thereby eliminating redundancy and increasing compression. MV prediction can work effectively, for example, when coding an input video signal derived from a camera (known as natural video), because there is a statistical likelihood that regions larger than the region to which a single MV is applicable will move in a similar direction, and therefore, in some cases, can be predicted using similar motion vectors derived from the MVs of adjacent regions. This results in the found MV for a given region being similar to or identical to the MV predicted from the surrounding MVs, and after entropy coding, it can be represented with fewer bits than would be used if the MV were coded directly. In some cases, MV prediction can be an example of lossless compression of the signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, MV prediction itself may be lossy, for example, due to rounding errors when calculating the predictor from some surrounding MVs.

[0013] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec.H.265, "High Efficiency Video Coding," December 2016). Here, we will describe a technique called "spatial merging," one of the many MV prediction mechanisms provided by H.265.

[0014] Referring to Figure 1, the current block (101) contains samples found by the encoder during the motion search process so that it is predictable from the previous block of the same size, which has been spatially shifted. Instead of directly coding its MV, the MV can be derived from metadata associated with one or more reference images, for example from the most recent (decoding order) reference image, using the MV associated with one of the five surrounding samples indicated as A0, A1, and B0, B1, B2 (102 to 106, respectively). In H.265, the MV prediction can use a predictor from the same reference image used by the adjacent block. [Overview of the project] [Means for solving the problem]

[0015] Aspects of this disclosure provide a first method of video decoding in a video decoder. The method may include receiving a first high-level syntax (HLS) element indicating whether explicit multiplex selection (MTS) is enabled or disabled for an intracoded block, and receiving a second HLS element indicating whether explicit MTS is enabled or disabled for an intercoded block. The first and second HLS elements control the same set of coding blocks, including intracoded blocks and intercoded blocks. Implicit MTS may be enabled for an intracoded block if the first HLS element indicates that explicit MTS is disabled for an intracoded block and the second HLS element indicates that explicit MTS is enabled for an intercoded block.

[0016] One embodiment of the method may further include the step of applying an implicit MTS to an intracoded block. The type of transformation used to process the intracoded block can be determined according to the size of the intracoded block. In various examples, the first or second HLS element may be one of the following: a video parameter set (VPS) syntax element, a sequence parameter set (SPS) syntax element, an image parameter set (PPS) syntax element, a slice header syntax element, a tile header syntax element, or a tile group header syntax element. In one embodiment, the intracoded block is not coded in intra-subpartitioning (ISP) mode. In one embodiment, the method may further include the step of receiving a third HLS element indicating that MTS is enabled for each of the interconnected block and the intracoded block.

[0017] Aspects of the present disclosure provide a second method for video decoding in a video decoder. The second method may include receiving a first HLS element indicating whether MTS is enabled or disabled for an intracoded block, and receiving a second HLS element indicating whether an unseparated quadratic transform (NSST) or matrix-based intraprediction (MIP) is disabled or enabled for an intracoded block. Implicit MTS may be enabled for an intracoded block if the first HLS element indicates that explicit MTS is disabled for the intracoded block, and the second HLS element indicates that NSST or MIP is disabled for the intracoded block.

[0018] The present disclosure can also provide a third method of video decoding in a video decoder. The third method can include receiving an intra-coded block related to a first block-level syntax element indicating whether MTS is applied and a second block-level syntax element indicating whether NSST is applied. When the first block-level syntax element indicates that MTS is not applied and the second block-level syntax element indicates that NSST is not applied, an implicit MTS can be enabled for the intra-coded block.

[0019] It should be noted that although this application refers to NSST, the disclosed methods and systems can be applied to variations of NSST such as reduced-size transform (RST) and low-frequency non-separable secondary transform (LFNST). Therefore, NSST, RST, and / or LFNST can be used interchangeably throughout this application.

[0020] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

Brief Description of the Drawings

[0021] [Figure 1] It is a schematic diagram of the current block and its surrounding spatial merge candidates in an example. [Figure 2] It is a schematic diagram of a simplified block diagram of a communication system (200) according to an embodiment. [Figure 3] It is a schematic diagram of a simplified block diagram of a communication system (300) according to an embodiment. [Figure 4] It is a schematic diagram of a simplified block diagram of a decoder according to an embodiment. [Figure 5] It is a schematic diagram of a simplified block diagram of an encoder according to an embodiment. [Figure 6] It is a block diagram of an encoder according to another embodiment. [Figure 7] This is a block diagram of a decoder according to another embodiment. [Figure 8A] This figure shows an exemplary transformation core matrix of a 4-point DCT-2 according to one embodiment. [Figure 8B] This figure shows an exemplary transformation core matrix of an 8-point DCT-2 according to one embodiment. [Figure 8C] This figure shows an exemplary transformation core matrix of a 16-point DCT-2 according to one embodiment. [Figure 8D] This figure shows an exemplary transformation core matrix of a 32-point DCT-2 according to one embodiment. [Figure 9A] This figure shows the subblock types, sizes, and positions supported by subblock transformation (SBT) according to one embodiment. [Figure 9B] This figure shows the subblock types, sizes, and positions supported by subblock transformation (SBT) according to one embodiment. [Figure 9C] This figure shows the subblock types, sizes, and positions supported by subblock transformation (SBT) according to one embodiment. [Figure 9D] This figure shows the subblock types, sizes, and positions supported by subblock transformation (SBT) according to one embodiment. [Figure 10] This figure shows the number of subpartitions according to the block size in an intra-subpartition (ISP) coding mode according to one embodiment. [Figure 11] This figure shows an example where a block is divided into two subpartitions in ISP coding mode. [Figure 12] This figure shows an example where a block is divided into four subpartitions in ISP coding mode. [Figure 13A] This figure shows a 64x64 transformation core matrix for a 64-point DCT-2 transformation according to one embodiment. [Figure 13B] This figure shows a 64x64 transformation core matrix for a 64-point DCT-2 transformation according to one embodiment. [Figure 13C] This figure shows a 64x64 transformation core matrix for a 64-point DCT-2 transformation according to one embodiment. [Figure 13D] This figure shows a 64x64 transformation core matrix for a 64-point DCT-2 transformation according to one embodiment. [Figure 13E] This figure shows a 64x64 transformation core matrix for a 64-point DCT-2 transformation according to one embodiment. [Figure 14] This figure shows the conversion basis functions for DST / DCT conversion according to one embodiment. [Figure 15] This figure shows a table illustrating the mapping relationship between the mts_idx value and its respective horizontal or vertical transformation, according to one embodiment. [Figure 16A] This figure shows the conversion core matrix of a DST-7 conversion type according to one embodiment. [Figure 16B] This figure shows the conversion core matrix of a DST-7 conversion type according to one embodiment. [Figure 16C] This figure shows the conversion core matrix of a DST-7 conversion type according to one embodiment. [Figure 16D] This figure shows the conversion core matrix of a DST-7 conversion type according to one embodiment. [Figure 17A] This figure shows the conversion core matrix of a DCT-8 conversion type according to one embodiment. [Figure 17B] This figure shows the conversion core matrix of a DCT-8 conversion type according to one embodiment. [Figure 17C] This figure shows the conversion core matrix of a DCT-8 conversion type according to one embodiment. [Figure 17D] This figure shows the conversion core matrix of a DCT-8 conversion type according to one embodiment. [Figure 18] This figure shows an example of controlling the use of Multiple Transformation Selection (MTS) using Sequence Parameter Set (SPS) syntax elements. [Figure 19] This figure shows a mapping table between the intra-prediction mode and the transformation set according to one embodiment. [Figure 20]This figure shows an alternative conversion coding process (2000) for RST 8x8 using a 16x64 conversion core, according to one embodiment. [Figure 21] This figure shows an alternative conversion coding process (2100) for RST 8x8 using a 16x48 conversion core, according to one embodiment. [Figure 22] This figure shows an exemplary CU level syntax table (2200), where the syntax element lfnst_idx, indicating the selection of a low-frequency unseparated quadratic transform (LFNST) kernel, is signaled at the end of the CU level syntax. [Figure 23] This figure shows the process of a reduction transformation (2301) and the process of an inverse reduction transformation (2302) according to one embodiment. [Figure 24A] This figure shows the entire upper-left 8x8 coefficient (shaded subblock) of the residual block (2410) used as input for calculating the quadratic transformation in RST 8x8. [Figure 24B] This figure shows the top-left three 4x4 subblock coefficients (shaded subblocks) of the residual block (2410) used as input for calculating the quadratic transformation in RST 8x8. [Figure 25] This figure shows a table for selecting a conversion set based on an intra-prediction mode according to one embodiment. [Figure 26] This figure shows an exemplary process (2600) in matrix-based intra-prediction (MIP) mode. [Figure 27] This figure shows a CU-level syntax table according to one embodiment, with a flag signaling matrix-based intra-prediction (MIP) mode indicated in frame (2701). [Figure 28A] This diagram combines text (2800) that specifies a transformation coding process that performs explicit or implicit transformation selection of the current block based on relevant syntax elements received from the bitstream. [Figure 28B]This diagram combines text (2800) that specifies a transformation coding process that performs explicit or implicit transformation selection of the current block based on relevant syntax elements received from the bitstream. [Figure 29] This figure shows the modification (2900) to text (2800) corresponding to an implicit conversion enable scheme that allows implicit conversion of intra residual blocks and explicit conversion of inter residual blocks to coexist. [Figure 30] This figure shows the correction (3000) to text (2800) corresponding to a scenario where implicit conversion is enabled when non-separated quadratic conversion (NSST) is disabled. [Figure 31] This figure shows the correction (3100) to text (2800) corresponding to a scenario where implicit conversion is enabled when MIP is disabled. [Figure 32] This figure shows the correction (3200) to text (2800) for a scenario where neither MTS nor NSST applies to the current block. [Figure 33] This figure shows the correction (3300) to text (2800) for scenarios where neither MTS, NSST, nor MIP applies to the current block. [Figure 34] This figure shows a flowchart of the conversion coding process (3400) according to an embodiment of the present disclosure. [Figure 35] This figure shows a flowchart of the conversion coding process (3500) according to an embodiment of the present disclosure. [Figure 36] This figure shows a flowchart of the conversion coding process (3600) according to an embodiment of the present disclosure. [Figure 37] This is a schematic diagram of a computer system according to one embodiment. [Modes for carrying out the invention]

[0022] I. Video Coding Encoders and Decoders Figure 2 shows a simplified block diagram of a communication system (200) according to one embodiment of the present disclosure. The communication system (200) includes a plurality of terminal devices that can communicate with each other, for example, over a network (250). For example, the communication system (200) includes a first pair of terminal devices (210) and (220) interconnected over the network (250). In the example of Figure 2, the first pair of terminal devices (210) and (220) perform one-way transmission of data. For example, terminal device (210) can code video data (e.g., a stream of video images captured by terminal device (210)) for transmission to another terminal device (220) over the network (250). The encoded video data can be transmitted in the form of one or more coded video bitstreams. Terminal device (220) can receive coded video data from the network (250), decode the coded video data to restore video images, and display video images according to the restored video data. One-way data transmission can be common in applications such as media serving.

[0023] In another example, the communication system (200) includes a second pair of terminal devices (230) and (240) that perform bidirectional transmission of coded video data, which may occur, for example, during a video conference. For bidirectional transmission of data, in one example, each terminal device of terminal devices (230) and (240) can code video data (e.g., a stream of video images captured by the terminal device) for transmission to the other terminal device of terminal devices (230) and (240) via the network (250). Each terminal device of terminal devices (230) and (240) can also receive coded video data transmitted by the other terminal device of terminal devices (230) and (240), decode the coded video data to restore video images, and display video images on an accessible display device according to the restored video data.

[0024] In the example in Figure 2, terminal devices (210), (220), (230), and (240) may be represented as a server, a personal computer, and a smartphone, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure find applications in laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (250) represents any number of networks that transmit coded video data between terminal devices (210), (220), (230), and (240), including, for example, wired and / or wireless communication networks. Communication network (250) can exchange data via circuit switching and / or packet switching channels. Typical networks include communication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of network (250) may not be important to the operation of this disclosure unless described herein below.

[0025] Figure 3 shows an example of the application of the disclosed subject matter, illustrating the arrangement of a video encoder and video decoder in a streaming environment. The disclosed subject matter may be equally applicable to other video-enabled applications, such as video conferencing, digital TV, and the storage of compressed video on digital media including CDs, DVDs, and memory sticks.

[0026] The streaming system may include a capture subsystem (313) which may include a video source (301), such as a digital camera, that generates a stream of uncompressed video images (302). In one example, the stream of video images (302) may include a sample captured by a digital camera. The stream of video images (302), shown as a thick line to highlight a higher data volume compared to encoded video data (304) (or encoded video bitstream), may be processed by an electronic device (320) which includes a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject, as will be described in more detail below. The encoded video data (304) (or encoded video bitstream (304)), shown as a thin line to highlight a lower data volume compared to the stream of video images (302), may be stored in a streaming server (305) for future use. One or more streaming client subsystems, such as client subsystems (306) and (308) in Figure 3, can access a streaming server (305) to obtain copies (307) and (309) of the encoded video data (304). Client subsystem (306) may include, for example, a video decoder (310) in an electronic device (330). The video decoder (310) decodes the input copy (307) of the encoded video data and creates an output stream of video images (311) that can be rendered on a display (312) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (304), (307), and (309) (e.g., video bitstreams) can be encoded according to specific video coding / compression standards. Examples of these standards include ITU-T Recommendation H.265.For example, a video coding standard under development is informally known as Multipurpose Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0027] It should be noted that electronic devices (320) and (330) may include other components (not shown). For example, electronic device (320) may include a video decoder (not shown), and electronic device (330) may also include a video encoder (not shown).

[0028] Figure 4 shows a block diagram of a video decoder (410) according to one embodiment of the present disclosure. The video decoder (410) may be included in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., a receiving circuit). The video decoder (410) can be used in place of the video decoder (310) in the example of Figure 3.

[0029] The receiver (431) can receive one or more codec video sequences to be decoded by the decoder (410), which in the same or different embodiments may be one coded video sequence at a time, and the decoding of each coded video sequence is independent of other coded video sequences. The coded video sequences may be received from a channel (401), which may be a hardware / software link to a storage device that stores coded video data. The receiver (431) may receive coded video data together with other data that may be transferred to their respective usage entities (not shown), e.g., coded audio data and / or auxiliary data streams. The receiver (431) may isolate coded video sequences from other data. To counteract network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420) (hereinafter "Parser (420)"). In certain applications, the buffer memory (415) is part of the video decoder (410). In other cases, it may be located outside the video decoder (410) (not shown). Still other than that, there may be a buffer memory (not shown) outside the video decoder (410) to counteract network jitter, for example, and another buffer memory (415) inside the video decoder (410) to handle playback timing, for example. The buffer memory (415) may not be necessary or may be small if the receiver (431) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isochronous network. For use in best-effort packet networks such as the Internet, the buffer memory (415) may be necessary, may be relatively large, may be advantageously adaptive in size, and may be at least partially implemented in an operating system or similar element (not shown) outside the video decoder (410).

[0030] The video decoder (410) may include a parser (420) for reconstructing symbols (421) from a coded video sequence. These symbol categories may include information used to manage the operation of the video decoder (410), and potentially information for controlling rendering devices, such as a rendering device (412) (e.g., a display screen) which is not an integral part of the electronic device (430) but can be coupled to the electronic device (430), as shown in Figure 4. The rendering device control information may be in the form of supplemental extension information (SEI messages) or video usability information (VUI) parameter set fragments (not shown). The parser (420) can parse / entropy decode the received coded video sequence. The coding of the coded video sequence may conform to video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, and arithmetic coding with or without context dependency. The parser (420) may extract a set of subgroup parameters from the coded video sequence for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to the group. Subgroups can include groups of images (GOP), images, tiles, slices, macroblocks, coding units (CU), blocks, transformation units (TU), and prediction units (PU). The parser (420) can also extract information from coded video sequence information such as transformation coefficients, quantizer parameter values, and motion vectors.

[0031] The parser (420) can perform entropy decoding / syntax analysis operations on the video sequence received from the buffer memory (415) to create symbols (421).

[0032] The reconstruction of symbol (421) may include a number of different units, depending on the type of coded video image or part thereof (inter-image and intra-image, inter-block and intra-block, etc.) and other factors. Which units are included and how they are included may be controlled by subgroup control information parsed from the coded video sequence by the parser (420). The flow of such subgroup control information between the parser (420) and the following number of units is not shown for clarity.

[0033] In addition to the functional blocks already described, the video decoder (410) can be conceptually subdivided into several functional units, as described below. In actual embodiments operating under commercial constraints, many of these units may interact closely with each other and, at least partially, be integrated with each other. However, the following conceptual subdivision into functional units is appropriate for illustrating the disclosed subject matter.

[0034] The first unit is the scaler / inverse unit (451). The scaler / inverse unit (451) receives control information from the parser (420) as symbols (421), including the quantized transformation coefficients, as well as the transformation to be used, block size, quantization coefficients, and quantization scaling matrix. The scaler / inverse unit (451) can output a block containing sample values ​​that can be input to the aggregator (455).

[0035] In some cases, the output samples of the scaler / inverse transform (451) may relate to intracoded blocks, i.e., blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-image predictive unit (452). In some cases, the intra-image predictive unit (452) generates blocks of the same size and shape as the block being reconstructed, using surrounding already reconstructed information fetched from the current image buffer (458). The current image buffer (458) buffers, for example, a partially reconstructed current image and / or a fully reconstructed current image. The aggregator (455) may, in some cases, add the predictive information generated by the intra-predictive unit (452) to the output sample information provided by the scaler / inverse transform unit (451) on a sample-by-sample basis.

[0036] In other cases, the output samples of the scaler / inverse unit (451) may relate to an intercoded, and possibly motion-compensated, block. In such cases, the motion-compensated prediction unit (453) can access the reference image memory (457) to fetch samples to be used for prediction. After the fetched samples are motion-compensated according to the symbols (421) related to the block, these samples can be added by the aggregator (455) to the output of the scaler / inverse unit (451) (in this case, called residual samples or residual signals) to generate output sample information. The address in the reference image memory (457) from which the motion-compensated prediction unit (453) fetches the prediction samples can be controlled by a motion vector and may be available to the motion-compensated prediction unit (453) in the form of a symbol (421) which may have, for example, X, Y, and reference image components. Motion compensation may also include interpolation of sample values ​​fetched from the reference image memory (457) when the precise motion vectors of the subsamples are used, a motion vector prediction mechanism, and so on.

[0037] The output samples of the aggregator (455) may be subjected to various loop filtering techniques in the loop filter unit (456). The video compression technique may include in-loop filtering techniques that are controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and provided to the loop filter unit (456) as symbols (421) from the parser (420), but may also respond to metadata obtained during decoding of previous (in decoding order) portions of the coded image or coded video sequence, and to previously reconstructed and loop-filtered sample values.

[0038] The output of the loop filter unit (456) can be output to the rendering device (412) and can be a sample stream that can be stored in the reference image memory (457) for use in future inter-image prediction.

[0039] A specific coded image, once fully reconstructed, can be used as a reference image for future predictions. For example, once the coded image corresponding to the current image has been fully reconstructed and the coded image has been identified as a reference image (e.g., by the parser (420)), the current image buffer (458) can become part of the reference image memory (457), and the new current image memory can be reallocated before starting the reconstruction of the next coded image.

[0040] The video decoder (410) can perform decoding operations according to a specified video compression technique in a standard such as ITU-T Rec.H.265. A coded video sequence can conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, a profile can select a particular tool from all the tools available in the video compression technique or standard as the only tool usable under that profile. Compliance also requires that the complexity of the coded video sequence is within the range defined at the level of the video compression technique or standard. In some cases, the level may limit the maximum image size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second), maximum reference image size, etc. Limitations set by the level may, in some cases, be further restricted by the virtual reference decoder (HRD) specification and metadata for HRD buffer management notified in the coded video sequence.

[0041] In one embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (410) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, a temporal, spatial, or signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant images, forward error correction codes, etc.

[0042] Figure 5 shows a block diagram of a video encoder (503) according to one embodiment of the present disclosure. The video encoder (503) is included in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmitting circuit). The video encoder (503) can be used in place of the video encoder (303) in the example of Figure 3.

[0043] The video encoder (503) can receive video samples from a video source (501) (not part of the electronic device (520) in the example in Figure 5) which can capture video images to be coded by the video encoder (503). In another example, the video source (501) is part of the electronic device (520).

[0044] The video source (501) can provide a source video sequence coded by a video encoder (503) in the form of a digital video sample stream, which can be any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (501) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (501) may be a camera that captures local image information as a video sequence. The video data may be provided as a series of separate images that give motion when viewed in sequence. The images themselves can be organized as a spatial array of pixels, and each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description will focus on samples.

[0045] According to one embodiment, the video encoder (503) can encode images from a source video sequence in real time or under any other time constraints as needed depending on the application, and compress them into a coded video sequence (543). Enforcing an appropriate coding speed is one of the functions of the controller (550). In some embodiments, the controller (550) controls and is functionally coupled to other functional units, as described below. For clarity, coupling is not depicted. Parameters set by the controller (550) may include parameters related to rate control (such as image skips, quantizers, lambda values ​​for rate distortion optimization techniques), image size, group of images (GOP) layout, and maximum motion vector search range. The controller (550) may be configured to have other appropriate functions for the video encoder (503) optimized for a particular system design.

[0046] In some embodiments, the video encoder (503) is configured to operate in a coding loop. For an overly simplified explanation, in one example, the coding loop may include a source coder (530) (responsible for generating symbols, such as a symbol stream, based on, for example, an input image to be coded and a reference image) and a (local) decoder (533) built into the video encoder (503). The decoder (533) reconstructs the symbols to create sample data in a similar manner to how a (remote) decoder would also create it (since any compression between the symbols and the coded video bitstream is reversible in the video compression techniques considered in the disclosed subject). The reconstructed sample stream (sample data) is input to a reference image memory (534). The decoding of the symbol stream yields bit-accurate results regardless of the decoder's location (local or remote), so the contents of the reference image memory (534) are also bit-accurate between the local encoder and the remote encoder. In other words, the predictive part of the encoder "sees" the exact same sample values ​​as the reference image samples that the decoder would "see" if predictions were used during decoding. This fundamental principle of reference image synchronization (and the drift that occurs when synchronization cannot be maintained due to, for example, channel errors) is also used in several related techniques.

[0047] The operation of the “local” decoder (533) may be the same as that of a “remote” decoder, such as the video decoder (410), which has already been described in detail in relation to Figure 4. However, as also briefly referring to Figure 4, since symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (545) and parser (420) may be reversible, the entropy decoding portion of the video decoder (410), including the buffer memory (415), and the parser (420) may not be fully implemented in the local decoder (533).

[0048] An observation that can be made at this point is that any decoder techniques other than syntax analysis / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject focuses on decoder operation. The description of encoder techniques can be omitted as it is the inverse of the comprehensively described decoder techniques. More detailed explanations are necessary only in specific areas and are provided below.

[0049] During operation, in some examples, the source coder (530) may perform motion-compensated predictive coding, predictively coding the input picture by referencing one or more previously coded images from a video sequence designated as “reference images”. In this way, the coding engine (532) codes the difference between the pixel blocks of the input image and the pixel blocks of the reference images that may be selected as predictive references for the input image.

[0050] The local video decoder (533) can decode coded video data of an image that may be designated as a reference image, based on symbols created by the source coder (530). The operation of the coding engine (532) may, advantageously, be a lossy process. If coded video data can be decoded by a video decoder (not shown in Figure 5), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (533) can replicate the decoding process that may be performed by the video decoder on the reference image and have the reconstructed reference image stored in the reference image cache (534). In this way, the video encoder (503) can locally store a copy of the reconstructed reference image having common content as a reconstructed reference image (without transmission errors) obtained by the far-end video decoder.

[0051] The predictor (535) can perform a predictive search of the coding engine (532). That is, for a new image to be coded, the predictor (535) can search the reference image memory (534) for specific metadata such as motion vectors and block shapes of sample data or reference images (as candidate reference pixel blocks), which can serve as appropriate predictive references for the new image. The predictor (535) can operate on a sample block-by-pixel block basis to find appropriate predictive references. In some cases, the input image may have predictive references drawn from a number of reference images stored in the reference image memory (534), as determined by the search results obtained by the predictor (535).

[0052] The controller (550) can manage the coding operations of the source coder (530), including, for example, setting parameters and subgroup parameters used to encode video data.

[0053] The outputs of all the aforementioned functional units can undergo entropy coding by the entropy coder (545). The entropy coder (545) converts the symbols generated by the various functional units into coded video sequences by lossless compression of the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0054] The transmitter (540) can buffer the coded video sequence created by the entropy coder (545) and prepare it for transmission via the communication channel (560), which may be a hardware / software link to a storage device that will store the coded video data. The transmitter (540) can merge the coded video data from the video coder (503) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0055] The controller (550) can manage the operation of the video encoder (503). During coding, the controller (550) can assign a specific coded image type to each coded image, which can influence the coding technique that may be applied to each image. For example, an image may often be assigned as one of the following image types:

[0056] An intra-image (I-image) may be one that can be coded and decoded without using any other images in the sequence as a source of prediction. Some video codecs enable different types of intra-images, including, for example, independent decoder refresh ("IDR") images. Those skilled in the art are familiar with these variations of I-images and their respective uses and characteristics.

[0057] The predicted image (P-image) may be coded and decoded using intra-prediction or inter-prediction, which uses up to one motion vector and reference index to predict the sample values ​​for each block.

[0058] Bidirectional prediction images (B images) may be coded and decoded using intra-prediction or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​for each block. Similarly, multiple prediction images may use three or more reference images and associated metadata to reconstruct a single block.

[0059] The source image is typically divided spatially into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block is coded. Blocks may be coded predictively by referencing other (already coded) blocks, as determined by the coding assignment applied to each image in the block. For example, blocks in image I may be coded non-predictively, or they may be coded predictively by referencing already coded blocks (spatial or intra-predictive) of the same image. Pixel blocks in image P may be coded predictively via spatial prediction or via temporal prediction referencing one previously coded reference image. Blocks in image B may be coded predictively via spatial prediction or via temporal prediction referencing one or two previously coded reference images.

[0060] The video encoder (503) can perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec.H.265. In these operations, the video encoder (503) can perform various compression operations, including predictive coding operations that leverage temporal and spatial redundancy in the input video sequence. Thus, the coded video data can conform to the syntax specified by the video coding technique or standard being used.

[0061] In one embodiment, the transmitter (540) may transmit additional data along with the encoded video. The source coder (530) may include such data as part of the coded video sequence. The additional data may include time / space / SNR extension layers, other forms of redundant data such as redundant images and slices, SEI messages, VUI parameter set fragments, and the like.

[0062] Video may be captured in time series as multiple source images (video images). Intra-image prediction (often abbreviated as intra-prediction) utilizes spatial correlations in a given image, while inter-image prediction utilizes (temporal or other) correlations between images. In one example, a particular image being coded / decoded, called the current image, is divided into blocks. When a block in the current image is analogous to a reference block in a previously coded and still-buffered reference image in the video, the block in the current image can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference image and may have a third dimension to identify the reference image if multiple reference images are used.

[0063] In some embodiments, a dual prediction technique can be used for inter-image prediction. According to the dual prediction technique, two reference images are used, such as a first reference image and a second reference image, both of which are prior to the decoding order of the current image in the video (however, the display order may be past and future, respectively). Blocks in the current image can be coded by a first motion vector pointing to a first reference block in the first reference image, and a second motion vector pointing to a second reference block in the second reference image. Blocks can be predicted by combinations of the first and second reference blocks.

[0064] Furthermore, to improve coding efficiency, merge mode techniques can be used for inter-image prediction.

[0065] According to some embodiments of this disclosure, predictions such as inter-image predictions and intra-image predictions are performed in block units. For example, according to the HEVC standard, images in a sequence of video images are divided into coding tree units (CTUs) for compression, and the CTUs in an image have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU contains three coding tree blocks (CTBs), which are one luminance CTB and two saturation CTBs. Each CTU can be recursively quadtree-partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, or four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, such as inter-prediction type or intra-prediction type. The CUs are divided into one or more prediction units (PUs) depending on their temporal and / or spatial predictability. Generally, each PU includes a luminance prediction block (PB) and two saturation PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. Using a luminance prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luminance values) such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0066] Figure 6 shows a diagram of a video encoder (603) according to another embodiment of the present disclosure. The video encoder (603) is configured to receive a processing block (e.g., a prediction block) of sample values ​​in the current video image within a sequence of video images, and to encode the processing block into a coded image which is part of a coded video sequence. In one example, the video encoder (603) is used instead of the video encoder (303) in the example of Figure 3.

[0067] In the HEVC example, the video encoder (603) receives a matrix of sample values ​​for a processing block, such as an 8x8 sample prediction block. The video encoder (603) determines whether the processing block is best coded using intra-mode, inter-mode, or bi-prediction mode, for example, using rate-distortion optimization. If the processing block is coded in intra-mode, the video encoder (603) can use the intra-prediction technique to encode the processing block into a coded image; if the processing block should be coded in inter-mode or bi-prediction mode, the video encoder (603) can use the inter-prediction technique or the bi-prediction technique, respectively, to encode the processing block into a coded image. In certain video coding techniques, the merge mode may be an inter-image prediction submode in which the motion vector is derived from one or more motion vector predictors without benefiting from the coded motion vector component outside the predictor. In certain other video coding techniques, there may be motion vector components applicable to the target block. In one example, the video encoder (603) includes other components, such as a mode determination module (not shown) for determining the mode of the processing block.

[0068] In the example shown in Figure 6, the video encoder (603) includes an interencoder (630), an intraencoder (622), a residual calculator (623), a switch (626), a residual encoder (624), a general-purpose controller (621), and an entropy encoder (625), all coupled together as shown in Figure 6.

[0069] The interencoder (630) is configured to receive a sample of the current block (e.g., a processing block), compare that block with one or more reference blocks in a reference image (e.g., blocks in the previous and subsequent images), generate interprediction information (e.g., a description of redundant information by intercoding technique, motion vectors, and merge mode information), and compute an interprediction result (e.g., a predicted block) based on the interprediction information using any appropriate technique. In some examples, the reference image is a decoded reference image decoded based on encoded video information.

[0070] The intra encoder (622) is configured to receive a sample of the current block (e.g., a processing block), possibly compare the block to a block already coded in the same image, generate quantization coefficients after transformation, and possibly also generate intra prediction information (e.g., intra prediction direction information by one or more intra coding techniques). In one example, the intra encoder (622) calculates an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same image.

[0071] The general-purpose controller (621) is configured to determine general-purpose control data and to control other components of the video encoder (603) based on the general-purpose control data. In one example, the general-purpose controller (621) determines the mode of a block and provides control signals to the switch (626) based on the mode. For example, if the mode is intra-mode, the general-purpose controller (621) controls the switch (626) to select the intra-mode result to be used by the residual calculator (623) and controls the entropy encoder (625) to select the intra-prediction information to be included in the bitstream. If the mode is inter-mode, the general-purpose controller (621) controls the switch (626) to select the inter-prediction result to be used by the residual calculator (623) and controls the entropy encoder (625) to select the inter-prediction information to be included in the bitstream.

[0072] The residual calculator (623) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra-encoder (622) or inter-encoder (630). The residual encoder (624) is configured to operate on the residual data to encode the residual data in order to generate conversion coefficients. In one example, the residual encoder (624) is configured to convert the residual data from the spatial domain to the frequency domain and generate conversion coefficients. The conversion coefficients are then subjected to a quantization process to obtain quantized conversion coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be suitably used in the intra-encoder (622) and inter-encoder (630). For example, an interencoder (630) can generate a decoded block based on decoded residual data and interprediction information, and an intraencoder (622) can generate a decoded block based on decoded residual data and intraprediction information. In some examples, the decoded block is appropriately processed to generate a decoded image, which can be buffered in a memory circuit (not shown) and used as a reference image.

[0073] The entropy encoder (625) is configured to format the bitstream to include encoded blocks. The entropy encoder (625) is configured to include various information according to an appropriate standard, such as the HEVC standard. For example, the entropy encoder (625) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. Note that, according to the disclosed subject, residual information is not present when coding blocks in either inter-mode or bi-prediction mode merge submodes.

[0074] Figure 7 shows a diagram of a video decoder (710) according to another embodiment of the present disclosure. The video decoder (710) is configured to receive a coded image which is part of a coded video sequence, and to decode the coded image to produce a reconstructed image. In one example, the video decoder (710) is used instead of the video decoder (310) in the example of Figure 3.

[0075] In the example shown in Figure 7, the video decoder (710) includes an entropy decoder (771), an interdecoder (780), a residual decoder (773), a reconstruction module (774), and an intradecoder (772) coupled together as shown in Figure 7.

[0076] The entropy decoder (771) may be configured to reconstruct specific symbols from the coded image that represent the syntax elements that make up the coded image. Such symbols may include, for example, the mode in which the blocks are coded (e.g., intra-mode, inter-mode, bi-prediction mode, etc., the latter two of which are merge submodes or other submodes), prediction information (e.g., intra-prediction information or inter-prediction information, etc.) that can identify specific samples or metadata used for prediction by the intra-decoder (772) or inter-decoder (780), respectively, as well as residual information, for example, in the form of quantization transformation coefficients. In one example, if the prediction mode is inter-prediction mode or bi-prediction mode, the inter-prediction information is provided to the inter-decoder (780), and if the prediction type is intra-prediction type, the intra-prediction information is provided to the intra-decoder (772). The residual information may undergo inverse quantization and be provided to the residual decoder (773).

[0077] The interdecoder (780) is configured to receive interprediction information and generate interprediction results based on the interprediction information.

[0078] The intra decoder (772) is configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0079] The residual decoder (773) is configured to perform inverse quantization to extract the inversely quantized transformation coefficients, and to process the inversely quantized transformation coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require certain control information (including quantizer parameters (QP)), which may be provided by the entropy decoder (771) (data paths not shown here may only contain low-volume control information).

[0080] The reconstruction module (774) is configured to combine the residuals (as output from the residual decoder (773)) and the prediction results (possibly as output from the inter or intra prediction module) in the spatial domain to form reconstructed blocks that may be part of the reconstructed image, and these reconstructed blocks may be part of the reconstructed video. Note that other appropriate operations, such as deblocking operations, may be performed to improve visual quality.

[0081] It should be noted that the video encoders (303), (503), and (603), as well as the video decoders (310), (410), and (710), can be implemented using any suitable technique. In one embodiment, the video encoders (303), (503), and (603), as well as the video decoders (310), (410), and (710), can be implemented using one or more integrated circuits. In another embodiment, the video encoders (303), (503), and (503), as well as the video decoders (310), (410), and (710), can be implemented using one or more processors that execute software instructions.

[0082] II. Conversion Coding Techniques and Related Techniques 1. Example of DCT-2 linear transformation In some embodiments, 4-point, 8-point, 16-point, and 32-point DCT-2 transforms are used as linear transforms. Figures 8A to 8D show the transform core matrices for 4-point, 8-point, 16-point, and 32-point DCT-2, respectively. The elements of these transform core matrices can be represented using 8-bit integers, and therefore these transform core matrices are called 8-bit transform cores. As shown, the transform core matrices of smaller DCT-2 are part of the transform core matrices of larger DCT-2.

[0083] The DCT-2 core matrix exhibits symmetry / antisymmetry properties. Therefore, it can support a so-called "partial butterfly" implementation to reduce the number of operations (multiplication, addition / subtraction, shift). Using a partial butterfly implementation allows for the same results of matrix multiplication compared to not using a partial butterfly implementation.

[0084] 2. Example of subblock transformation coding 2.1 Subblock Transformation (SBT) In some embodiments, a subblock transformation (SBT), also known as spatial variation transformation (SVT), is used. In some embodiments, the SBT is applied to interpredictive residuals. For example, a coding block can be divided into subblocks, and only a portion of the subblock is processed in the residual block. For the remainder of the subblock, zero residuals are assumed. Therefore, the residual block is smaller than the coding block, and the transformation size in the SBT is smaller than the coding block size. No transformation is performed for regions not covered by the residual block.

[0085] Figures 9A–9D show the subblock types (SVT-H, SVT-V) supported by SBT (e.g., divided vertically or horizontally), size, and location (e.g., left half, left quarter, right half, right quarter, upper half, upper quarter, lower half, lower quarter). Shaded areas labeled with the letter "A" are residual blocks that are transformed and coded, while other areas are assumed to be zero residuals without transformation.

[0086] 2.2. Intra-subpartition (ISP) coding mode In some embodiments, an intra-subpartition (ISP) coding mode is used. In ISP coding mode, a luminance intra-prediction block can be divided vertically or horizontally into two or four subpartitions. The number of subpartitions can depend on the size of the block. Figure 10 shows the number of subpartitions depending on the block size. Figure 11 shows a scenario in which the block is divided into two subpartitions. Figure 12 shows a scenario in which the block is divided into four subpartitions. In one example, all subpartitions satisfy the condition that they have at least 16 samples. In one example, ISP is not applied to the chroma component.

[0087] In one example, for each subpartition separated from the coding block, a residual signal is generated by entropy decoding each coefficient transmitted from the encoder, and then inverse quantization and inverse transformation of them. Next, the first subpartition of the subpartitions is intra-predicted to generate a prediction signal. The prediction signal is added to each residual signal of the first subpartition to obtain the corresponding reconstructed sample. Subsequently, the reconstructed sample values ​​of the first subpartition may be available to generate a prediction for the second subpartition. This process can be repeated for each subpartition until all subpartitions from the coding block have been reconstructed. In one example, all subpartitions share the same intra-mode.

[0088] In one embodiment, the ISP coding mode is tested only in intra mode, which is part of the Most Probable Mode (MPM) list. Therefore, if a block uses an ISP, the MPM flag can be inferred to be 1. In addition, when an ISP is used for a particular block, the respective MPM lists are modified to exclude DC mode and to prioritize horizontal intra mode for horizontal ISP splitting and vertical intra mode for vertical ISP splitting.

[0089] In ISP coding mode, conversion and reconstruction are performed individually for each subpartition, so each subpartition can be considered a subTU.

[0090] 3. Transform Coding with Extended DCT-2 Transform and Multiple Transform Selection (MTS) In some embodiments, if both the height and width of the coding block are 64 samples or less, the transformation size is always the same as the coding block size. If either the height or width of the coding block is greater than 64 samples, when performing a transformation or intra-prediction, the coding block is further divided into multiple sub-blocks, each sub-block having a width and height of 64 or less, and the transformation process is performed for each sub-block.

[0091] 3.1 Conversion Coding Using Extended DCT-2 Conversion In some embodiments, in addition to the 4-point, 8-point, 16-point, and 32-point DCT-2 transforms described above, 2-point and 64-point DCT-2 transforms can be used. Figures 13A to 13E show the 64x64 transform core matrix for the 64-point DCT-2 transform.

[0092] 3.2 Explicitly Signaled Transformations In some embodiments, in addition to DCT-2 and 4x4 DST-7 transform coding, multiple transform selection (MTS) (also known as extended multiple transform (EMT) or adaptive multiple transform (AMT)) can be used for residual coding of both intercoded and intracoded blocks. MTS uses multiple selected transforms from the discrete cosine transform (DCT) / discrete sine transform (DST) family other than DCT-2 and 4x4 DST-7 transforms. The selection is performed by the encoder and may be explicitly signaled from the encoder to the decoder. For example, the selected transforms may include DST-7 or DCT-8 transforms. Figure 14 shows the transform basis functions for DST / DCT transforms. In some embodiments, the DST / DCT transform core matrix used in MTS is represented in an 8-bit representation.

[0093] In some embodiments, MTS can be applied to CUs having both width and height of 32 samples or less. Whether or not to apply MTS can be controlled by a flag represented by mts_flag. For example, if mts_flag is equal to 0, only DCT-2 is applied to the coding of residual blocks. If mts_flag is equal to 1, indicating that MTS is applied, the selected transformation can be used. For example, two bins can be used to specify the horizontal and vertical transformations to be used, and the index indicated by mts_idx can be further signaled.

[0094] Figure 15 shows Table (1500), which illustrates the mapping relationship between the mts_idx value and the respective horizontal or vertical transformations. Row (1301) where the mts_idx value is -1 corresponds to a scenario where mts_flag is 0 (indicating that MTS is not applied), and the DCT-2 transformation is used. Rows (1302) to (1305) with mts_idx values ​​of 0, 1, 2, or 3 correspond to a scenario where mts_flag is equal to 1 (indicating that MTS is applied). In the two rightmost columns of Table (1500), 0 represents the DCT-2 transformation type, 1 represents the DST-7 transformation type, and 2 represents the DCT-8 transformation type.

[0095] Figures 16A to 16D show the conversion core matrices for the DST-7 conversion type. Figures 17A to 17D show the conversion core matrices for the DCT-8 conversion type.

[0096] In some embodiments, MTS can be enabled or disabled using high-level syntax (HLS) elements. Each HLS element may be a video parameter set (VPS) syntax element, a sequence parameter set (SPS) syntax element, an image parameter set (PPS) syntax element, a slice header syntax element, a tile header syntax element, or a tile group header syntax element. Figure 18 shows an example of controlling the use of MTS using an SPS syntax element. As shown, the SPS syntax element sps_mts_enabled_flag can signal whether MTS is enabled for a video sequence. When MTS is enabled, two syntax elements, sps_explicit_mts_intra_enabled_flag and sps_explicit_mts_inter_enabled_flag, can signal whether MTS is enabled for coding inter-prediction blocks or intra-prediction blocks, respectively.

[0097] In one embodiment, if the signaling-based MTS (referred to as an explicit MTS) described above is not used, an implicit MTS is applied. In an implicit MTS, instead of being based on signaling, transformation selection can be made according to the block width and height. For example, using an implicit MTS, a DST-7 transformation can be selected for the shorter side of a transformation block, and a DCT-2 transformation can be selected for the longer side of the transformation block.

[0098] 3.3 Scenarios where implicit conversion selection is applied In various embodiments, in certain scenarios, DST-7 and / or DCT-8 can be used without explicit signaling. For example, DST-7 and / or DCT-8 can be used implicitly based on information available to both the encoder and the corresponding decoder. These scenarios include:

[0099] 3.3.1 Intra-subpartitioning (ISP) For residual blocks coded in ISP mode, the horizontal transformation is selected as DST-7 as long as the block width of the residual block is between 4 and 16, and the vertical transformation is selected as DST-7 as long as the block height of the residual block is between 4 and 16.

[0100] 3.3.2 Subblock Conversion (SBT) In SBT mode, for sub-TUs located in the left half (or quarter) or right half (or quarter) of the current CU, the horizontal transformer can be DCT-8 or DST-7, respectively. Otherwise (the sub-TU has the same width as the current CU), DCT-2 can be used. For sub-TUs located in the upper half (or quarter) or lower half (or quarter) of the current CU, the vertical transformer can be DCT-8 or DST-7, respectively. Otherwise (the sub-TU has the same height as the current CU), DCT-2 can be used.

[0101] 3.3.3 MTS disabled by HLS element For example, if sps_mts_enabled_flag is signaled as true, but both sps_explicit_mts_intra_enabled_flag and sps_explicit_mts_inter_enabled_flag are signaled as false, then for intra-predicted residuals, if the width of each block is between 4 and 16, the horizontal transformation can be selected as DST-7, and if the height of each block is between 4 and 16, the vertical transformation can be selected as DST-7.

[0102] 4. Non-separable quadratic transform (NSST) 4.1 Initial Design of NSST In some embodiments, a mode-dependent unseparated quadratic transform (NSST) can be applied between the forward core transform and quantization (in the encoder) and between the inverse quantization and inverse core transform (in the corresponding decoder). To maintain low complexity, in some embodiments, the NSST can be applied only to the low-frequency coefficients after the linear transform. If both the width (W) and height (H) of the transform coefficient block are 8 or greater, an 8×8 unseparated quadratic transform can be applied to the 8×8 region in the upper left of the transform coefficient block. Otherwise, if either W or H of the transform coefficient block is equal to 4, a 4×4 unseparated quadratic transform can be applied, and the 4×4 unseparated quadratic transform is performed on the min(8,W)×min(8,H) region in the upper left of the transform coefficient block. The above transform selection rules apply to both the luminance and chroma components.

[0103] The implementation of matrix multiplication for inseparable transformations is explained below using a 4x4 input block as an example. To apply the inseparable transformation, the 4x4 input block X

number

number

number

[0104] The non-separable transformation is calculated as follows:

number

number

number

[0105] In one example of NSST, there can be a total of 35 × 3 non-separable quadratic transforms for both 4 × 4 and 8 × 8 block sizes, where 35 is the number of transform sets corresponding to each intra-prediction mode, and 3 is the number of NSST candidates for each intra-prediction mode. The mapping from intra-prediction modes to transform sets is defined in Table (1900) shown in Figure 19. For example, the transform set applied to the luminance / saturation transform coefficient can be specified by the corresponding luminance / saturation intra-prediction mode according to Table (1900). For intra-prediction modes greater than 34 (e.g., diagonal prediction direction), the transform coefficient blocks are transposed before / after the quadratic transform in the encoder / decoder.

[0106] For each transformation set, the selected non-separable quadratic transformation candidate is further specified by an explicitly signaled CU-level NSST index. The index is signaled once in the bitstream for each intra-CU after the transformation coefficients and truncated unary binarization have been used. The truncation value is 2 for planar or DC mode, and 3 for angular intra-predictive mode. This NSST index is signaled only if the CU has multiple non-zero coefficients. If not signaled, the default value is 0. A value of zero for this syntax element indicates that no quadratic transformation is applied to the current CU, while values ​​1-3 indicate which quadratic transformation from the set should be applied.

[0107] NSST does not have to be applied to blocks coded in transform-skip mode. When an NSST index is signaled for a CU and is not equal to zero, NSST is not used for blocks of components coded in transform-skip mode within a CU. If a CU containing blocks of all components is coded in transform-skip mode, or if the number of non-zero coefficients in a non-transform-skip mode CB is less than two, the NSST index is not signaled for the CU.

[0108] 4.2 Reduction Size Conversion (RST) In some embodiments, a variation of NSST called a reduced-size transformation (RST) or low-frequency unseparated quadratic transformation (LFNST) is used. RST uses a transformation zero-out scheme. Whether the intra-prediction mode is planar or DC is checked to entropy code the transformation index of the NSST.

[0109] In one example, four transformation sets are applied, each containing three RST transformation cores. The RST transformation cores can be 16×48 (or 16×64) (applied to transformation factor blocks where both height and width are 8 or greater) or 16×16 (applied to transformation factor blocks where either height or width is 4). For convenience of notation, the 16×48 (or 16×64) transformation is shown as RST 8×8, and the 16×16 transformation is shown as RST 4×4.

[0110] For RST 8×8, two alternative translation coding processes, (2000) and (2100), using 16×64 and 16×48 translation cores respectively, are shown in Figures 20 and 21. VVC Draft 5 employs the one using 16×48 translation cores.

[0111] In the example process (2000) in Figure 20, the encoder can first perform a forward linear transformation (2010) on the residual block, and then perform a forward quadratic transformation (2012) on the coefficients generated from the forward linear transformation (2010). In the forward quadratic transformation (2012), the 64 coefficients of the 4x4 subblocks A / B / C / D in the upper left corner of the coefficient block (2013) are represented by 64 length vectors, which are multiplied by a 16x64 transformation matrix according to equation (3) to obtain a vector of length 16. The elements in the 16 length vector are backfilled into the 4x4 subblock A in the upper left of the coefficient block (2013). The coefficients in subblocks B / C / D can take the value of zero. The coefficients obtained after the forward quadratic transformation 2012 are then quantized in step (2014) and entropy coded to generate coded bits in the bitstream (2016).

[0112] The coded bits are received by the decoder, entropy decoded, and then inversely quantized (2024) to generate a coefficient block (2023). An inverse quadratic transform (2022) can be performed over the 16 coefficients in the upper left 4x4 subblock E to obtain 64 coefficients that are then filled back into the 4x4 subblocks E / F / G / H. Subsequently, the coefficients in the block (2023) after the inverse quadratic transform (2022) can be processed by an inverse linear transform (2020) to obtain the restored residual block.

[0113] Process (2100) in the example in Figure 21 is similar to process (2000), except that fewer coefficients (48) are processed during the forward quadratic transformation (2012). Specifically, the 48 coefficients in subblocks A / B / C are processed by a smaller transformation matrix of size 16 × 48. By using a smaller transformation matrix, the memory size required to store the transformation matrix and the complexity of each calculation can be reduced.

[0114] FIG. 22 shows an exemplary CU level syntax table (2200), and at the end of the CU level syntax, a syntax element lfnst_idx indicating the selection of the LFNST kernel is signaled.

[0115] 4.3 Example of RST Calculation The main concept of the reduction transform (RT) is to map an N-dimensional vector to an R-dimensional vector in a different space, where R / N (R < N) is the reduction factor.

[0116] The RST matrix is an R×N matrix as follows.

Equation

[0117] FIG. 23 shows the process of reduction transform (2301) and the process of reduction inverse transform (2302). T represents an RST transform matrix having dimensions of RxN, and T T represents the transpose matrix of T having dimensions of NxR.

[0118] In RST 8×8, a reduction ratio of 4 (1 / 4 size) is achieved. For example, instead of a 64×64 which is the size of a conventional 8×8 non-separable transform matrix, a 16×64 direct matrix is used. The 64×16 inverse RST matrix is used on the decoder side to generate core (primary) transform coefficients in the upper left region of an 8×8. The forward RST 8×8 uses 16×64 (or 8×64 in the case of an 8×8 block) matrices such that non-zero coefficients are generated only in the upper left 4×4 region within a given 8×8 region. In other words, when RST is applied, the 8×8 region excluding the upper left 4×4 region will have only zero coefficients. In the case of RST 4×4, 16×16 (or 8×16 in the case of a 4×4 block) direct matrix multiplications can be applied.

[0119] In addition, in the case of RST 8×8, to further reduce the transformation matrix size, instead of using the entire upper-left 8×8 coefficient of the residual block (2410) (shaded subblock in Figure 24A) as input to compute the quadratic transformation, the three upper-left 4×4 subblock coefficients of the residual block (2410) (shaded subblocks in Figure 24B) are used as input to compute the quadratic transformation.

[0120] For example, the inverse RST is conditionally applied if the following two conditions are met: (i) each block size is greater than or equal to a given threshold (W>=4 &&H>=4), and (ii) the conversion skip mode flag is equal to 0. For example, if both the width (W) and height (H) of the conversion coefficient block are greater than 4, the RST 8x8 is applied to the top-left 8x8 region of the conversion coefficient block. Otherwise, the RST 4x4 is applied to the top-left min(8,W) x min(8,H) region of the conversion coefficient block.

[0121] In one example, when the RST index is equal to 0, RST is not applied. Otherwise, RST is applied, and the kernel is selected using the RST index. In one example, RST is applied to intraCUs in both intra and interslice, as well as to both luminance and saturation. When dual-tree is enabled, the RST indices for luminance and saturation are signaled separately. In the case of interslice (where dual-tree is disabled), a single RST index is signaled and used for both luminance and saturation. When ISP mode is selected, RST is disabled, and the RST index is not signaled.

[0122] 4.4 Example of RST transformation matrix selection In one example, the RST matrix can be selected from four transformation sets, each consisting of two transformations. The choice of which transformation set to apply can be determined based on the applied intraprediction mode, as follows: Transformation set 0 can be selected when one of the three cross-component linear model (CCLM) modes is indicated. Otherwise, the transformation set selection can be performed according to Table (2500) shown in Figure 25. The index for accessing Table (2500), denoted as IntraPredMode, can be, for example, the transformed mode index used for wide-angle intraprediction, in the range of [-14, 83].

[0123] 5. Matrix-based intra-prediction (MIP) mode In some embodiments, a matrix-based intra-prediction (MIP) mode is used. Figure 26 shows an exemplary process (2600) in MIP mode. To predict a sample of a rectangular block (2610) of width W and height H, MIP takes as input one line of H reconstructed adjacent boundary samples to the left of the block (2610) and one line of W reconstructed adjacent boundary samples above the block (2610). If reconstructed samples are not available, they can be generated in a manner similar to conventional intra-prediction.

[0124] The generation of the prediction signal can be based on the following three steps (2601) to (2603). In step (2601), four boundary samples are extracted if W=H=4, and eight samples otherwise, by averaging them.

[0125] In step (2602), matrix-vector multiplication A k ·bdry red Next, offset b k The sum of the averaged samples bdry red The process is executed with input A. The result is a reduced predicted signal on a subsampled set of samples (2621) within the original block. Matrix Ak and offset b k can be selected based on the MIP mode index k

[0126] In step (2603), the predicted signal at the remaining positions (2622) is generated from the predicted signal on the set (2621) subsampled by linear interpolation, which is single-step linear interpolation in each direction

[0127] Matrix A required to generate the predicted signal k and offset vector b k can be obtained from three sets of matrices S0, S1, S2. Set S0 consists of 18 matrices each having 16 rows and 4 columns

Number

Number

Number

Number

Number

Number

[0128] As illustrated, given an 8x8 block (2610), MIP takes the average of four along each axis of the boundary. The resulting eight input samples are then subjected to matrix-vector multiplication. The matrix is ​​taken from set S1. This yields 16 samples (2621) at odd positions in the prediction block. Thus, a total of (816) / (88) = 2 multiplications are performed per sample. After adding the offset, these samples are interpolated vertically using the reduced upper boundary samples. Horizontal interpolation follows by using the original left boundary samples. In this case, the interpolation process does not require multiplication.

[0129] 5.1 Signaling in MIP Mode In some embodiments, for each coding unit (CU) in intra-mode, a bitstream flag can be sent indicating whether or not the MIP mode is applied to the corresponding prediction unit (PU). If the MIP mode is applied, the MIP mode index predmode is signaled using an MPM list containing three MPMs.

[0130] The MPM can be derived using the above and the intra mode of the PU on the left as follows: Three fixed mapping tables map_angular_to_mip idx There are ,idx∈{0,1,2}, and each table is for each conventional intra predictive mode predmode as shown in the following formula. Angular Associate it with a specific MIP mode. preddrive MIP =map_angular_to_mip[predmode Angular ] (Formula 5) Here, `map_angular_to_mip` is a fixed lookup table. The index of the mapping table is determined based on the width W and height H of the PU, and a total of three indexes are available, as described below. idx(PU)=idx(W,H)∈{0,1,2} (Equation 6) This indicates which of the three sets the MIP parameters should be obtained from above.

[0131] In some examples, the above MIP mode, i.e., generates an MPM list of the current block coded by the MIP mode.

number

number

[0132]

number

number

number

number

number

number

[0133]

number

number

[0134] Finally, the derived

number

number

number

number

[0135] As an example, Figure 27 shows a CU level syntax table in which the flag signaling MIP mode is indicated in frame (2701).

[0136] 5.2 Derivation of MPM list for conventional intra-predictive mode In some embodiments, the MIP mode is harmonized with the MPM-based coding of the conventional intra-predictive mode as follows: The conventional intra-predictive mode luminance and saturation MPM list derivation process is used for the MIP mode predmode MIP A separate fixed table map_mip_to_angular maps to one of the conventional intra prediction modes. idx Using idx∈{0,1,2}, preddrive Angular =map_mip_to_angular[predmode MIP ] (Formula 11) Here, map_mip_to_angular is a fixed lookup table. In the case of luminance MPM list derivation, adjacent luminance blocks are MIP mode predmode MIP Whenever this block is coded by the conventional intra predictive mode, it is predmode. Angular It is treated as if it were using [a specific method]. In the case of saturation MPM list derivation, whenever the current luminance block uses MIP mode, the same mapping is used to convert the MIP mode to the conventional intra predictive mode.

[0137] III. Enabling implicit transformation selection based on high-level syntax elements or block-level syntax elements In some embodiments, two transformation coding schemes can be used: implicit transformation (or implicit transformation selection) and explicit transformation (or explicit transformation selection).

[0138] Implicit transformations allow selection of a group of non-DCT 2 transformations (e.g., DST-1, DCT-5, DST-7, DCT-8, DST-4, DCT-4) without transformation index signaling. For example, a group of non-DCT 2 transformations can be selected using pre-coded information available to both the encoder and the corresponding decoder. This pre-coded information may include, but is not limited to, intra-prediction modes (e.g., planar mode, DC mode, angular mode), block size, block width, block height, block aspect ratio, block area size, intra-coding modes (e.g., whether multiple reference lines (MRL), ISP, MIP are used), the positions of selected spatial merge candidates (e.g., top merge candidate, left merge candidate), and inter-prediction modes (e.g., position-dependent prediction synthesis (PDPC) mode, combined inter-intra-prediction (CIIP) mode).

[0139] In contrast, explicit conversions allow you to select one conversion from a group of candidate conversion types (such as DCT-2, DST-1, DCT-5, DST-7, DCT-8, DST-4, DCT-4) using a signaled index to indicate which conversion type is selected.

[0140] 1. Enable implicit conversion when explicit MTS is disabled. In some embodiments, as described in Section II.3.2, implicit transformation selection may be enabled for intra-predictive residual blocks that are not coded by the ISP, if explicit MTS is disabled for both intra-predictive residual blocks and inter-predictive residual blocks, as indicated by the High-Level Syntax (HLS) element. For example, if both the SPS syntax elements, sps_explicit_mts_intra_enabled_flag and sps_explicit_mts_inter_enabled_flag are 0, the decoder may decide to enable implicit transformation selection for the image or block associated with the SPS syntax element.

[0141] As an example, Figures 28A and 28B show a combination of text (2800) that specifies a transformation coding process that performs explicit or implicit transformation selection of the current block based on relevant syntax elements received from a bitstream. Text (2800) can be used as part of a video coding standard. Figure 28A shows two sections (2801) and (2802), while Figure 28B shows one section (2803) as well as two tables (2804) and (2805).

[0142] Section (2801) describes the inputs and outputs of the transformation coding process. Specifically, the inputs are the current block's position, size, color components, and inversely quantized transformation coefficients, and the output is the residual sample after the inverse transformation process using the selected transformation.

[0143] Section (2802) describes the derivation of the variable represented by implicitMtsEnabled. The variable indicates whether implicit selection is enabled or not. As described, if the SPS syntax element sps_mts_enabled_flag is equal to 1 (indicating that MTS is enabled for the image or block associated with this SPS syntax element), implicitMtsEnabled is equal to 1 if one of the following three conditions is true: (i) the ISP is used to code the current block, (ii) SBT is enabled and both sides of the current block are 32 or less, which indicates that SBT will be used, or (iii) sps_explicit_mts_intra_enabled_flag and sps_explicit_mts_inter_enabled_flag are both equal to 0 (which indicates that explicit MTS is disabled for both intra-coded and inter-coded blocks), then the current block is intra-coded. Otherwise, implicitMtsEnabled is set to 0, which indicates that implicit conversion selection is disabled for the current block.

[0144] As described in Section (2802), under condition (i), an implicit conversion for coding intra-prediction blocks coded in ISP mode may be enabled. Under condition (iii), for intra-prediction blocks not coded in ISP mode, an implicit conversion may be enabled for coding intra-prediction blocks not coded in ISP mode if explicit MTS is disabled for both the intra-prediction block and the inter-prediction block.

[0145] In section (2803), the vertical and horizontal transforms are determined according to the variable implicitMtsEnabled, the input to the process, and the associated syntax elements. For example, if both sps_explicit_mts_intra_enabled_flag and sps_explicit_mts_inter_enabled_flag are equal to 0, then the current block (of luminance components and intra predictions) can be given a horizontal transform kernel (denoted by trTypeHor) and a vertical transform kernel (denoted by trTypeBer) according to equations (3-1) and (3-2), respectively.

[0146] In Tables (2804) and (2805), numbers 1 and 2, which indicate the horizontal or vertical transform kernel type, represent DST-7 and DCT-8 transforms, respectively, and number 0 represents DCT-2 transforms. In some examples, DST-4 transforms can be used instead of DST-7 transforms.

[0147] In the examples in Figures 28A and 28B, for blocks controlled by the HLS elements sps_explicit_mts_intra_enabled_flag and sps_explicit_mts_inter_enabled_flag, it is not possible to have both explicit MTS for intercoded blocks and implicit transformation for intracoded blocks (non-ISP coded). However, the implicit transformation selection for intra-predictive residual blocks does not necessarily depend on the HLS element (e.g., sps_explicit_mts_inter_enabled_flag) that controls the MTS enabling inter-predictive residuals. Therefore, in some embodiments, whether an implicit transformation can be applied to intracoded blocks not predicted by ISP mode depends on whether the MTS can be applied to intra-predictive residuals (e.g., the value of sps_explicit_mts_intra_enabled_flag), but not on whether the MTS can be applied to inter-predictive residuals (the value of sps_explicit_mts_inter_enabled_flag). This control mechanism allows both inter-MTS (explicit MTS applied to inter-predicted residual blocks) and implicit conversion (for intra-predicted residuals) to be enabled simultaneously.

[0148] For example, the decoder can receive the first and second HLS elements. The first and second HLS elements control the same set of images or regions (e.g., video sequences, images, slices, tiles, etc.) that can contain intra-coded residual blocks and inter-coded residual blocks. The first HLS element indicates whether an explicit MTS is enabled or disabled for each intra-coded residual block, and the second HLS element indicates whether an explicit MTS is enabled or disabled for each inter-coded residual block. With respect to the inter-MTS section, an explicit MTS applied to an intra-coded residual block can be called an intra-MTS.

[0149] If the first HLS element indicates that explicit MTS is disabled for intracoded residual blocks, the decoder may decide to enable implicit transformation (or implicit MTS) for the intracoded blocks accordingly, without considering the value of the second HLS element. For example, the second HLS element can be either 0 or 1, which does not affect the enabling of implicit transformation for intracoded residual blocks.

[0150] As an example, Figure 29 shows a modification (2900) to text (2800) corresponding to an implicit conversion enable scheme where implicit conversion of intra residual blocks and explicit conversion of inter residual blocks can coexist. In modification (2900), removed text is marked with a strikethrough, and added text is marked with an underline. As shown in the figure, the sps_explicit_mts_inter_enabled_flag condition, which is 0, has been removed.

[0151] 2. Disable implicit conversions when other coding tools are enabled. In some embodiments, whether an implicit transformation can be applied to an intracoded block (not predicted by ISP mode) depends on whether a particular coding tool is enabled as indicated by the HLS element.

[0152] 2.1 When NSST is enabled In one embodiment, if the HLS element indicates that NSST is enabled, implicit transformations can be disabled for intra-coded blocks (that are not predicted by the ISP mode). For example, as described in Section II.4.1, the transformation matrix T is used in (Equation 3) for quadratic transformation processing. The transformation matrix T may contain constant elements and can be designed assuming specific statistics for target coefficient blocks, such as coefficient blocks resulting from explicit transformation selection. Therefore, coefficient blocks resulting from implicit transformation selection may not match NSST with respect to the statistics of each coefficient. For example, applying NSST to the result of an implicit transformation does not improve coding performance. For the reasons above, it may be desirable to disable implicit transformations when NSST is enabled.

[0153] In this disclosure, the term NSST can be used to refer to a family of non-separable quadratic transformation coding schemes such as NSST, RST, LFNST, and other initial designs.

[0154] For example, a decoder may receive two HLS elements, one of which (e.g., sps_explicit_mts_intra_enabled_flag) indicates that explicit MTS is disabled, and the other (e.g., sps_lfnst_enabled_flag) indicates that NSST is enabled. Thus, the decoder can decide not to enable the implicit conversions for the intra-coded blocks controlled by these two HLS elements. In contrast, if one HLS element indicates that explicit MTS is disabled, and the other HLS element also indicates that NSST is disabled, the decoder can decide to enable the implicit conversions for each intra-coded block.

[0155] As an example, Figure 30 shows a modification (3000) to text (2800) corresponding to a scenario where implicit conversion is enabled when NSST is disabled. As illustrated, the condition (iii) for deriving the variable implicitNtsEnabled is increased by the additional restriction that "sps_lfnst_enabled_flag is equal to 0".

[0156] 2.2 When MIP is enabled Similar to scenarios where NSST is enabled, in some embodiments, if the HLS element indicates that MIP is enabled, implicit transformations are disabled for intra-coded blocks (not predicted by the ISP mode). For example, applying the MIP intra-coding mode to a block results in a residual block with different statistics than a residual block coded in normal intra-coding mode. Therefore, a residual block coded with MIP may not match an implicit transformation that can assume residual statistics from normal intra-mode. Thus, it may be desirable to disable implicit transformations when MIP is enabled.

[0157] For example, a decoder may receive two HLS elements. One HLS element (e.g., sps_explicit_mts_intra_enabled_flag) indicates that explicit MTS is disabled, and the other HLS element (e.g., sps_mip_enabled_flag) indicates that MIP is enabled. Thus, the decoder can decide not to enable the implicit transformations for the intracoded blocks controlled by these two HLS elements. In contrast, if both explicit MTS and MIP are disabled, as indicated by the two HLS elements, the decoder can decide to enable the implicit transformations for each intracoded block.

[0158] As an example, Figure 31 shows a modification (3100) to text (2800) corresponding to a scenario where implicit conversion is enabled when MIP is disabled. As illustrated, the condition (iii) for deriving the variable implicitNtsEnabled is increased by the additional restriction that "sps_mip_enabled_flag is equal to 0".

[0159] 3. Enable implicit conversions based on block-level syntax element directives. In some embodiments, whether an implicit transformation (not predicted by ISP mode) can be applied to an intracoded block depends on whether MTS, NSST, or MIP is applied to the intracoded block, as indicated by the block-level syntax element.

[0160] 3.1 If MTS and NSST are not applied In one embodiment, whether an implicit transformation can be applied to an intracoded block depends on whether both MTS and NSST are applied to the intracoded block, as indicated by the block-level syntax element.

[0161] For example, the decoder may receive first and second CU-level (or block-level) syntax elements associated with the current block, which are intracoded and not predicted by the ISP mode. The current block may be located at coordinates [x0][y0] in the image.

[0162] A first CU-level syntax element (e.g., tu_mts_idx[x0][y0]) can indicate that a DCT-2 transformation may be used for the current block instead of the transformations used by MTS (e.g., DST-7, DCT-8, DCT-4, etc.). For example, in Table (2804), when tu_mts_idx[x0][y0] has a value of 0, both the trTypeHor and trTypeVer variables have a value of 0, which indicates that the horizontal and vertical transformations are DCT-2 transformations and that no DST-7 or DCT-8 transformations apply. A second CU-level syntax element (e.g., lfnst_idx[x0][y0]) can indicate that NSST does not apply to the current block. Based on the first and second CU-level syntax elements described above, the decoder may decide to enable the implicit MTS for the current block.

[0163] In contrast, if the syntax elements at the first CU level and the second CU level indicate that MTS is applied (tu_mts_idx[x0][y0] has a value of 1, 2, 3, or 4) or that NSST is applied (lfnst_idx[x0][y0] has a non-zero value), the decoder may decide to disable implicit MTS.

[0164] As an example, Figure 32 shows a modification (3200) to text (2800) that corresponds to a scenario where neither MTS nor NSST applies to the current block. As illustrated, the original condition (iii) in text (2800) is replaced with the condition that both CU-level syntax elements "tu_mts_idx[x0][y0] and lfnst_idx[x0][y0] are equal to 0".

[0165] 3.2 If MTS, NSST, and MIP are not applied In one embodiment, whether an implicit transformation can be applied to an intracoded block depends on whether MTS, NSST, and MIP are not applied to the intracoded block, as indicated by the block-level syntax element. Another coding tool, MIP, is further considered in comparison to Section III.3.1.

[0166] For example, the decoder may receive first, second, and third CU-level (or block-level) syntax elements associated with the current block that are intracoded and not predicted by the ISP mode. The current block may be located at coordinates [x0][y0] in the image.

[0167] A first CU-level syntax element (e.g., tu_mts_idx[x0][y0]) can indicate that the DCT-2 transformation can be used for the current block instead of the transformation used in MTS. A second CU-level syntax element (e.g., lfnst_idx[x0][y0]) can indicate that NSST is not applied to the current block. A third CU-level syntax element (e.g., tu_mip_flag) can indicate that MIP is not applied to the current block. Based on the above three CU-level syntax elements, the decoder can decide to enable implicit MTS for the current block.

[0168] In contrast, if the above three CU-level syntax elements indicate that MTS is applied (tu_mts_idx[x0][y0] has a value of 1, 2, 3, or 4), NSST is applied (lfnst_idx[x0][y0] has a non-zero value), or MIP is applied (tu_mip_flag has a value of 1), the decoder may decide to disable implicit MTS.

[0169] As an example, Figure 33 shows a modification (3300) to text (2800) that corresponds to a scenario where neither MTS, NSST, nor MIP applies to the current block. As illustrated, the original condition (iii) in text (2800) is replaced with the condition "tu_mts_idx[x0][y0], intra_mip_flag[x0][y0] and lfnst_idx[x0][y0] are all equal to 0".

[0170] 4. Example of a conversion coding process Figures 34 to 36 show flowcharts of the conversion coding processes (3400), (3500), and (3600) according to some embodiments of the present disclosure. Processes (3400), (3500), and (3600) can be used in the inverse conversion process in the decoder to generate residual blocks of the block being reconstructed. In various embodiments, processes (3400), (3500), and (3600) can be executed by processing circuits such as the processing circuits of terminal devices (210), (220), (230), and (240), the processing circuit that performs the function of the video decoder (310), and the processing circuit that performs the function of the video decoder (410). In some embodiments, processes (3400), (3500), and (3600) can be implemented by software instructions, and so when the processing circuit executes the software instructions, the processing circuit executes processes (3400), (3500), and (3600).

[0171] 4.1 Process (3400): Enable implicit conversion when explicit MTS is disabled Process (3400) starts at (S3401) and proceeds to (S3410).

[0172] In (S3410), a first HLS element is received in the decoder. The first HLS element can indicate whether an explicit MTS is disabled for an intracoded block. For example, an intracoded block can be called the current block because it is being reconstructed. The first HLS element can be associated with a set of coding blocks that includes the current block.

[0173] In (S3420), a second HLS element is received by the decoder. The second HLS element can indicate whether explicit MTS is enabled for the intercoded block. For example, the second HLS element can be associated with the same set of coding blocks as the first HLS element. Both the current block and the intercoded block are included in the set of coding blocks.

[0174] In (S3430), the decoder may decide to enable implicit MTS for the current block if the first HLS element indicates that explicit MTS is disabled for the intracoded block and the second HLS element indicates that explicit MTS is enabled for the intercoded block. The process (3400) may proceed to (S3499), where it terminates.

[0175] 4.2 Process (3500): Disabling implicit conversions when other coding tools are enabled Process (3500) starts at (S3501) and proceeds to (S3510).

[0176] In (S3510), the first HLS element is received by the decoder. The first HLS element can indicate whether explicit MTS is enabled for an intracoded block. An intracoded block can be a block being rebuilt. The first HLS element controls the set of coding blocks that include the intracoded block.

[0177] In (S3520), a second HLS element is received by the decoder. The second HLS element can indicate whether NSST or MIP is disabled for an intracoded block. For example, the second HLS element can control the same set of coding blocks as the first HLS element.

[0178] In (S3530), the decoder can decide whether to enable implicit MTS for processing the intracoded block based on the first HLS element and the second HLS element. For example, if the first HLS element indicates that explicit MTS is disabled for the intracoded block and the second HLS element indicates that NSST or MIP is disabled for the intracoded block, the decoder can decide to enable implicit MTS for the intracoded block. Alternatively, if the first HLS element indicates that explicit MTS is disabled for the intracoded block, but the second HLS element indicates that NSST or MIP is enabled for the intracoded block, the decoder can decide to disable implicit MTS for the intracoded block. The process (3500) can proceed to (S3599) and terminates in (S3599).

[0179] 4.3 Process (3600): Enabling implicit conversions based on block-level syntax element directives Process (3600) starts at (S3601) and proceeds to (S3610).

[0180] In S(3610), an intracoded block is received by the decoder. The intracoded block can be associated with a first block-level syntax element indicating whether MTS is applied and a second block-level syntax element indicating whether NSST is applied. For example, the intracoded block belongs to a CU that contains a CU-level syntax element that includes the first and second block-level syntax elements.

[0181] In (S3620), the decoder can decide whether or not to enable implicit MTS for the intracoded block based on a first block-level syntax element and a second block-level syntax element. For example, if the first block-level syntax element indicates that MTS is not applied and the second block-level syntax element indicates that NSST is not applied, the decoder can decide to enable implicit MTS for the intracoded block. Alternatively, if the first block-level syntax element indicates that MTS is not applied, but the second block-level syntax element indicates that NSST is applied, the decoder can decide to disable implicit MTS for the intracoded block. The process can proceed to (S3699) and terminate in (S3699).

[0182] IV. Computer Systems The techniques described above can be implemented as computer software that uses computer-readable instructions and is physically stored on one or more computer-readable media. For example, Figure 37 shows a computer system (3700) suitable for carrying out a particular embodiment of the disclosed subject matter.

[0183] Computer software can be coded using any suitable machine code or computer language that can create code containing instructions that may be subject to assembly, compilation, linking, or similar mechanisms by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., and which may be executed directly or through translation, microcode execution, etc.

[0184] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.

[0185] The components shown in Figure 37 for the computer system (3700) are essentially illustrative and are not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Furthermore, the configuration of the components should not be construed as having any dependence or requirement on any one or combination of components shown in the exemplary embodiments of the computer system (3700).

[0186] The computer system (3700) may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users, for example, through haptic input (keystrokes, swipes, data glove movements, etc.), audio input (voices, applause, etc.), visual input (gestures, etc.), and olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as voice (speech, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), and video (2D video, 3D video including stereoscopic video, etc.).

[0187] The input human interface device may include one or more of the following (only one of each shown): keyboard (3701), mouse (3702), trackpad (3703), touchscreen (3710), data glove (not shown), joystick (3705), microphone (3706), scanner (3707), and camera (3708).

[0188] The computer system (3700) may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (3710), data glove (not shown), or joystick (3705), although there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speakers (3709), headphones (not shown)), visual output devices (e.g., screens (3710), including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input functionality and with or without tactile feedback functionality - some of these may output two-dimensional visual output or output beyond three dimensions via means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0189] The computer system (3700) may also include human-accessible storage devices and associated media, such as optical media including CD / DVD ROM / RW (3720) with CD / DVD or similar media (3721), thumb drives (3722), removable hard drives or solid-state drives (3723), conventional magnetic media such as tapes and floppy disks (not shown), and dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0190] Those skilled in the art should also understand that the term “computer-readable medium” as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transient signals.

[0191] The computer system (3700) may also include interfaces to one or more communication networks. These networks may be, for example, wireless, wired, or optical. Furthermore, the networks may be local, wide-area, metropolitan, vehicle and industrial, real-time, or latency-tolerant. Examples of networks include local area networks such as Ethernet and Wi-Fi; cellular networks including GSM, 3G, 4G, 5G, and LTE; wired or wireless wide-area digital television networks including cable TV, satellite TV, and terrestrial broadcast TV; and vehicle and industrial networks including CANBus. Certain networks generally require an external network interface adapter connected to a specific general-purpose data port or peripheral bus (3749) (e.g., a USB port on the computer system (3700)), while other networks are generally integrated into the core of the computer system (3700) by mounting to a system bus, as described later (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (3700) can communicate with other entities. Such communications can be unidirectional, receive only (e.g., broadcast television), unidirectional transmit only (e.g., from a CANbus to a specific CANbus device), or bidirectional, such as communication to other computer systems using a local area or wide area digital network. As described above, specific protocols and protocol stacks can be used for each of these networks and network interfaces.

[0192] The aforementioned human interface devices, human-accessible storage devices, and network interfaces can be mounted on the core (3740) of the computer system (3700).

[0193] The core (3740) can include one or more central processing units (CPUs) (3741), a graphics processing unit (GPU) (3742), a dedicated programmable processing device in the form of a field programmable gate array (FPGA) (3743), a hardware accelerator (3744) for specific tasks, etc. These devices may be connected via a system bus (3748) together with internal mass storage devices such as read-only memory (ROM) (3745), random access memory (3746), an internal hard drive that is not accessible to the user, and a solid state drive (SSD) (3747). In some computer systems, the system bus (3748) can be accessed in the form of one or more physical plugs, enabling expansion by additional CPUs, GPUs, etc. Peripheral devices can be connected directly to the system bus (3748) of the core or via a peripheral bus (3749). Architectures of peripheral buses include PCI, USB, etc.

[0194] The CPU (3741), GPU (3742), FPGA (3743), and accelerator (3744) can execute specific instructions that can together constitute the above-mentioned computer code. That computer code can be stored in the ROM (3745) or RAM (3746). Migration data can also be stored in the RAM (3746), while persistent data can be stored, for example, in the internal mass storage device (3747). Fast storage and retrieval to any of the memory devices can be enabled by using a cache memory that can be closely associated with one or more CPUs (3741), GPUs (3742), the mass storage device (3747), ROM (3745), RAM (3746), etc.

[0195] A computer-readable medium can have computer code thereon for performing various computer-implemented operations. The media and computer code can be those specially designed and constructed for the purposes of this disclosure or they can be of the kinds well-known and available to those of ordinary skill in the computer software arts.

[0196] By way of example and not limitation, a computer system having an architecture (3700), specifically a core (3740), can provide functionality as a result of software executed by a processor (including a CPU, GPU, FPGA, accelerator, etc.) incorporated in one or more tangible computer-readable media. Such computer-readable media can be media associated with a mass storage device accessible by a user as introduced above, as well as specific storage devices of the core (3740) of a non-transitory nature such as an on-core mass storage device (3747) or ROM (3745). The software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (3740). The computer-readable media can include one or more memory devices or chips according to specific needs. The software can cause the core (3740) and the processors (including a CPU, GPU, FPGA, etc.) therein to execute a specific process or a specific part of a specific process described herein, such as defining a data structure stored in the RAM (3746) and changing such a data structure according to a process defined by the software. In addition or as an alternative, the computer system can provide functionality that can operate instead of or together with software that executes a specific process or a specific part of a specific process described herein as a result of logic incorporated in hardware or incorporated in a circuit (such as an accelerator (3744)). References to software can include logic, and vice versa where appropriate. References to computer-readable media can include software for execution, circuits that embody logic for execution, or, where appropriate, circuits (such as an integrated circuit (IC)) that store both. The present disclosure encompasses any suitable combination of hardware and software. Appendix A: Abbreviations AMT: Adaptive Multiplexed Transform ASIC: Application-Specific Integrated Circuit BMS: Benchmark Set CANBus: Controller Area Network Bus CCLM: Cross-Component Linear Model CD: Compact Disc COT: Compound Orthonormal Transform CPU: Central Processing Unit CRT: cathode ray tube CTB: Coding Tree Block CTU: Coding Tree Unit CU: Coding Unit DVD: Digital Video Disc EMT: Extended Multiplexing FPGA: Field-Programmable Gate Area GOP: Group of images GPU: Graphics Processing Unit GSM: Global Mobile Communications System HDR: High Dynamic Range HEVC: High-Efficiency Video Coding HLS: High-Level Syntax HRD: Virtual Reference Decoder IBC: Intrablock Copy IC: Integrated Circuit IDT: Identification Information Conversion ISP: Intra-subpartitioning JEM: Collaborative Search Model JVET: Joint Video Exploration Team KLT:Karhunen-Loeve transformation LAN: Local Area Network LCD: Liquid crystal display LFNST: Low-frequency unseparated second-order converter LTE: Long-Term Evolution MIP: Matrix-based intra-prediction mode MRL (or MRLP): Multireference Line Prediction MTS: Multiple Conversion Selection MV: Motion Vector NSST: Non-separable quadratic transform OLED: Organic Light-Emitting Diode PB: Prediction Block PCI: Peripheral Component Interconnect PLD: Programmable Logic Device PPS: Picture Parameter Set PU: Prediction Unit RAM: Random Access Memory ROM: Read Only Memory RST: Resize Transformation SBT: Sub-Block Transformation SDR: Standard Dynamic Range SEI: Supplemental Enhancement Information SNR: Signal-to-Noise Ratio SPS: Sequence Parameter Set SSD: Solid State Drive SVT: Spatial Variance Transformation TSM: Transform Skip Mode TU: Transform Unit USB: Universal Serial Bus VPS: Video Parameter Set VUI: Video User Interface Information VVC: Versatile Video Coding

[0197] Although the present disclosure has described several exemplary embodiments, there are changes, substitutions, and various alternative equivalents within the scope of the present disclosure. Therefore, those skilled in the art will understand that although not explicitly shown or described herein, many systems and methods that embody the principles of the present disclosure and are thus within its spirit and scope can be devised.

Description of Reference Numerals

[0198] 101 Current Block ​​​​​​​​​​​​​​​​302 Video Images 303 Video Encoder 304 Encoded Video Data 305 Streaming Server 306 Client Subsystem 307 copies 308 Client Subsystem 309 copies 310 Video Decoder 311 Video Images 312 displays 313 Capture Subsystem 320 Electronic Devices 330 Electronic Devices 401 Channel 410 Video Decoder 412 rendering devices 415 buffer memory 420 Parser 421 Symbols 430 Electronic Devices 431 Receiver 451 Scaler / Inverse Unit 452 Intra Image Prediction Unit 453 Motion Compensation Prediction Unit 455 Aggregator 456 Loop Filter Unit 457 Reference Image Memory 458 Current image buffer 501 Video Sources 503 Video Encoder 520 Electronic Devices 530 Source Coder 532 Coding Engine 533 Local Video Decoder 534 Reference Image Memory 535 Predictor 540 Transmitter 543 Coded Video Sequences 545 Entropy Coder 550 Controller 560 communication channels 603 Video Encoder 621 General-purpose controller 622 Intra Encoders 623 Residual Calculator 624 residual encoder 625 Entropy Encoder 626 switches 628 Residual Decoder 630 Interencoders 710 Video Decoder 771 Entropy Decoder 772 Intra Decoder 773 Residual Decoder 774 Reconstruction Module 780 Interdecoder Line 1301 Line 1302 Line 1303 Line 1304 Line 1305 1500 tables 1900 table 2000 Alternative Conversion Coding Process 2010 Forward Linear Transform 2012 Forward Quadratic Transform 2013 Coefficient Block 2016 Bitstream 2020 Inverse Linear Transform 2022 Inverse Quadratic Transform 2023 Coefficient Block 2024 Inverse quantization 2100 Alternative Conversion Coding Process 2200 CU Level Syntax Table 2301 Reduction Transformation Process 2302 Process of the Inverse Contraction Transform 2410 Residual Block 2500 tables Exemplary process in 2600 MIP mode 2610 Rectangular Block 2621 subsampled sets 2622 Remaining positions 2701 Frame 2800 text Section 2801 Section 2802 Section 2803 2804 table 2805 table 2900 Correction 3000 corrections 3100 Correction 3200 Corrections 3300 Correction 3400 Conversion Coding Process 3500 Conversion Coding Process 3600 Conversion Coding Process 3700 Computer Systems 3701 Keyboard 3702 Mouse 3703 Trackpad 3705 Joystick 3706 Microphone 3707 Scanner 3708 Camera 3709 Speaker 3710 Touchscreen 3720 CD / DVD ROM / RW 3721 Medium 3722 Thumb Drive 3723 Solid State Drive 3740 cores 3741 Central Processing Unit (CPU) 3742 Graphics Processing Unit (GPU) 3743 Field-Programmable Gate Area (FPGA) 3744 Hardware Accelerator 3745 Read-only memory (ROM) 3746 random access memory 3747 Internal Mass Storage 3748 System Bus 3749 Local buses

Claims

[Claim 1] A method for video decoding in a video decoder, The steps include receiving a first high-level syntax (HLS) element indicating whether explicit multiple transformation selection (MTS) is enabled or disabled for an intracoded block, A step of receiving a second HLS element indicating whether the explicit MTS is enabled or disabled for an intercoded block, wherein the first and second HLS elements control the same set of coding blocks including the intracoded block and the intercoded block; The steps include enabling implicit MTS for an intracoded block when the first HLS element indicates that the explicit MTS is disabled for the intracoded block and the second HLS element indicates that the explicit MTS is enabled for the intercoded block, and A method that includes this.

Images