Scan order of quadratic transformation coefficients

Abstract

Methods, apparatus, and computer-readable storage medium for processing video data. A method comprises the steps of: extracting a data block from video data; scanning a first number of data items of the data block along a first scan order to generate a first data sequence; performing a non-separable transformation on the first data sequence to obtain a second data sequence having a second number of data items; and replacing at least a portion of the first number of data items of the data block with part or all of the second data sequence along a second scan order.

Description

Technology Field

[0001] Inclusion by reference

[0002] This application is based on U.S. Provisional Application No. 63 / 238,646 filed on August 30, 2021 and U.S. Full Application No. 17 / 587,164 filed on January 28, 2022, and claims the benefit of priority to these, which are incorporated herein by reference in their entirety.

[0003] Technology field

[0004] The present disclosure describes a set of advanced video coding techniques. More specifically, the disclosed techniques include the implementation of non-separable transforms of data blocks in video encoding and decoding. Background Technology

[0005] The background description provided herein is intended for the purpose of generally presenting the context of the present disclosure. The work of the inventors named herein is not recognized, either explicitly or implicitly, as prior art to the present disclosure, not only to the extent that the work is described in this background section, but also to the extent that modes of description may not otherwise qualify as prior art at the time of filing this application.

[0006] Video coding and decoding can be performed using inter-picture prediction along with motion compensation. Uncompressed digital video may contain a series of pictures, each having, for example, a spatial dimension of 1920 x 1080 luminance samples and associated whole or subsampled chrominance samples. The series of pictures may have a fixed or variable picture rate (alternatively referred to as the frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, video with a pixel resolution of 1920 x 1080, a frame rate of 60 frames / second, and 4:2:0 chroma subsampling at 8 bits per pixel per color channel requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires storage space larger than 600 gigabytes.

[0007] One purpose of video coding and decoding may be to reduce the redundancy of the uncompressed input video signal through compression. In some cases, compression can help reduce the aforementioned bandwidth and / or storage requirements by more than two orders of magnitude. Lossless compression, lossy compression, and combinations thereof can all be adopted. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal through the decoding process. Lossy compression refers to coding / decoding processes where the original video information is not fully preserved during coding and is not fully recoverable during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is made small enough to render the reconstructed signal useful for the intended application, despite some loss of information. For video, lossy compression is widely adopted in many applications. The amount of acceptable distortion depends on the application. For example, users of certain consumer video streaming applications may tolerate higher distortion than users of movie or television broadcast applications. The compression ratios achievable by a particular coding algorithm may be selected or adjusted to reflect various distortion tolerances; higher acceptable distortion generally allows for coding algorithms that yield higher losses and higher compression ratios.

[0008] Video encoders and decoders can utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0009] Video codec techniques may include techniques known as intra-coding. In intra-coding, sample values ​​are represented without reference to other data from samples or previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture may be referred to as an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as the first picture of the coded video bitstream and video session, or as a still image. Then, the samples in the block after intra-prediction can be subjected to transformation into the frequency domain, and the resulting transformation coefficients can be quantized before entropy coding. Intra-prediction represents a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after conversion and the smaller the AC coefficients, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0010] For example, traditional intra-coding, such as that known from MPEG-2 generative coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to code / decode blocks based on surrounding sample data and / or metadata acquired during the encoding and / or decoding of spatially neighboring blocks and preceding the blocks of data being intra-coded or decoded in the decoding order. These techniques are hereinafter referred to as "intra-prediction" techniques. It should be noted that, in at least some cases, intra-prediction uses only reference data from the current picture being reconstructed and does not use reference data from other reference pictures.

[0011] There may be many different forms of intra-prediction. When two or more of these techniques are available in a given video coding technique, the technique in use may be referred to as an intra-prediction mode. A specific codec may provide one or more intra-prediction modes. In certain cases, modes may have submodes and / or may be associated with various parameters, and mode / submode information and intra-coding parameters for blocks of video may be coded individually or collectively and included in mode codewords. The codewords used for a given combination of mode, submode, and / or parameters may influence the coding efficiency gains through intra-prediction, and the same may apply to the entropy coding technique used to translate the codewords into a bitstream.

[0012] Intra prediction in specific modes was introduced with H.264, improved in H.265, and further enhanced in newer coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Generally, for intra prediction, a predictor block can be formed using available neighbor sample values. For example, available values ​​from a specific set of neighbor samples along a specific direction and / or lines can be copied into the predictor block. A reference to the direction in use may be encoded in the bitstream or predicted itself.

[0013] Referring to FIG. 1a, a subset of the nine predicted directions specified in the 33 possible intra predictor directions of H.265 (corresponding to the 33 angular modes of the 35 intra modes specified in H.265) is shown in the lower right. The point (101) where the arrows converge represents the sample being predicted. The arrows represent the directions from neighboring samples used to predict the sample at 101. For example, arrow (102) indicates that the sample (101) is predicted to the upper right from neighboring samples at a 45-degree angle from the horizontal direction. Similarly, arrow (103) indicates that the sample (101) is predicted to the lower left of the sample (101) from neighboring samples at a 22.5-degree angle from the horizontal direction.

[0014] Referring still to FIG. 1a, a square block (104) of 4 x 4 samples (indicated by a bold dashed line) is shown in the upper left corner. The square block (104) contains 16 samples, each of which is labeled “S”, its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample of the block (104) in both the Y and X dimensions. Since the block size is 4 x 4 samples, S44 is located in the lower right corner. Exemplary reference samples following a similar numbering scheme are additionally shown. The reference samples are labeled R, their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, prediction samples adjacent to the block being reconstructed are used.

[0015] Intra-picture prediction of block (104) can be initiated by copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, the coded video bitstream includes signaling indicating the prediction direction of the arrow (102) for this block (104), that is, assuming that samples are predicted from the upper right of the predicted samples at a 45-degree angle from the horizontal direction. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0016] In certain cases, to calculate reference samples, particularly when the directions cannot be divided equally into 45 degrees, for example, the values ​​of multiple reference samples can be combined through interpolation.

[0017] As video coding technology continues to advance, the number of possible directions is increasing. For example, in H.264 (2003), nine different directions are available for intra prediction. This increased to 33 in H.265 (2013), and at the time of this disclosure, JEM / VVC / BMS can support up to 65 directions. Experimental studies have been conducted to help identify the most appropriate intra prediction directions, and specific entropy coding techniques are used to encode those most appropriate directions with a small number of bits while accepting a specific bit penalty for the directions. Additionally, directions themselves can sometimes be predicted from the neighbor directions used for intra prediction of decoded neighbor blocks.

[0018] FIG. 1b illustrates a schematic diagram (180) showing 65 intra-predicted directions according to JEM to illustrate the increasing number of predicted directions in various encoding technologies developed over time.

[0019] The method for mapping bits representing intra-prediction directions to the prediction directions of a coded video bitstream may vary depending on the video coding technique; for example, it can range from simple direct mappings of prediction directions to complex adaptive methods involving intra-prediction modes, codewords, and most probable modes, as well as similar techniques. However, in all cases, there may be specific directions for intro prediction that are statistically less likely to occur than other specific directions in the video content. Since the goal of video compression is the reduction of redundancy, a well-designed video coding technique may allow these less likely directions to be represented by a greater number of bits than the more likely directions.

[0020] Inter-picture prediction, or inter-prediction, may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or part thereof (reference picture) may be used to predict a newly reconstructed picture or part of a picture (e.g., a block) after being spatially shifted in the direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. MVs may have two dimensions X and Y, or three dimensions, where the third dimension is an indication of the reference picture in use (similar to a time dimension).

[0021] In some video compression techniques, the current MV applicable to a specific region of sample data can be predicted from other MVs, for example, from other MVs related to other regions of sample data that are spatially adjacent to the region being reconstructed and precede the current MV in the decoding order. By doing so, the total amount of data required to code the MVs can be significantly reduced by relying on the redundancy removal of correlated MVs, thereby increasing compression efficiency. MV prediction can work effectively, for example when coding an input video signal derived from a camera (known as natural video), because there is a statistical likelihood that regions larger than the area where a single MV is applicable will move in similar directions within the video sequence; thus, in some cases, it can be predicted using similar motion vectors derived from MVs of neighboring regions. As a result, the actual MV for a given region becomes similar to or identical to the MV predicted from surrounding MVs. These MVs, in turn, after entropy coding, can be represented with fewer bits than is used when the MV is coded directly instead of being predicted from neighboring MV(s). In some cases, MV prediction can be an example of lossless compression of signals (i.e., MVs) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself may be lossy, for example, due to rounding errors when computing the predictor from multiple neighboring MVs.

[0022] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms specified by H.265, the technique referred to as "spatial merge" below is described below.

[0023] Specifically, referring to FIG. 2, the current block (201) includes samples discovered by an encoder during the motion search process so as to be predictable from a previous block of the same size that is spatially shifted. Instead of directly coding the MV, the MV may be derived from metadata associated with one or more reference pictures, for example, from the most recent reference picture (in the decoding order), using an MV associated with any one of five neighboring samples denoted as A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction may use predictors from the same reference picture used by neighboring blocks.

[0024] Aspects of the present disclosure provide methods and apparatus for video encoding and decoding.

[0025] Aspects of the present disclosure also provide non-transient computer-readable media storing instructions that, when executed by a computer for video decoding and / or encoding, cause the computer to perform methods for video decoding and / or encoding.

[0026] According to one aspect, an embodiment of the present disclosure provides a method for processing video data. The method comprises the steps of: extracting a data block from video data; scanning a first number of data items of the data block along a first scan order to generate a first data sequence; performing a non-separable transform on the first data sequence to obtain a second data sequence having a second number of data items; and replacing at least a portion of the first number of data items of the data block with part or all of the second data sequence along a second scan order.

[0027] According to another aspect, an embodiment of the present disclosure provides a method for entropy encoding transform factors associated with video data. The method comprises: scanning the transform factors when performing entropy encoding of the transform factors using a first scanning order which is either a horizontal scan order or a vertical scan order, in response that the transform associated with the transform factors is non-separable; and scanning the transform factors in a second scanning order different from the first scanning order when performing entropy encoding of the transform factors, in response that the transform associated with the transform factors is separable.

[0028] According to another aspect, an embodiment of the present disclosure provides a method for processing video data. The method comprises the steps of: receiving video data; determining whether a non-separable transform is applied as a secondary transform to the video data; in response to the application of a non-separable transform as a secondary transform to the video data: scanning a first number of secondary transform coefficients, wherein the secondary transform coefficients follow a first scanning order; performing a non-separable transform using the first number of secondary transform coefficients as input to obtain a second number of secondary transform coefficients as output, wherein the secondary transform coefficients follow a second scanning order; replacing at least the second number of secondary transform coefficients with secondary transform coefficients following a second scanning order; and performing an inverse secondary transform corresponding to the non-separable transform using the second number of secondary transform coefficients as input to obtain the first number of secondary transform coefficients as output. and includes the step of replacing at least the first number of second transformation coefficients with first transformation coefficients in the first scanning order.

[0029] According to another aspect, an embodiment of the present disclosure provides an apparatus for video encoding and / or decoding. The apparatus includes a memory storing instructions; and a processor communicating with the memory. When the processor executes the instructions, the processor is configured to cause the apparatus to perform the methods for video decoding and / or encoding.

[0030] According to another aspect, an embodiment of the present disclosure provides non-transient computer-readable media storing instructions that, when executed by a computer for video decoding and / or encoding, cause the computer to perform the methods for video decoding and / or encoding.

[0031] The above and other embodiments and implementations thereof are described in more detail in the drawings, detailed descriptions, and claims. Brief explanation of the drawing

[0032] Additional features, characteristics, and various advantages of the disclosed subject will become more apparent from the following detailed description and the accompanying drawings. FIG. 1a illustrates a schematic diagram of an exemplary subset of intra-predicted directional modes. FIG. 1b illustrates examples of exemplary intra-predicted directions. Figure 2 illustrates a schematic diagram of its surrounding space merge candidates for predicting the current block and motion vector in one example. FIG. 3 illustrates a schematic diagram of a simplified block diagram of a communication system (300) according to an exemplary embodiment. FIG. 4 illustrates a schematic diagram of a simplified block diagram of a communication system (400) according to an exemplary embodiment. FIG. 5 illustrates a schematic diagram of a simplified block diagram of a video decoder according to an exemplary embodiment. FIG. 6 illustrates a schematic diagram of a simplified block diagram of a video encoder according to an exemplary embodiment. FIG. 7 illustrates a block diagram of a video encoder according to another exemplary embodiment. FIG. 8 illustrates a block diagram of a video decoder according to another exemplary embodiment. FIG. 9 illustrates directional intra prediction modes according to exemplary embodiments of the present disclosure. FIG. 10 illustrates non-directional intra prediction modes according to exemplary embodiments of the present disclosure. FIG. 11 illustrates recursive intra prediction modes according to exemplary embodiments of the present disclosure. FIG. 12 illustrates conversion block partitioning and scanning of intra-prediction blocks according to exemplary embodiments of the present disclosure. FIG. 13 illustrates conversion block partitioning and inter-prediction block scanning according to exemplary embodiments of the present disclosure. FIG. 14 illustrates a low frequency non-separable transform process according to exemplary embodiments of the present disclosure. FIG. 15 illustrates a data flow for performing a non-separable conversion according to exemplary embodiments of the present disclosure. FIG. 16 illustrates a flowchart according to exemplary embodiments of the present disclosure. FIG. 17 illustrates a schematic diagram of a computer system according to exemplary embodiments of the present disclosure. Specific details for implementing the invention

[0033] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, through a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected through the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) can perform unidirectional transmission of data. For example, a terminal device (310) can encode video data (for example, a stream of video pictures captured by the terminal device (310)) to be transmitted to another terminal device (320) through the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) receives coded video data from the network (350), decodes the coded video data to recover video pictures, and can display the video pictures according to the recovered video data. Unidirectional data transmission can be implemented in media serving applications, etc.

[0034] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of coded video data, which may be implemented, for example, during a videoconferencing application. For bidirectional transmission of data, in the example, each terminal device of terminal devices (330) and (340) may code video data (e.g., a stream of video pictures captured by a terminal device) for transmission to another terminal device of terminal devices (330) and (340) via a network (350). Each terminal device of terminal devices (330) and (340) may also receive coded video data transmitted by another terminal device of terminal devices (330) and (340), decode the coded video data to recover video pictures, and display the video pictures on an accessible display device according to the recovered video data.

[0035] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) may be implemented as servers, personal computers, and smartphones, but the applicability of the basic principles of the present disclosure may not be so limited. Embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, etc. Network (350) represents any number or type of networks that transmit coded video data between the terminal devices (310), (320), (330), and (340), including, for example, wired and / or wireless communication networks. The communication network (350)9 may exchange data on circuit-switched, packet-switched, and / or other types of channels. Representative networks include communication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may not be important to the operation of the disclosure unless explicitly described herein.

[0036] FIG. 4 illustrates the placement of a video encoder and a video decoder in a video streaming environment as an example of an application for the disclosed object. The disclosed object may be equally applicable to other video applications, including, for example, video conferencing, digital TV broadcasting, games, virtual reality, storage of compressed video for digital media including CDs, DVDs, memory sticks, etc.

[0037] A video streaming system may include a video capture subsystem (413) that may include a video source (401), e.g., a digital camera, for generating a stream (402) of uncompressed video pictures or images. In the example, the stream (402) of video pictures includes samples recorded by the digital camera of the video source (401). The stream (402) of video pictures, illustrated in bold lines to emphasize the high data volume compared to encoded video data (404) (or coded video bitstreams), may be processed by an electronic device (420) comprising a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement the embodiments of the disclosed subject as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), illustrated with thin lines to emphasize the lower data volume compared to the stream (402) of uncompressed video pictures, may be stored directly on a streaming server (405) or downstream video devices (not illustrated) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, may access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include a video decoder (410) in, for example, an electronic device (430). A video decoder (410) decodes an incoming copy (407) of encoded video data and generates an outgoing stream (411) of video pictures that can be rendered on a display (412) (e.g., a display screen) or other rendering devices (not shown) without compression.The video decoder (410) may be configured to perform some or all of the various functions described in this disclosure. In some streaming systems, encoded video data (404), (407), and (409) (e.g., video bitstreams) may be encoded according to specific video coding / compression standards. Examples of such standards include ITU-T Recommendation H.265. In example, the video coding standard under development is informally known as Universal Video Coding (VVC). The disclosed subject matter may be used in the context of VVC and other video coding standards.

[0038] Note that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0039] FIG. 5 illustrates a block diagram of a video decoder (510) according to any embodiment of the present disclosure below. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuitry). The video decoder (510) may be used instead of the video decoder (410) of the example of FIG. 4.

[0040] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In the same or other embodiments, one coded video sequence may be decoded at a time, wherein the decoding of each coded video sequence is independent of other coded video sequences. Each video sequence may be associated with multiple video frames or images. The coded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data or to a streaming source transmitting the encoded video data. The receiver (531) may receive the encoded video data along with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective processing circuitries (not shown). The receiver (531) may separate the coded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be placed between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)"). In certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, it may be outside the video decoder (510) and separated from it (not shown). In other applications, for example, a buffer memory (not shown) may be outside the video decoder (510) for the purpose of preventing network jitter, and for example, another additional buffer memory (515) may be inside the video decoder (510) to handle playback timing.When the receiver (531) receives data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may not be needed or may be small. For use in best-effort packet networks such as the Internet, a buffer memory (515) of sufficient size may be required, and its size may be relatively large. This buffer memory may be implemented in an adaptive size and may be implemented at least partially in an operating system or similar elements (not shown) outside the video decoder (510).

[0041] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from a coded video sequence. Categories of the symbols include information used to manage the operation of the video decoder (510), and information for controlling a rendering device, such as a display (512) (e.g., a display screen), which may or may not be an essential part of the electronic device (530), but can be coupled to the electronic device (530) as illustrated in FIG. 5. The control information for the rendering device(s) may be in the form of Supplemental Enhancement Information (SEI messages) or Video Usability Information (VUI) parameter set fragments (not shown). The parser (520) may parse / entropy-decode the coded video sequence received by the parser (520). The entropy coding of the coded video sequence may follow video coding techniques or standards and may follow various principles including variable length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (520) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the subgroups. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc.The parser (520) may also extract information such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc. from the coded video sequence.

[0042] The parser (520) can generate symbols (521) by performing an entropy decoding / parsing operation on a video sequence received from a buffer memory (515).

[0043] The reconstruction of the symbols (521) may involve a number of different processing or functional units depending on the type of the coded video picture or parts thereof (e.g., inter- and intra-picture, inter- and intra-block), and other factors. The units involved and the method by which they are involved may be controlled by subgroup control information parsed from the video sequence coded by the parser (520). The flow of this subgroup control information between the parser (520) and the number of processing or functional units below is not illustrated for the sake of simplification.

[0044] Beyond the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these functional units may interact closely with one another and at least partially integrate with one another. However, for the purpose of clearly describing the various functions of the subject of disclosure, the conceptual subdivision into functional units is adopted in the disclosure below.

[0045] The first unit may include a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) may receive control information including information indicating the type of inverse transform to be used, block size, quantization factor / parameters, quantization scaling matrices, as well as quantized transform coefficients as symbol(s) (521) from the parser (520). The scaler / inverse transform unit (551) may output blocks containing sample values ​​that can be input to an aggregator (555).

[0046] In some cases, the output samples of the scaler / inverse transform (551) may belong to an intra-coded block, that is, a block that can use prediction information from previously reconstructed parts of the current picture without using prediction information from previously reconstructed pictures. This prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may generate a block of the same size and shape as the block being reconstructed by using surrounding block information that has already been reconstructed and stored in the current picture buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator (555) may add the prediction information generated by the intra-prediction unit (552) per sample basis to the output sample information provided by the scaler / inverse transform unit (551).

[0047] In other cases, the output samples of the scaler / inverse unit (551) may belong to an inter-coded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for inter-picture prediction. After motion-compensating the fetched samples according to the symbols (521) belonging to the block, these samples may be added by the aggregator (555) to the output of the scaler / inverse unit (551) (the output of the unit (551) may be referred to as residual samples or residual signals) to generate output sample information. Addresses in the reference picture memory (557) where the motion compensation prediction unit (553) fetches prediction samples may be controlled by motion vectors available to the motion compensation prediction unit (553) in the form of symbols (521) that may have, for example, X, Y components (shift), and reference picture components (time). Motion compensation may also include interpolation of sample values ​​such as those fetched from the reference picture memory (557) when accurate motion vectors of sub-samples are in use, and may also be associated with motion vector prediction mechanisms, etc.

[0048] The output samples of the aggregator (555) may be subject to various loop filtering techniques in the loop filter unit (556). Video compression techniques may include in-loop filter techniques that are included in the coded video sequence (also referred to as the coded video bitstream) and controlled by parameters available to the loop filter unit (556) as symbols (521) from the parser (520), and may also respond to meta-information acquired during the decoding of previous parts (in the decoding order) of the coded picture or coded video sequence, as well as to previously reconstructed and loop-filtered sample values. As described in more detail below, various types of loop filters may be included as part of the loop filter unit (556) in various orders.

[0049] The output of the loop filter unit (556) may be a sample stream that can be output to the rendering device (512) as well as stored in the reference picture memory (557) for use in future inter-picture prediction.

[0050] Certain coded pictures, when fully reconstructed, can be used as reference pictures for future inter-picture prediction. For example, if a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a fresh current picture buffer can be reallocated before starting the reconstruction of the next coded picture.

[0051] A video decoder (510) can perform decoding operations according to a predetermined video compression technique adopted in a standard such as ITU-T Rec. H.265. The coded video sequence may follow the syntax specified by the video compression technique or standard used, in that the coded video sequence complies with both the syntax of the video compression technique or standard and the profiles documented in the video compression technique or standard. Specifically, the profile may select specific tools from all tools available in the video compression technique or standard as the only tools available for use under that profile. For standard-compliance, the complexity of the coded video sequence may be within bounds defined by the levels of the video compression technique or standard. In some cases, the levels limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (measured, for example, in megasamples per second), maximum reference picture size, etc. The limits set by the levels may, in some cases, be further restricted through Hypothetical Reference Decoder (HRD) specifications signaled in the coded video sequence and metadata for HRD buffer management.

[0052] In some exemplary embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, time, spatial, or signal noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0053] FIG. 6 illustrates a block diagram of a video encoder (603) according to an exemplary embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., a transmitting circuitry). The video encoder (603) may be used instead of the video encoder (403) of the example of FIG. 4.

[0054] The video encoder (603) can receive video samples from a video source (601) (not part of the electronic device (620) in the example of FIG. 6) capable of capturing video image(s) to be encoded by the video encoder (603). In another example, the video source (601) may be implemented as part of the electronic device (620).

[0055] A video source (601) may provide a source video sequence to be coded by a video encoder (603) in the form of a digital video sample stream that may have any appropriate bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCb, RGB, XYZ...), and any appropriate sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device capable of storing previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures or images that impart motion when viewed in sequence. The pictures themselves may be composed of a spatial array of pixels, wherein each pixel may include one or more samples depending on the sampling structure, color space, etc. being used. A person skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0056] According to some exemplary embodiments, the video encoder (603) can code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other time constraints as required by the application. Implementing an appropriate coding rate constitutes a function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to other functional units as described below to control this. The coupling is not illustrated for simplification. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda values ​​of rate-distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions belonging to the video encoder (603) optimized for a specific system design.

[0057] In some exemplary embodiments, the video encoder (603) may be configured to operate in a coding loop. For the sake of oversimplification, in the example, the coding loop may include a source coder (630) (responsible for generating symbols, such as a symbol stream, based on, for example, the input picture to be coded and reference picture(s)), and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in a manner similar to that generated by the (remote) decoder, even if the embedded decoder (633) processes the video stream coded by the source coder (630) without entropy coding (since any compression between the symbols and the video bitstream coded in entropy coding in the video compression techniques considered in the subject of disclosure may be lossless). The reconstructed sample stream (sample data) is input into the reference picture memory (634). Because the decoding of the symbol stream leads to bit-exact results regardless of the decoder location (local or remote), the content of the reference picture memory (634) is also bit-exact between the local encoder and the remote encoder. In other words, when the prediction part of the encoder uses prediction during decoding, it “sees” the exact same sample values ​​as the reference picture samples that the decoder “sees.” This fundamental principle of reference picture synchronicity (and, for example, the resulting drift where synchronicity cannot be maintained due to channel errors) is used to improve coding quality.

[0058] The operation of the “local” decoder (633) may be the same as the operation of the “remote” decoder, such as the video decoder (510) already described in detail above in relation to FIG. 5. However, with brief reference to FIG. 5 as well, since symbols are available and encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portions of the video decoder (510), including the buffer memory (515) and parser (520), may not be fully implemented in the local decoder (633) of the encoder.

[0059] An observation that can be made at this point is that any decoder technology, excluding parsing / entropy decoding which may exist only in the decoder, may also necessarily exist in the corresponding encoder in substantially the same functional form. For this reason, the subject matter disclosed may sometimes focus on the decoder operation associated with the decoding part of the encoder. Accordingly, the description of the encoder technologies may be abbreviated, as it is the opposite of the comprehensively described decoder technologies. A more detailed description of the encoder is provided below only in specific domains or modalities.

[0060] During operation in some exemplary implementations, the source coder (630) may perform motion-compensated predictive coding to predictively code an input picture by referencing one or more previously coded pictures from a video sequence designated as "reference pictures." In this way, the coding engine (632) codes the differences (or residuals) in color channels between pixel blocks of the input picture and pixel blocks of the reference picture(s) that can be selected as predictive reference(s) for the input picture. The term "residual" and its adjective form "residual" may be used interchangeably.

[0061] The local video decoder (633) can decode the coded video data of pictures that can be designated as reference pictures based on the symbols generated by the source coder (630). The operations of the coding engine (632) may advantageously be lossy processes. When the coded video data can be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (633) can replicate the decoding processes that can be performed by the video decoder on the reference pictures and cause the reconstructed reference pictures to be stored in the reference picture cache (634). In this way, the video encoder (603) can locally store copies of reconstructed reference pictures having common content as reconstructed reference pictures to be acquired by a far-end (remote) video decoder (no transmission errors).

[0062] The predictor (635) can perform predictive searches for the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (634) for specific metadata, such as sample data (as candidate reference pixel blocks) or reference picture motion vectors, block shapes, etc., which can serve as appropriate predictive references for the new pictures. The predictor (635) can operate on a sample block-by-pixel block basis to find appropriate predictive references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have predictive references drawn from a number of reference pictures stored in the reference picture memory (634).

[0063] The controller (650) can manage the coding operations of the source coder (630), including, for example, the settings of parameters and subgroup parameters used to encode video data.

[0064] The outputs of all the above-mentioned function units can be subject to entropy coding in the entropy coder (645). The entropy coder (645) converts the symbols generated by the various function units into a coded video sequence by lossless compression of the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0065] The transmitter (640) may buffer the coded video sequence(s) generated by the entropy coder (645) to prepare for transmission over a communication channel (660) which may be a hardware / software link to a storage device storing the encoded video data. The transmitter (640) may merge the coded video data from the video coder (603) with other data to be transmitted, e.g., coded audio data and / or auxiliary data streams (sources not shown).

[0066] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a specific coded picture type to each coded picture that can affect the coding techniques that can be applied to the individual pictures. For example, pictures can often be assigned as one of the following picture types:

[0067] An Intra Picture (I Picture) may be one that can be coded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of Intra Pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. A person skilled in the art is aware of the applicable variations of I Pictures and their individual applications and characteristics.

[0068] The predictive picture (P picture) may be coded and decoded using intra-prediction or inter-prediction, using at most one motion vector and reference index to predict sample values ​​of each block.

[0069] A bi-directionally predictive picture (Picture B) may be coded and decoded using intra-prediction or inter-prediction, which uses up to two motion vectors and reference indices to predict sample values ​​of each block. Similarly, multiple-predictive pictures may use three or more reference pictures and associated metadata to reconstruct a single block.

[0070] Source pictures are generally spatially subdivided into multiple sample coding blocks (e.g., blocks of 4 x 4, 8 x 8, 4 x 8, or 16 x 16 samples, respectively) and can be coded on a block-by-block basis. Blocks can be predictively coded by referencing other (already coded) blocks as determined by the coding assignment applied to the individual pictures of the blocks. For example, blocks of pictures I may be non-predictively coded, or they may be predictively coded by referencing already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of pictures P may be predictively coded via spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks of pictures B may be predictively coded via spatial prediction or temporal prediction by referencing one or two previously coded reference pictures. Source pictures or intermediately processed pictures may be subdivided into different types of blocks for different purposes. The division of coding blocks and other types of blocks may or may not follow the same method as described in more detail below.

[0071] The video encoder (603) can perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In the operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancies in the input video sequence. The coded video data may follow the syntax specified by the video coding technique or standard in use accordingly.

[0072] In some exemplary embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include this data as part of the encoded video sequence. The additional data may include other forms of redundant data such as time / space / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0073] Video can be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra prediction) utilizes the spatial correlation of a given picture, while inter-picture prediction utilizes the temporal or other correlation between pictures. For example, a specific picture being encoded / decoded, referred to as the current picture, can be partitioned into blocks. A block of the current picture can be coded by a vector referred to as a motion vector when it resembles a reference block of a reference picture that was previously encoded and is still buffered in the video. The motion vector points to the reference block of the reference picture and, in cases where multiple reference pictures are in use, may have a third dimension that identifies the reference picture.

[0074] In some exemplary embodiments, a bi-prediction technique may be used for inter-picture prediction. According to this bi-prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture (however, in the display order, they may each be past or future) that proceed through the current picture of the video in the decoding order. A block of the current picture may be coded by a first motion vector pointing to a first reference block of the first reference picture, and a second motion vector pointing to a second reference block of the second reference picture. A block may be jointly predicted by a combination of the first reference block and the second reference block.

[0075] In addition, merge mode technology can be used for inter-picture prediction to improve coding efficiency.

[0076] According to some exemplary embodiments of the present disclosure, predictions such as inter-picture predictions and intra-picture predictions are performed in units of blocks. For example, a picture in a sequence of video pictures is partitioned into coding tree units (CTUs) for compression, and the CTUs of the picture may have the same size, such as 64 x 64 pixels, 32 x 32 pixels, or 16 x 16 pixels. Generally, a CTU may include three parallel coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU may be a quadtree recursively split into one or more coding units (CUs). For example, a CTU of 64 x 64 pixels may be split into one CU of 64 x 64 pixels or four CUs of 32 x 32 pixels. Each of one or more of the 32 x 32 blocks may be further split into four CUs of 16 x 16 pixels. In some exemplary embodiments, each CU may be analyzed during encoding to determine the prediction type for the CU among various prediction types, such as inter-prediction type or intra-prediction type. The CU may be split into one or more prediction units (PUs) based on temporal and / or spatial predictability. Generally, each PU includes a luminance prediction block (PB) and two chroma PBs. In the embodiments, the prediction operation during coding (encoding / decoding) is performed by the prediction block units. Splitting the CU into PUs (or PBs of different color channels) may be performed in various spatial patterns.For example, the luminance or chroma PB may include a matrix of values ​​(e.g., luminance values) for samples such as 8 x 8 pixels, 16 x 16 pixels, 8 x 16 pixels, 16 x 8 samples, etc.

[0077] FIG. 7 illustrates a diagram of a video encoder (703) according to another exemplary embodiment of the present disclosure. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures and to encode the processing block into a coded picture that is part of a coded video sequence. The exemplary video encoder (703) may be used instead of the video encoder (403) of FIG. 4.

[0078] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8 x 8 samples. Then, the video encoder (703) determines whether the processing block is best coded using, for example, an intra-mode, an inter-mode, or a bidirectional prediction mode using rate-distortion optimization (RDO). When it is determined that the processing block is coded in intra-mode, the video encoder (703) may use an intra-prediction technique to encode the processing block into a coded picture; when it is determined that the processing block is coded in inter-mode or bidirectional prediction mode, the video encoder (703) may use an inter-prediction or bidirectional prediction technique, respectively, to encode the processing block into a coded picture. In some exemplary embodiments, a merge mode may be used as a submode of inter-picture prediction, where the motion vector is derived from one or more motion vector predictors without the benefit of the coded motion vector component outside the predictors. In some other exemplary embodiments, there may be motion vector components applicable to the target block. Accordingly, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode determination module, to determine the perdition mode of the processing blocks.

[0079] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) that are coupled together as shown in the exemplary arrangement of FIG. 7.

[0080] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks of reference pictures (e.g., blocks of previous and subsequent pictures in the display order), generate inter-prediction information (e.g., merge mode information, motion vectors, description of redundant information according to the inter-encoding technique), and calculate inter-prediction results (e.g., predicted blocks) based on the inter-prediction information using any suitable technique. In some examples, the reference pictures are decoded reference pictures that are decoded based on video information encoded using a decoding unit (633) embedded in the exemplary encoder (620) of FIG. 6 (illustrated as the residual decoder (728) of FIG. 7, as described further in detail below).

[0081] The intra-encoder (722) is configured to receive samples of the current block (e.g., processing block), compare the block with blocks already coded in the same picture, generate quantized coefficients after conversion, and in some cases also generate intra-prediction information (e.g., intra-prediction direction information according to one or more intra-encoding techniques). The intra-encoder (722) can calculate intra-prediction results (e.g., predicted blocks) based on the intra-prediction information and reference blocks of the same picture.

[0082] A general controller (721) may be configured to determine general control data and to control other components of the video encoder (703) based on the general control data. In an example, the general controller (721) determines the prediction mode of the block and provides a control signal to the switch (726) based on the prediction mode. For example, when the prediction mode is intra mode, the general controller (721) controls the switch (726) to select an intra mode result to be used by the residual calculator (723), and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream; when the prediction mode for the block is inter mode, the general controller (721) controls the switch (726) to select an inter prediction result to be used by the residual calculator (723), and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0083] The residual calculator (723) may be configured to calculate the difference (residual data) between the received block and the prediction results for the block selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) may be configured to encode the residual data to generate transformation coefficients. For example, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain to generate transformation coefficients. Then, the transformation coefficients are subjected to quantization processing to obtain quantized transformation coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transformation and generate decoded residual data. The decoded residual data may be appropriately used by the intra-encoder (722) and the inter-encoder (730). For example, the inter encoder (730) can generate decoded blocks based on decoded residual data and inter prediction information, and the intra encoder (722) can generate decoded blocks based on decoded residual data and intra prediction information. The decoded blocks are appropriately processed to generate decoded pictures, and the decoded pictures can be buffered in a memory circuit (not shown) and used as reference pictures.

[0084] The entropy encoder (725) may be configured to format the bitstream to include the encoded block and to perform entropy coding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other appropriate information in the bitstream. When coding the block in the merge submode of either the inter mode or the bidirectional prediction mode, there may be no residual information.

[0085] FIG. 8 illustrates a diagram of an exemplary video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and to decode the coded pictures to generate reconstructed pictures. In the example, the video decoder (810) may be used instead of the video decoder (410) of FIG. 4.

[0086] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) that are coupled together as shown in the exemplary arrangement of FIG. 8.

[0087] The entropy decoder (871) may be configured to reconstruct specific symbols representing the syntax elements that constitute the coded picture from the coded picture. These symbols may include, for example, the mode in which the block is coded (e.g., intra mode, inter mode, bidirectional prediction mode, merge submode, or other submode), prediction information (e.g., intra prediction information or inter prediction information) capable of identifying specific samples or metadata used for prediction by the intra decoder (872) or inter decoder (880), for example, residual information in the form of quantized transformation coefficients. In the example, when the prediction mode is an inter or bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information may be subject to inverse quantization and is provided to the residual decoder (873).

[0088] The inter decoder (880) can be configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0089] The intra decoder (872) can be configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0090] The residual decoder (873) may be configured to perform inverse quantization to extract de-quantized transform coefficients and process the de-quantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize specific control information (including a Quantizer Parameter (QP)) that may be provided by the entropy decoder (871) (the data path is not shown as this may be dedicated to low data volume control information).

[0091] The reconstruction module (874) may be configured to combine residuals output by the residual decoder (873) and prediction results (as output by inter or intra prediction modules, depending on the case) to form a reconstructed block that forms a part of the reconstructed picture as a part of the reconstructed video in the spatial domain. Note that other appropriate operations, such as deblocking operations, may also be performed to improve visual quality.

[0092] It should be noted that video encoders (403), (603), and (703), and video decoders (410), (510), and (810) can be implemented using any suitable technology. In some exemplary embodiments, video encoders (403), (603), and (703), and video decoders (410), (510), and (810) can be implemented using one or more integrated circuits. In other embodiments, video encoders (403), (603), and (603), and video decoders (410), (510), and (810) can be implemented using one or more processors that execute software instructions.

[0093] Returning to the intra prediction process, to generate a prediction block, samples of a block (e.g., a luminal or chroma prediction block, or a coding block if not further split into prediction blocks) are predicted by samples from neighbors, next neighbors, or other lines or lines, or combinations thereof. Then, the residuals between the actual block being coded and the prediction block are processed through a transformation, which may be followed by quantization. Various intra prediction modes may be available, and parameters related to intra mode selection and other parameters may be signaled in the bitstream. For example, various intra prediction modes may relate to line positions or positions for predicting samples, the directions in which prediction samples are selected from the predicting lines or lines, and other specific intra prediction modes.

[0094] For example, a set of intra prediction modes (referred interchangeably as “intra modes”) may include a predefined number of directional intra prediction modes. As described above in relation to the exemplary implementation of FIG. 1, these intra prediction modes may correspond to a predefined number of directions in which out-of-block samples are selected as predictions for samples being selected in a particular block. In another specific exemplary implementation, eight (8) primary directional modes corresponding to angles of 45 to 207 degrees with respect to the horizontal axis may be supported and predefined.

[0095] In some other implementations of intra prediction, to further utilize more diverse spatial redundancy in directional textures, directional intra modes may be further extended to angles set with finer granularity. For example, the above 8-angle implementation may be configured to provide 8 nominal angles designated as V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, as exemplified in FIG. 9, and for each nominal angle, a predefined number (e.g., 7) of finer angles may be added. With this extension, a larger total number (e.g., 56 in this example) of directional angles may be available for intra prediction corresponding to the same number of predefined directional intra modes. The predicted angle can be expressed by the nominal intra angle plus the angle delta. In the specific example above, where there are 7 finer angle directions for each nominal angle, the angle delta can be -3 to 3 times the step size of 3 degrees.

[0096] In some implementations, as an alternative to or in addition to the above directional intra modes, a predefined number of non-directional intra prediction modes may also be predefined and made available. For example, five non-directional intra modes referred to as smooth intra prediction modes may be specified. These non-directional intra prediction modes may specifically be referred to as DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H intra modes. Predictions of samples of a specific block under these exemplary non-directional modes are illustrated in FIG. 10. For example, FIG. 10 illustrates a 4 x 4 block (1002) being predicted by samples from the top neighbor line and / or the left neighbor line. A specific sample (1010) of block (1002) may correspond to the sample immediately above (1004) of the sample (1010) in the upper neighbor line of block (1002), the left-upper sample (1006) of the sample (1010) as the intersection of the upper and left neighbor lines, and the sample immediately to the left (1008) of the sample (1010) in the left neighbor line of block (1002). In the case of an exemplary DC intra prediction mode, the average of the left and upper neighbor samples (1008 and 1004) may be used as a predictor for the sample (1010). In the case of an exemplary PAETH intra prediction mode, top, left, and left-top reference samples (1004, 1008, and 1006) may be fetched, and then any of these three reference samples closest to (top + left - top-left) may be set as the predictor for sample (1010).In the case of an exemplary SMOOTH_V intra prediction mode, a sample (1010) can be predicted by quadratic interpolation in the vertical direction of the left-top neighbor sample (1006) and the left neighbor sample (1008). In the case of an exemplary SMOOTH_H intra prediction mode, a sample (1010) can be predicted by quadratic interpolation in the horizontal direction of the left-top neighbor sample (1006) and the top neighbor sample (1004). In the case of an exemplary SMOOTH intra prediction mode, a sample (1010) can be predicted by the average of quadratic interpolations in the vertical and horizontal directions. The above non-directional intra mode implementations are illustrated merely as non-limiting examples. Other neighbor lines, other non-directional selections of samples, and ways of combining prediction samples to predict a specific sample in a prediction block are also considered.

[0097] The selection of a specific intra prediction mode by the encoder from the above directional or non-directional modes at various coding levels (picture, slice, block, unit, etc.) can be signaled in the bitstream. In some exemplary implementations, eight exemplary nominal directional modes (a total of 13 options) along with five non-angular smooth modes may be signaled first. Then, if the signaled mode is one of the eight nominal angular intra modes, an index is additionally signaled to indicate the selected angular delta for the corresponding signaled nominal angle. In some other exemplary implementations, all intra prediction modes may be indexed together for signaling (e.g., 56 directional modes plus 5 non-directional modes to yield 61 intra prediction modes).

[0098] In some exemplary implementations, 56 or other exemplary directional intra prediction modes may be implemented as a unified directional predictor that projects each sample of the block to a reference sub-sample position and interpolates the reference sample by a 2-tap bilinear filter.

[0099] In some implementations, additional filter modes, referred to as FILTER INTRA modes, may be designed to capture decaying spatial correlations with references on edges. For these modes, predicted samples within a block may be used as intra-prediction reference samples for some patches within a block, in addition to samples outside the block. For example, these modes may be predefined and made available for intra-prediction for at least (or only) luminance blocks. A predefined number (e.g., 5) of filter intra modes may be pre-designed, each of which is represented by a set of n-tap filters (e.g., 7-tap filters) reflecting the correlation between samples in a 4 x 2 patch and their adjacent n neighbors. In other words, the weighting factors for the n-tap filters may be position-dependent. Taking an 8 x 8 block, 4 x 2 patch, and 7-tap filtering as examples, the 8 x 8 block (1102) can be split into 8 4 x 2 patches, as illustrated in FIG. 11. These patches are denoted by B0, B1, B3, B4, B5, B6, and B7 in FIG. 11. For each patch, its 7 neighbors, denoted by R0 through R7 in FIG. 11, can be used to predict samples in the current patch. In the case of patch B0, all neighbors may already have been reconstructed. However, for the other patches, some of the neighbors are in the current block and therefore may not have been reconstructed, and then the predicted values ​​of the immediate neighbors are used as a reference. For example, since all neighbors of patch B7, as shown in FIG. 11, are not reconstructed, the predicted samples of the neighbors are used instead.

[0100] In some implementations of intra-prediction, a single color component can be predicted using one or more other color components. The color component can be any one of the components such as YCrCb, RGB, or XYZ color spaces. For example, prediction of a chroma component (e.g., a chroma block) from a lumina component (e.g., lumina reference samples) can be implemented, referred to as Chroma from Luma or CfL. In some exemplary implementations, the majority of cross-color predictions may be allowed only from lumina to chroma. For example, the chroma samples of a chroma block can be modeled as a linear function of the matching reconstructed lumina samples. CfL prediction can be implemented as follows:

[0101]

[0102] Here, L AC represents the AC contribution of the Luma component, and α represents the parameters of the linear model, and DCrepresents the DC contribution of the chroma component. For example, AC components are acquired for each sample of a block, whereas DC components are acquired for the entire block. Specifically, reconstructed luminance samples can be subsampled to the chroma resolution, and then the average luminance value (DC of luminance) can be subtracted from each luminance value to form the AC contribution from the luminance. Then, the AC contribution of the luminance can be used in the linear mode of equation (1) to predict the AC values ​​of the chroma components. Instead of requiring the decoder to calculate scaling parameters to approximate or predict the chroma AC components from the luminance AC contribution, an exemplary CfL implementation can determine the parameter α based on the original chroma samples and signal them in the bitstream. This reduces decoder complexity and yields more accurate predictions. For the DC contribution of the chroma component, in some exemplary implementations, it can be calculated using the intra-DC mode within the chroma component.

[0103] Then, a transformation of the residuals of either the intra prediction block or the inter prediction block may be implemented, followed by quantization of the transformation coefficients. For the purpose of performing the transformation, both the intra and inter coded blocks may be further partitioned into multiple transformation blocks prior to the transformation (although the term “unit” is generally used to denote a collection of 3-color channels, it is sometimes used interchangeably as “transform units,” for example, “coding units” will include luminance coding blocks and chroma coding blocks). In some implementations, a maximum partitioning depth of the coded blocks (or prediction blocks) may be specified (the term “coded blocks” may be used interchangeably with “coding blocks”). For example, such partitioning may not exceed two levels. The partitioning of prediction blocks into transformation blocks may be handled differently between intra prediction blocks and inter prediction blocks. However, in some implementations, this partitioning may be similar between intra-prediction blocks and inter-prediction blocks.

[0104] In some exemplary implementations, and for intra-coded blocks, transformation partitioning may be performed such that all transformation blocks have the same size and the transformation blocks are coded in raster scan order. An example of such transformation block partitioning for an intra-coded block is illustrated in FIG. 12. Specifically, FIG. 12 illustrates a coded block (1202) being partitioned into 16 transformation blocks of the same block size through intermediate level quadtree splitting (1204), as illustrated by FIG. 1206. An exemplary raster scan order for coding is illustrated by the aligned arrows in FIG. 12.

[0105] In some exemplary implementations, and for intercoded blocks, transformation unit partitioning may be performed recursively with a partitioning depth of up to a predefined number of levels (e.g., 2 levels). Splitting may be recursively interrupted or continued for any sub-partition and at any level, as illustrated in FIG. 13. In particular, FIG. 13 illustrates an example in which a block (1302) is split into four quadtree sub-blocks (1304), one of the sub-blocks is further split into four second-level transformation blocks, while the splitting of the other sub-blocks is interrupted after the first level, yielding a total of seven transformation blocks of two different sizes. An exemplary raster scan order for coding is further illustrated by the arrows aligned in FIG. 13. FIG. 13 illustrates an exemplary implementation of a quadtree split of up to two levels of square transformation blocks, but in some generation implementations, transformation partitioning may support 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1 transformation block shapes and sizes ranging from 4 × 4 to 64 × 64. In some exemplary implementations, if the coding block is 64 × 64 or less, transformation block partitioning may be applied only to the luminance component (in other words, the chroma transformation block will be identical to the coding block under that condition). Alternatively, if the coding block width or height is greater than 64, the luminance and chroma coding blocks may both be implicitly split into multiples of min(W, 64) x min(H, 64) and min(W, 32) x min(H, 32) transformation blocks, respectively.

[0106] Then, each of the above transformation blocks can be subject to a primary transform. The primary transform essentially moves the residuals of the transformation block from the spatial domain to the frequency domain. In some implementations of the actual primary transform, to support the above exemplary extended coding block partitions, multiple transformation sizes (ranging from 4-point to 64-point for each of the two dimensions) and transformation shapes (squares; rectangles with width / height ratios of 2:1 / 1:2 and 4:1 / 1:4) may be allowed.

[0107] Returning to the actual first-order transformation, in some exemplary implementations, the 2-D transformation process may involve the use of hybrid transformation kernels (which may consist, for example, of different 1-D transformations for each dimension of the coded residual transformation block). Exemplary 1-D transformation kernels may include, but are not limited to, a) 4-point, 8-point, 16-point, 32-point, 64-point DCT-2; b) 4-point, 8-point, 16-point asymmetric DSTs (DST-4, DST-7) and their flipped versions; and c) 4-point, 8-point, 16-point, 32-point identity transforms. The selection of transformation kernels to be used for each dimension may be based on the rate-distortion (RD) criterion. For example, basis functions for DCT-2 and asymmetric DSTs that can be implemented are listed in Table 1.

[0108]

[0109] In some exemplary implementations, the availability of hybrid transform kernels for a specific first-order transform implementation may be based on the transform block size and the prediction mode. Exemplary dependencies are listed in Table 2. For the chroma component, the selection of the transform type may be performed implicitly. For example, for intra-predicted residuals, the transform type may be selected according to the intra-predicted mode, as specified in Table 3. For inter-predicted residuals, the transform type for the chroma blocks may be selected according to the selection of the transform type for the co-located luminance blocks. Therefore, for the chroma component, there is no transform type signaling in the bitstream.

[0110]

[0111]

[0112] In some implementations, a second transformation may be performed on the first transformation coefficients. For example, to further decorate the first transformation coefficients, a low-frequency non-separable transform (LFNST), known as a reduced second transformation, may be applied between the forward primary transform (at the encoder) and quantization, and between the inverse quantization and inverse primary transform (at the decoder side), as illustrated in FIG. 14. Essentially, the LFNST may proceed to the second transformation by taking a portion of the first transformation coefficients, for example, the low-frequency portion (and thus "reduced" from the entire set of first transformation coefficients of the transform block). In an exemplary LFNST, a 4 x 4 non-separable transform or an 8 x 8 non-separable transform may be applied depending on the transform block size. For example, a 4 x 4 LFNST can be applied to small transform blocks (e.g., min(width, height) < 8), whereas an 8 x 8 LFNST can be applied to larger transform blocks (e.g., min(width, height) > 8). For example, when an 8 x 8 transform block is the subject of a 4 x 4 LFNST, only the low-frequency 4 x 4 portion of the 8 x 8 first-order transform coefficients undergoes an additional second-order transform.

[0113] Specifically, as illustrated in FIG. 14, the transform block may be 8 x 8 (or 16 x 16). Thus, the forward first transform (1402) of the transform block yields an 8 x 8 (or 16 x 16) first transform coefficient matrix (1404), where each square unit represents a 2 x 2 (or 4 x 4) portion. For example, the input to the forward LFNST may not be the entire 8 x 8 (or 16 x 16) first transform coefficients. For example, a 4 x 4 (or 8 x 8) LFNST may be used for the second transform. As such, only the 4 x 4 (or 8 x 8) low-frequency first transform coefficients of the first transform coefficient matrix (1404) may be used as input to the LFNST, as indicated in the shaded portion (top left) (1406). The remaining parts of the first transformation coefficient matrix may not be subject to the second transformation. Thus, after the second transformation, the parts of the first transformation coefficients subject to LFNST become the second transformation coefficients, while the remaining parts not subject to LFNST (e.g., the unshaded parts of the matrix (1404)) retain the corresponding first transformation coefficients. In some exemplary implementations, the remaining parts not subject to the second transformation may all be set to coefficients of 0.

[0114] An example for the application of a non-separable transformation used in an LFNST is described below. To apply an exemplary 4 x 4 LFNST, a 4 x 4 input block X (representing, for example, the 4 x 4 low-frequency portion of a first-order transformation coefficient block, such as the shaded portion (1406) of the first-order transformation matrix (1404) in FIG. 14) can be represented as follows:

[0115]

[0116] This 2-D input matrix is ​​a vector in an exemplary order It can be linearized or scanned first:

[0117]

[0118] Then, the non-separable transform for the 4 x 4 LFNST is It can be calculated as, where, displays the output transformation coefficient vector, and T is a 16 x 16 transformation matrix. The resulting 16 x 1 coefficient vector It is subsequently reverse scanned as a 4 x 4 block using the scanning order (e.g., horizontal, vertical, or diagonal) for the corresponding block. Coefficients with smaller indices can be placed with smaller scanning indices in a 4 x 4 coefficient block. In this way, the redundancy of the first transformation coefficients X is the second transformation T It can be additionally utilized through, and thereby, additional compression improvement can be provided.

[0119] The exemplary LFNST above is based on a direct matrix multiplication approach to apply the non-separate transform so that it is implemented in a single pass without multiple iterations. In some additional exemplary implementations, the dimensionality of the non-separate transform matrix (T) for the exemplary 4 x 4 LFNST can be further reduced to minimize computational complexity and memory space requirements for storing transform coefficients. These implementations may be referred to as the reduced non-separate transform (RST). More specifically, the main idea of ​​the RST is N (In the example above, N is 4 x 4 = 16, but for 8 x 8 blocks, it could be equal to 64) dimensional vectors of different spaces R It is mapping to a dimensional vector, and here, N / R ( R < N) represents the dimension reduction factor. Therefore, instead of an N x N transformation matrix, the RST matrix is ​​as follows: R × N Becomes a matrix:

[0120]

[0121] Here, of the transformation matrix R The rows are N Reduced dimensional space R It is a basis of. Therefore, the transformation is the input vector or N Dimensionally reduced R It is converted into an output vector of dimension. In this way, and as illustrated in FIG. 14, the second transformation coefficients (1408) transformed from the first coefficients (1406) have their dimension reduced by a factor or N / R. The three squares around 1408 in FIG. 14 can be zero-padded.

[0122] The inverse transform matrix for the RTS may be the transpose of its forward transform. For an exemplary 8 x 8 LFNST (contrasted with the 4 x 4 LFNST above for more diverse description herein), an exemplary reduction factor of 4 may be applied, and thus, the 64 x 64 direct non-separable transform matrix is ​​reduced accordingly to a 16 x 64 direct matrix. Additionally, in some implementations, only a portion, rather than all, of the input first-order coefficients may be linearized into the input vector for the LFNST. For example, only a portion of the exemplary 8 x 8 input first-order transform coefficients may be linearized into the X vector above. For a specific example, among the four 4 x 4 quadrants of the 8 x 8 first-order transform coefficient matrix, the bottom right (high-frequency coefficients) may be omitted, and only the other three quadrants are linearized into 48 x 1 vectors using a predefined scan order rather than 64 x 1 vectors. In these implementations, the non-separable transformation matrix can be further reduced from 16 x 64 to 16 x 48.

[0123] Therefore, an exemplary reduced 48 x 16 inverse RST matrix can be used on the decoder side to generate the top-left, top-right, and bottom-left 4 x 4 quadrants of the 8 x 8 core (first) transform coefficients. Specifically, when additionally reduced 16 x 48 RST matrices are applied instead of the 16 x 64 RST using the same transform set configuration, the non-separable second transform will take as input 48 matrix elements vectorized from three 4 x 4 quadrant blocks of the 8 x 8 first coefficient block, excluding the bottom-right 4 x 4 block. In these implementations, the omitted bottom-right 4 x 4 first transform coefficients will be ignored in the second transform. This additional reduced transformation will convert a 48 x 1 vector into a 16 x 1 output vector, which is reverse-scanned into a 4 x 4 matrix to fill 1408 in Fig. 14. Three squares of the quadratic transformation coefficients around 1408 can be zero-padded.

[0124] With the help of these dimensionality reductions in RST, memory usage for storing all LFNST matrices is reduced. In the example above, memory usage can be reduced from 10KB to 8KB with, for example, an insignificant performance degradation compared to an implementation without dimensionality reduction.

[0125] In some implementations, to reduce complexity, LFNST may be further restricted to be applicable only when all coefficients outside the first-order transformation coefficient portion targeted by LFNST (e.g., outside the 1406 portion of 1404 in FIG. 14) are non-significant. Thus, all first-order transformation coefficients (e.g., the unshaded portion of the first-order coefficient matrix (1404) in FIG. 4) may be close to zero when LFNST is applied. This restriction allows for conditioning of LFNST index signaling for the last-significant position and thus avoids some redundant coefficient scanning that might need to be performed for significant coefficients at certain positions when this restriction is not applied. In some implementations, the worst-case handling of LFNST (in terms of multiplications per pixel) may limit non-separable transforms for 4 x 4 and 8 x 8 blocks to 8 x 16 and 8 x 48 transforms, respectively. In these cases, for other sizes smaller than 16, the last significant scan position must be less than 8 when LFNST is applied. For blocks with shapes of 4 x N, N x 4, and N > 8, the above limitation means that LFNST is now applied only once to the top-left 4 x 4 region. Since all first-order dedicated coefficients are zero when LFNST is applied, the number of operations required for first-order transforms in these cases is reduced. From the encoder perspective, the quantization of coefficients can be simplified when LFNST transforms are tested. Rate-distortion optimized quantization (RDO) should be performed to the maximum for the first 16 coefficients (in the scan order), and the remaining coefficients can be forced to be 0.

[0126] In some exemplary implementations, available RST kernels may be specified as multiple transformation sets, each containing multiple non-separable transformation matrices. For example, there may be a total of four transformation sets and two non-separable transformation matrices (kernels) per transformation set used in LFNST. These kernels can be pre-trained offline and are therefore data-driven. Offline-trained transformation kernels may be stored in memory or hardcoded into the encoding or decoding device for use during the encoding / decoding process. The selection of transformation sets during the encoding or decoding process may be determined by intra-prediction modes. The mapping from intra-prediction modes to transformation sets may be predefined. Examples of such predefined mappings are shown in Table 4. For example, as shown in Table 4, if one of the three Cross-Component Linear Model (CCLM) modes (INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for the current block (i.e., 81 <= predModeIntra <= 83), transform set 0 may be selected for the current chroma block. For each transform set, the selected non-separable secondary transform candidate may be further specified by an explicitly signaled LFNST index. For example, the index may be signaled in the bit-stream once per intra CU after the transform coefficients.

[0127]

[0128] Because LFNST is restricted in the exemplary implementations above to be applicable only when all external coefficients of the first coefficient subgroup or some of them are not significant, LFNST index coding depends on the position of the last significant coefficient. Additionally, LFNST indexes can be context-coded but do not depend on the intra prediction mode, and only the first bin can be context-coded. Furthermore, LFNST can be applied to the intra CU in both intra and inter slices, and to both Luma and Chroma. When dual tree is enabled, LFNST indices for Luma and Chroma can be signaled separately. In the case of inter slices (where dual tree is disabled), a single LFNST index is signaled and can be used for both Luma and Chroma.

[0129] In some exemplary implementations, when the Intra Sub-Partitioning (ISP) mode is selected, LFNST may be disabled and the RST index may not be signaled because there is a possibility that applying RST to all executable partition blocks will result in minimal performance improvement. Additionally, disabling RST for ISP-predicted residuals can reduce encoding complexity. In some additional implementations, when the Multiple linear regression Intra Prediction (MIP) mode is selected, LFNST may also be disabled and the RST index may not be signaled.

[0130] Considering that large CUs larger than 64 x 64 (or any other predefined sizes representing the maximum transform block size) are implicitly split due to the existing maximum transform size limit (e.g., 64 x 64) (e.g., TU tiling), LFNST index lookup can increase data buffering by a factor of four for a certain number of decoding pipeline stages. Therefore, in some implementations, the maximum size allowed for LFNST may be limited to, for example, 64 x 64. In some implementations, LFNST may be enabled only as a first-order transform with DCT2.

[0131] In some other implementations, an intra secondary transform (IST) is provided for the luma component by defining, for example, 12 sets of secondary transforms, each having 3 kernels. An intra mode dependent index may be used for selecting the transform sets. The selection of kernels within a set may be based on signaled syntax elements. The IST may be enabled when either DCT2 or ADST is used as both horizontal and vertical primary transforms. In some implementations, depending on the block size, a 4 x 4 non-separable transform or an 8 x 8 non-separable transform may be selected. If min(tx_width, tx_height) < 8, a 4 x 4 IST may be selected. For larger blocks, an 8 x 8 IST may be used. Here, tx_width and tx_height correspond to the transform block width and height, respectively. The input to the IST can be low-frequency first-order transform coefficients of a zigzag scan sequence.

[0132] None of the various transformations of video coding or decoding processes, such as the first transformations of samples in residual blocks or the second transformations of blocks of first transformation coefficients, may all be efficient for capturing directional texture patterns, such as edges at 45 degrees (e.g., directions substantially away from horizontal or vertical directions), when using only separable transform schemes. As described above, in some exemplary implementations, one or more non-separable transform designs may be used for the second transformation of the first transformation coefficients. As further explained below, these non-separable transform schemes may also be adopted for the first transformation of residual sample blocks to generate the first transformation coefficients. In some exemplary implementations, first-order transform coefficients generated through either separate or non-separable transforms may be subject to entropy coding following direct quantization, or alternatively, to either separate or non-separable second-order transforms before quantization and entropy coding. In implementations where first-order transform coefficients obtained through non-separable transforms are additionally subjected to non-separable second-order transforms, two non-separable transforms will be utilized in a cascading manner.

[0133] The disclosure below further describes some exemplary implementations of non-separable transformation schemes applicable to both first and second transformations. These non-separable transformation designs described below aim to improve coding efficiency, particularly for directional image patterns. In particular, intra-mode dependent non-separable first and / or second transformation schemes are disclosed. These exemplary embodiments focus on the data scanning sequences of blocks being coded / decoded during either forward or reverse transformation processes. The block being processed may generally be referred to as a data block, which, in the case of forward transformations, may include samples of the residual transformation / coding / prediction block or first transformation coefficients, and in the case of reverse transformations, may include second transformation coefficients that are inversely transformed into first transformation coefficients or first transformation coefficients that are inversely transformed into residual samples.

[0134] To optimize energy compaction achieved by non-separable first-order or second-order transformation and to effectively quantize the first-order or second-order transformation coefficients (to discard high-frequency coefficients), the input and output scanning processes of the non-separable first-order or second-order transformation can be performed considering an intra-prediction mode, block size, and / or first-order transformation type, as described in more detail below.

[0135] In some exemplary implementations, a transformation set may refer to a group of one or more transformation kernels as candidates or options for the encoder to select for a data block during the coding process.

[0136] In some implementations, the first-order transformation may be performed using a non-separable transformation or achieved by performing a series of 1-D transformations. For example, in the DCT_DCT combination, the DCT is applied horizontally and vertically to the block. As another example, in the ADST_ADST combination, the 1-D ADST is applied horizontally and vertically to the block. In some implementations, different transformation types may be used horizontally and vertically. These transformations may be referred to as hybrid first-order transformations.

[0137] FIG. 15 illustrates an exemplary data flow (1500) for forward and reverse non-separable transformations. In FIG. 15, various data scanning processes are identified as S1 and S2, and forward and reverse non-separable transformations are identified as 1502 and 1504. The operation of the exemplary data flow (1500) applies to both first and second non-separable transformation processes and is described in more detail below.

[0138] Data scanning in non-separable quadratic transformations

[0139] In this exemplary embodiment, the forward and reverse second-order transformations may be non-separable transformations, as illustrated by 1502 and 1504 in FIG. 15. From the encoding side, the input of the forward second-order transformation (1502) may be N first-order transformation coefficients (1506) of a first-order coefficient data block (1508) scanned into a data sequence (1510) along a first scanning order S1 in a scanning process (1512), and the output of the forward second-order transformation (1502) may be a sequence of K second-order transformation coefficients (1514) of K data items that replace K input first-order transformation coefficients (1516) to generate a modified first-order transformation coefficient data block (1507) along a data scan (1518) using a second scanning order S2. N and K are positive integers.

[0140] From the decoding side, the input of the reverse secondary transformation (1504) is a sequence of secondary transformation coefficients (1520) of K data items obtained after reverse scanning the K input coefficients (1517) of the primary coefficient data block (1509) modified in the reverse S2 scanning order in the scanning process (1522), and the output of the reverse secondary transformation (1504) may be a sequence of primary transformation coefficients (1524) of N coefficients that replace the N primary transformation coefficients (1526) of 1509 to generate the primary coefficient data block (1528) using the reverse scanning process (1530) following the reverse S1 scanning order.

[0141] In some exemplary implementations, N may be greater than K. That is, the non-separable second transformation (1502) may be a reduced non-separable second transformation. In some exemplary embodiments, the N first transformation coefficients (1506) may be the first N coefficients (1506) of the first coefficient data block, for example, the upper left corner (or low-frequency portion) (1506) of the first transformation coefficient data block.

[0142] In one exemplary implementation, the scanning order S1 may depend on at least one of: an intra-prediction mode associated with the input data block (1508), a primary transformation type associated with the data block, or a block size of the data block.

[0143] In one exemplary implementation, the scanning order S2 may depend on at least one of: an intra-prediction mode associated with the input data block (1508), a primary transformation type associated with the data block, or a block size of the data block.

[0144] In one exemplary implementation, the S1 and S2 scanning order may include at least one of a zigzag scanning order, a diagonal scanning order, or a row and column scanning order.

[0145] In one implementation, when the same set of secondary transformations (having one or more secondary transformation kernels) is used as secondary transformation candidates for multiple intra prediction modes, the S1 and S2 scanning order may depend on the intra prediction mode associated with the data block.

[0146] In some exemplary implementations, the S1 and S2 scanning sequences may differ.

[0147] In some exemplary implementations, N and K include any integers from 0 to 127 (including the boundary), but are not limited thereto.

[0148] In some exemplary implementations, when the data block is a square block, the S1 and / or S2 scanning order may include a zigzag order. When the block is a non-square block, the S1 and / or S2 scanning order may include a diagonal scanning order.

[0149] FIG. 15 further illustrates quantization and entropy encoding (1540) and the corresponding entropy decoding and inverse quantization (1550).

[0150] Data scanning in non-separable first-order transformation as a first-order transformation

[0151] In this exemplary embodiment, the non-separable transformation process (1502 and 1504) may be performed for the first transformation. Thus, the input (1508) to the data flow of 1500 will be, for example, a residual sample data block that is not a first transformation coefficient, and the output (1507) will be a modified residual sample data block containing K first transformation coefficients (1516) from the non-separable transformation process (1502). Specifically, from the encoder side, the input to the forward first transformation may be the first N residual samples following the first scanning order S1, and the output of the forward first transformation is K non-separable transformation coefficients that replace the K input residual samples with the second scanning order S2. N and K are positive integers. From the decoder side, the input to the inverse non-separable transform is K non-separable transform coefficients following the S2 scanning order, and the output to the inverse non-separable transform is N residual samples replacing N non-separable transform coefficients following the S1 scanning order, where N is an integer.

[0152] In one exemplary implementation, the scanning order S1 may depend on at least one of: an intra-prediction mode associated with the data block, a primary transformation type associated with the data block, or a block size of the data block.

[0153] In one exemplary implementation, the scanning order S2 may depend on at least one of: an intra-prediction mode associated with the data block, a primary transformation type associated with the data block, or a block size of the data block.

[0154] In one exemplary implementation, the S1 and S2 scanning order may include at least one of a zigzag scanning order, a diagonal scanning order, or a row and column scanning order.

[0155] In one exemplary implementation, when the same set of transformations (having one or more second transformation kernels) is used as secondary transformation candidates for multiple intra prediction modes, the S1 and S2 scanning order may depend on the intra prediction mode associated with the data block.

[0156] In some exemplary implementations, the S1 and S2 scanning sequences may differ.

[0157] In one exemplary implementation, N and K include any integers from 0 to 127 (including boundaries), but are not limited thereto.

[0158] In one exemplary implementation, when the data block is a square block, the S1 and / or S2 scanning order may include a zigzag scanning order. When the block is a non-square block, the S1 and / or S2 scanning order may include a diagonal scanning order. In other words, a diagonal scanning order may be enforced for either square or non-square data blocks.

[0159] In one exemplary implementation, the S1 scanning order may include: a horizontal (row) scanning order, or a vertical (column) scanning order.

[0160] For these implementations, the quantization / entropy process (1540) and the entropy decoding / inverse quantization process (1550) may be performed directly on 1507 or 1509 without additional second-order transformation. In some other alternative implementations, the additional second-order transformation may be performed on the encoding side before the process (1540) and on the decoding side after the process (1550). The additional second-order transformation may be separable, or, for example, non-separable according to the implementations described above for non-separable second-order transformations.

[0161] Data scanning in non-separable first-order transformation as a first-order transformation

[0162] In some embodiments, the type of transformation (i.e., separate or non-separable) may affect the scanning order when performing transformation factor coding (e.g., entropy encoding of transformation factors). In one implementation, when a non-separable transformation is applied (to a first-order transformation and / or a second-order transformation), a horizontal (row) or vertical (column) scanning order may be used for transformation factor coding.

[0163] FIG. 16 illustrates an exemplary method (1600) for processing video data. The method (1600) may include some or all of the following steps: extracting a data block from video data (1610); scanning a first number of data items of the data block along a first scan order to generate a first data sequence (1620); performing a non-separable transformation on the first data sequence to obtain a second data sequence having a second number of data items (1630); and replacing at least some of the first number of data items of the data block with some or all of the second data sequence along a second scan order (1640).

[0164] In embodiments of the present disclosure, any steps and / or operations may be combined or arranged in any quantity or order as desired. Two or more of the steps and / or operations may be performed in parallel.

[0165] The embodiments of the present disclosure may be used individually or may be combined in any order. Additionally, each of the methods (or embodiments), the encoder, and the decoder may be implemented by a processing network (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transient computer-readable medium. The embodiments of the present disclosure may be applied to a luminance block or a chroma block.

[0166] The technologies described above may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media. For example, FIG. 17 illustrates a computer system (1700) suitable for implementing specific embodiments of the disclosed object.

[0167] Computer software may be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to generate code containing instructions that can be executed directly or through interpretation, micro-code execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0168] The commands can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0169] The components illustrated in FIG. 17 for the computer system (1700) are essentially exemplary and are not intended to suggest any limitations on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be interpreted as having any dependency or requirement related to any one or combination of the components illustrated in the exemplary embodiments of the computer system (1700).

[0170] The computer system (1700) may include specific human interface input devices. These human interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipe, data glove movement), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture specific media that are not necessarily directly related to conscious input by humans, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0171] Input human interface devices may include one or more of a keyboard (1701), a mouse (1702), a trackpad (1703), a touch screen (1710), a data glove (not shown), a joystick (1705), a microphone (1706), a scanner (1707), and a camera (1708) (each only one is shown).

[0172] The computer system (1700) may also include specific human interface output devices. These human interface output devices may stimulate one or more human user's senses through, for example, tactile output, sound, light, and smell / taste. These human interface output devices may include haptic output devices (e.g., touch screen (1710), data glove (not shown), or haptic feedback via joystick (1705), however, there may also be haptic feedback devices that do not serve as input devices), audio output devices (e.g., speakers (1709), headphones (not shown)), visual output devices (e.g., screens (1710) including CRT screens, LCD screens, plasma screens, and OLED screens, each having or not having touch screen input capability and each having or not having haptic feedback capability—some of which may output stereographic output; two-dimensional visual output or four-dimensional or higher output through means such as virtual-reality glasses (not shown), holographic displays and smoke tanks (not shown)), and printers (not shown).

[0173] The computer system (1700) may also include human-accessible storage devices and their associated media, such as optical media including a CD / DVD ROM / RW (1720) having a CD / DVD or similar media (1721), a thumb-drive (1722), a removable hard drive or solid state drive (1723), legacy magnetic media such as tape and floppy disk (not shown), and specialized ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0174] Those skilled in the art should also understand that the term "computer-readable media" as used in connection with the subject matter of the present disclosure does not include transmission media, carriers, or other transient signals.

[0175] The computer system (1700) may also include an interface (1754) to one or more communication networks (1755). The networks may be, for example, wireless, wired, optical. The networks may additionally be local, wide area, metropolitan, vehicular and industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks, for example, Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, and vehicular and industrial networks including CAN buses, etc. Certain networks generally require external network interface adapters attached to certain general-purpose data ports or peripheral buses (1749) (for example, USB ports of the computer system (1700), etc.); Others are generally integrated into the core of the computer system (1700) by attachment to the system bus as described below (e.g., an Ethernet interface for a PC computer system or a cellular network interface for a smartphone computer system). Using any of these networks, the computer system (1700) can communicate with other entities. This communication can be unidirectional, receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., from a CANbus to specific CANbus devices), or bidirectional, using local or wide-area digital networks to other computer systems. Specific protocols and protocol stacks may be used in each of the corresponding networks and network interfaces as described above.

[0176] The core (1740) of the computer system (1700) may be equipped with the previously mentioned human interface devices, human-accessible storage devices, and network interfaces.

[0177] The core (1740) may include one or more Central Processing Units (CPUs) (1741), Graphics Processing Units (GPUs) (1742), specialized programmable processing units in the form of Field Programmable Gate Areas (FPGAs) (1743), hardware accelerators (1744) for specific tasks, graphics adapters (1750), etc. These devices may be connected via a system bus (1748), together with read-only memory (ROM) (1745), random access memory (1746), and internal mass storage, e.g., internal non-user accessible hard drives, SSDs, etc. (1747). In some computer systems, the system bus (1748) may be accessible in the form of one or more physical plugs to enable extensions by additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus (1748) or via a peripheral bus (1749). In the example, the screen (1710) can be connected to a graphics adapter (1750). Architectures for peripheral buses include PCI, USB, etc.

[0178] CPUs (1741), GPUs (1742), FPGAs (1743), and accelerators (1744) can be combined to execute specific instructions that can construct the computer code mentioned above. The computer code may be stored in ROM (1745) or RAM (1746). Transitional data may also be stored in RAM (1746), while persistent data may be stored, for example, in internal mass storage (1747). High-speed storage and retrieval of any of the memory devices may be enabled through the use of a cache memory that may be closely associated with one or more CPUs (1741), GPUs (1742), mass storage (1747), ROM (1745), RAM (1746), etc.

[0179] Computer-readable media may have computer code for performing various computer-implemented operations. The media and computer code may be those specifically designed and configured for the purposes of the present disclosure, or they may be of a type well known to those skilled in the art of computer software and available for use.

[0180] As a non-limiting example, a computer system having an architecture (1700), specifically a core (1740), may provide functionality as a result of processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software implemented on one or more types of tangible computer-readable media. These computer-readable media may be media associated with specific storage of the core (1740) having non-transient characteristics, such as core-internal mass storage (1747) or ROM (1745), as well as the user-accessible mass storage introduced above. Software implementing various embodiments of the present disclosure may be stored on these devices and executed by the core (1740). The computer-readable media may include one or more memory devices or chips depending on specific requirements. Software may enable the core (1740) and, specifically, internal processors (including CPU, GPU, FPGA, etc.) to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (1746) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired to a circuit (e.g., accelerator (1744)) or otherwise embodied, which may operate instead of or together with the software to execute specific processes or specific parts of specific processes described herein. Reference to software may include logic, and where appropriate, vice versa. Reference to computer-readable media may include software for execution, a circuit implementing logic for execution, or, where appropriate, a circuit storing both (e.g., an integrated circuit (IC)).The present disclosure includes any suitable combination of hardware and software.

[0181] Although the present disclosure describes some exemplary embodiments, there are modifications, substitutions, and various alternative equivalents that fall within the scope of the present disclosure. Accordingly, it will be understood that those skilled in the art will be able to devise numerous systems and methods that implement the principles of the present disclosure and thus fall within the spirit and scope thereof, even though they are not expressly illustrated or described herein.

[0182] Appendix A: Acronyms

[0183] JEM: Joint exploration model

[0184] VVC: Versatile Video Coding

[0185] BMS: Benchmark set

[0186] MV: Motion Vector

[0187] HEVC: High Efficiency Video Coding

[0188] SEI: Supplementary Enhancement Information

[0189] VUI: Video Usability Information

[0190] GOPs: Groups of Pictures

[0191] TUs: Transform Units,

[0192] PUs: Prediction Units

[0193] CTUs: Coding Tree Units

[0194] CTBs: Coding Tree Blocks

[0195] PBs: Prediction Blocks

[0196] HRD: Hypothetical Reference Decoder

[0197] SNR: Signal Noise Ratio

[0198] CPUs: Central Processing Units

[0199] GPUs: Graphics Processing Units

[0200] CRT: Cathode Ray Tube

[0201] LCD: Liquid-Crystal Display

[0202] OLED: Organic Light-Emitting Diode

[0203] CD: Compact Disc

[0204] DVD: Digital Video Disc

[0205] ROM: Read-Only Memory

[0206] RAM: Random Access Memory

[0207] ASIC: Application-Specific Integrated Circuit

[0208] PLD: Programmable Logic Device

[0209] LAN: Local Area Network

[0210] GSM: Global System for Mobile communications

[0211] LTE: Long-Term Evolution

[0212] CANBus: Controller Area Network Bus

[0213] USB: Universal Serial Bus

[0214] PCI: Peripheral Component Interconnect

[0215] FPGA: Field Programmable Gate Areas

[0216] SSD: Solid-state drive

[0217] IC: Integrated Circuit

[0218] HDR: High dynamic range

[0219] SDR: Standard Dynamic Range

[0220] JVET: Joint Video Exploration Team

[0221] MPM: most probable mode

[0222] WAIP: Wide-Angle Intra Prediction

[0223] CU: Coding Unit

[0224] PU: Prediction Unit

[0225] TU: Transform Unit

[0226] CTU: Coding Tree Unit

[0227] PDPC: Position Dependent Prediction Combination

[0228] ISP: Intra Sub-Partitions

[0229] SPS: Sequence Parameter Setting

[0230] PPS: Picture Parameter Set

[0231] APS: Adaptation Parameter Set

[0232] VPS: Video Parameter Set

[0233] DPS: Decoding Parameter Set

[0234] ALF: Adaptive Loop Filter

[0235] SAO: Sample Adaptive Offset

[0236] CC-ALF: Cross-Component Adaptive Loop Filter

[0237] CDEF: Constrained Directional Enhancement Filter

[0238] CCSO: Cross-Component Sample Offset

[0239] LSO: Local Sample Offset

[0240] LR: Loop Restoration Filter

[0241] AV1: AOMedia Video 1 (AOMedia Video 1)

[0242] AV2: AOMedia Video 2

Claims

Claim 1 A method for processing video data, comprising: extracting a data block from video data; scanning a first number of data items of the data block along a first scan order to generate a first data sequence; performing a non-separable transform on the first data sequence to obtain a second data sequence having a second number of data items; and replacing at least a portion of the first number of data items of the data block with part or all of the second data sequence along a second scan order, wherein the first scan order is determined based on at least one of: the type of the non-separable second transform; or the size of the data block. Claim 2 A method according to claim 1, wherein the data block comprises a primary transform coefficient; the non-separable transform comprises a forward non-separable secondary transform; and the step of replacing at least a portion of a first number of data items of the data block comprises replacing a second number of data items within the first number of data items of the data block with the second data sequence according to the second scan order. Claim 3 delete Claim 4 A method according to paragraph 2, wherein the second scan order is determined based on at least one of: an intra-prediction mode associated with the data block; a type of forward non-separable second transformation; or the size of the data block. Claim 5 In paragraph 2, the first scan order or the second scan order comprises one of: a zigzag scan order; a diagonal scan order; or a row and column scan order. Claim 6 delete Claim 7 In paragraph 2, the method wherein the first number and the second number are integers from 0 to 127 (including boundaries). Claim 8 A method according to paragraph 2, wherein the first scan order and the second scan order are: a zigzag scan order when the data block is a square block; or a diagonal scan order when the data block is a non-square block. Claim 9 The method of claim 1, wherein the data block comprises residual samples; the non-separable transformation comprises a forward non-separable primary transform; and the step of replacing at least a portion of a first number of data items of the data block comprises replacing a second number of data items within the first number of data items of the data block with the second data sequence according to the second scan order. Claim 10 delete Claim 11 In claim 9, the second scan order is determined based on at least one of: an intra-prediction mode associated with the data block; the type of the forward non-separable second transformation; or the size of the data block. Claim 12 In claim 9, the first scan order or the second scan order comprises one of: a zigzag scan order; a diagonal scan order; or a row and column scan order. Claim 13 delete Claim 14 In claim 9, the method wherein the first number and the second number are integers from 0 to 127 (including boundaries). Claim 15 A method according to claim 9, wherein the first scan order and the second scan order are: a zigzag scan order when the data block is a square block; or a diagonal scan order when the data block is a non-square block. Claim 16 In claim 9, the method wherein the first scan sequence is a horizontal or vertical scan sequence. Claim 17 A method for entropy encoding transform coefficients associated with video data, comprising: a step of scanning the transform coefficients when performing entropy encoding of the transform coefficients using a first scanning order which is one of a horizontal scanning order or a vertical scanning order, in response that the transform associated with the transform coefficients is non-separable; and a step of scanning the transform coefficients in a second scanning order different from the first scanning order when performing entropy encoding of the transform coefficients in response that the transform associated with the transform coefficients is separable, wherein the first scanning order is determined based on at least one of: a type of non-separable second transformation; or a size of a data block. Claim 18 In paragraph 17, the above transformation comprises: a first transformation; or a second transformation, a method. Claim 19 A method for processing video data, comprising: receiving said video data; determining whether a non-separable transformation is applied as a secondary transformation to said video data; in response to said non-separable transformation being applied as a secondary transformation to said video data: a step of scanning first transformation coefficients of a first number, wherein said first transformation coefficients follow a first scanning order; a step of performing a non-separable transformation using said first transformation coefficients of a first number as input to obtain second transformation coefficients of a second number as output, wherein said second transformation coefficients follow a second scanning order; at least a step of replacing said second transformation coefficients of a second number with said second transformation coefficients following the second scanning order; and a step of performing an inverse secondary transform corresponding to said non-separable transformation using said second transformation coefficients of a second number as input to obtain said first transformation coefficients of a first number as output. and at least the step of replacing the first number of second transformation coefficients with the first transformation coefficients according to the first scanning order, wherein the first scanning order is determined based on: the type of non-separable second transformation; or the size of the data block. Claim 20 A device comprising a circuitry configured to implement the method of any one of claims 1, 2, 4, 5, 7 through 9, 11, 12 and 14 through 19. Claim 21 A non-transient computer-readable storage medium in which computer code is stored, wherein the computer code, when executed by one or more processors, enables the one or more processors to implement the method of any one of claims 1, 2, 4, 5, 7 through 9, 11, 12 and 14 through 19.

Images