Video processing method and device, and storage medium

Abstract

The present disclosure relates generally to video coding, and in particular to cross component level reconstruction. For example, a method for processing video data is disclosed, which can include extracting, from a bitstream of a video block, a first transform block of a first color component and a second transform block of a second color component; determining that all transform coefficients in the first transform block are zeros; determining that CCLR is applied to the first transform block; modifying one or more of the transform coefficients in the first transform block to obtain a modified first transform block; determining a target transform kernel for the modified first transform block; performing inverse transform on the modified first transform block based on the target transform kernel to obtain a target block; and reconstructing the first color component of the video block based at least on the target block.

Description

[0001] Incorporation by Reference

[0002] This application is based on and claims priority to U.S. Non-Provisional Application No. 17 / 966,390, filed October 14, 2022, entitled “PRIMARY TRANSFORMS FOR CROSS-COMPONENT LEVEL RECONSTRUCTION,” which is based on and claims priority to U.S. Provisional Application No. 63 / 289,133, filed December 13, 2021, and U.S. Provisional Application No. 63 / 273,789, filed October 29, 2021, both entitled “PRIMARY TRANSFORMS FOR CROSS-COMPONENT LEVEL RECONSTRUCTION.” These prior patent applications are incorporated herein by reference in their entireties. TECHNICAL FIELD

[0003] The present disclosure relates generally to a set of advanced video coding / decoding techniques, and more specifically to primary transforms for offset-based cross-component methods that modify transform coefficients before or after dequantization. BACKGROUND

[0004] The background description provided herein is for the purpose of generally presenting the context of the disclosure. The work of the inventors, to the extent the inventors' work is described in this background section, and the aspects of that work that are not specifically and clearly claimed in this disclosure are neither admitted to be prior art nor specifically and explicitly incorporated by reference in this disclosure as being material to the disclosure.

[0005] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each picture having, for example, a resolution of 1920 x 1080 luminance samples and associated full or subsampled chrominance samples. The series of pictures can have a fixed or variable picture rate (alternatively referred to as frame rate), of, for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bit rate requirements. For example, a video with a pixel resolution of 1920 x 1080, a frame rate of 60 frames / second, and a chroma subsampling of 4:2:0 (8 bit per pixel per color channel) requires close to 1.5 Gbit / s bandwidth. An hour of such video requires more than 600 GBytes of storage space.

[0006] One purpose of video encoding and decoding can be to reduce redundancy in the uncompressed input video signal by compression. Compression can help reduce the bandwidth and / or storage space requirements as described above, in some cases by two orders of magnitude or more. Both lossless and lossy compression, and combinations thereof, can be employed. Lossless compression refers to techniques where an exact copy of the original signal is reconstructed from the compressed original signal via a decoding process. Lossy compression refers to encoding / decoding processes in which the original video information is not completely retained during encoding and cannot be completely recovered during decoding. When using lossy compression, the reconstructed signal can not be identical to the original signal, but the distortion between original and reconstructed signals is small enough to make the reconstructed signal usable for the intended application, despite some information loss. In the case of video, lossy compression is widely used in many applications. The amount of distortion that can be tolerated depends on the application. For example, users of certain consumer video streaming applications can tolerate higher distortion than users of film or television broadcast applications. The compression ratio achievable by a particular encoding algorithm can be chosen or adjusted to reflect various distortion tolerances: higher tolerable distortion typically allows encoding algorithms that produce higher losses and higher compression ratios.

[0007] Video encoders and decoders can utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transforms, quantization, and entropy coding.

[0008] Video codec techniques can include techniques known as intra-coding. In intra-coding, sample values are represented without reference to information from previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into blocks of samples. When all blocks of a picture are coded in intra-mode, that picture can be referred to as an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and a video session, or as a still image. The samples of the blocks after intra prediction can then be subjected to a transform into the frequency domain, and the transform coefficients thus generated can be quantized before entropy coding. Intra prediction is a technique that minimizes the sample values in the pre-transform domain. In some cases, the smaller the DC value after the transform is, and the smaller the AC coefficients are, the fewer the bits that are required to represent the block after entropy coding, at a given quantization step size.

[0009] Conventional intra coding, such as known from e.g. MPEG-2 generation encoding techniques, does not use intra prediction. However, some newer video compression technologies include techniques that attempt to encode / decode a block based on, e.g., surrounding sample data and / or metadata that was obtained during spatially neighboring encoding and / or decoding and that is in decoding order before the block of data is intra coded or decoded. Such techniques are hereafter referred to as "intra prediction" techniques. Note that in at least some cases, intra prediction only uses reference data from the current picture in reconstruction, without using reference data from other reference pictures.

[0010] There can be many different forms of intra prediction. When more than one such technique is available in a given video coding technology, the technique used can be referred to as an intra prediction mode. One or more intra prediction modes can be provided in a particular codec. In some cases, modes can have submodes and / or can be associated with various parameters, and mode / submode information and intra coding parameters for a video block can be encoded separately or jointly included in a mode codeword. Such a codeword for a given mode, submode, and / or parameter combination can impact coding efficiency gains through intra prediction, and thus can impact the entropy coding techniques used to convert the codeword into a bitstream.

[0011] Certain modes of intra prediction were introduced with H.264, improved upon in H.265, and further improved in newer coding technologies such as Joint Exploration Model (JEM), Versatile Video Coding (VVC), and BMS. In general, for intra prediction, available neighboring sample values can be used to form a predictor block. For example, available values of a particular set of neighboring samples along a particular direction and / or line can be copied into the predictor block. The reference to the direction used can be coded in the bitstream or can itself be predicted.

[0012] Reference Figure 1A A subset of nine predictor directions specified in the 33 possible intra predictor directions of H.265 (corresponding to the 33 angular modes out of the 35 intra modes specified in H.265) is depicted in the lower right. The point where the arrows converge (101) represents the sample being predicted. The arrows indicate the direction from which neighboring sample or samples are used to predict the sample at 101. For example, arrow (102) indicates that the sample (101) is predicted from a neighboring sample or samples to the upper right at a 45 degree angle from horizontal. Similarly, arrow (103) indicates that the sample (101) is predicted from a neighboring sample or samples to the lower left of the sample (101) at a 22.5 degree angle from horizontal.

[0013] Still referring to Figure 1AA square block of 4x4 samples is depicted in the upper left (104), shown by the dashed bold lines. The square block (104) includes 16 samples, each labeled with an "S", its position, e.g., row index, in the Y dimension and its position, e.g., column index, in the X dimension. For example, sample S21 is the second sample from the top in the Y dimension and the first sample from the left in the X dimension. Similarly, sample S44 is the fourth sample in the Y dimension and the X dimension in the block (104). As the block size is 4x4 samples, S44 is at the bottom right. Further shown are example reference samples following a similar numbering scheme. The reference samples are labeled with an R, its Y position, e.g., row index, and X position, e.g., column index, relative to the block (104). In H.264 and H.265, the prediction samples immediately adjacent to the block under reconstruction are not used as reference samples.

[0014] Intra picture prediction for block 104 can start by copying reference sample values from neighboring samples according to the signaled prediction direction. For example, assume the coded video bitstream includes signaling that, for this block 104, indicates a prediction direction of arrow (102) - i.e., the samples are predicted from the sample or samples to the upper right at a 45 degree angle from the horizontal. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Sample S44 is then predicted from reference sample R08.

[0015] In certain cases, a value for a reference sample can be combined with neighboring reference samples, e.g., through interpolation, to calculate a reference sample value; especially when the direction does not divide evenly with the 45 degree increments.

[0016] The number of possible directions has increased as video coding technology has developed. For example, in H.264 (year 2003), nine different direction are available for intra prediction. That increased to 33 in H.265 (year 2013), and JEM / VVC / BMS can support up to 65 directions at the time of this disclosure. Experimental research has been done to help identify the most suitable intra prediction directions, and certain techniques in the entropy coding can be used to encode those most suitable directions in a small number of bits, accepting a certain bit penalty for directions. Further, the directions themselves can sometimes be predicted from neighboring directions used in the intra prediction of the already decoded neighboring blocks.

[0017] Figure 1B A diagram (180) depicting 65 intra prediction directions according to JEM is shown to illustrate the increasing number of prediction directions in various coding technologies over time.

[0018] The manner in which bits representing intra prediction directions are mapped to prediction directions in the coded video bitstream can vary between video coding techniques and can range, for example, from simple direct mappings of a prediction direction to an intra prediction mode, to codewords, to complex adaptive schemes involving most probable modes, and the like. In all cases, however, there can be certain directions for intra prediction that are statistically less likely to occur in video content than certain other directions. As the goal of video compression is the reduction of redundancy, in well-designed video coding techniques those less likely directions can be represented by a larger number of bits than those more likely directions.

[0019] Inter picture prediction can be based on motion compensation. In motion compensation, sample data from previously reconstructed pictures or portions thereof (reference pictures) can be spatially shifted in the direction indicated by a motion vector (MV henceforth) and used as a prediction for a newly reconstructed picture or picture portion (e.g. a block). In some cases, the reference picture can be the same as the picture currently under reconstruction. An MV can have two dimensions, X and Y, or three dimensions, the third being an indication of the reference picture in use (similar to the temporal dimension).

[0020] In some video compression techniques, a current MV applicable to a certain region of sample data can be predicted from other MVs, for example from those other MVs related to other regions of the sample data that are spatially adjacent to the region under reconstruction and that precede the current MV in decoding order. In doing so, the total amount of data needed to code the MVs can be reduced, thereby improving compression efficiency, by relying on the removal of redundancy between related MVs. MV prediction can work effectively, for example, because when coding an input video signal derived from a camera (known as natural video), there is a statistical likelihood that regions larger than the region to which a single MV is applicable move in a similar direction in the video sequence and, therefore, in some cases, the MV for a region can be predicted using the MVs derived from neighboring regions. This results in the MV for the given region to be similar or the same as the MV predicted from the surrounding MVs. Such an MV, after entropy coding, can in turn be represented in a smaller number of bits than if the MV were directly coded rather than predicted from the neighboring MV or MVs. In some cases, MV prediction can be an example of lossless compression of a signal (namely: the MVs) derived from the original signal (namely: the sample stream). In other cases, MV prediction itself can be lossy, for example due to rounding errors when calculating a predictor from several surrounding MVs.

[0021] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Out of the many MV prediction mechanisms specified in H.265, the following is described, which is hereafter referred to as "spatial merge."

[0022] In particular, with reference to Figure 2 , the current block (201) comprises samples that have been found by the encoder during a motion search process, which can be predicted from a same-sized previous block that has been spatially shifted. Instead of directly coding this MV, an MV associated with any of the five surrounding samples denoted as A0, Al and B0, Bl, B2 (202 to 206, respectively) can be used, the MV being derived from metadata associated with one or more reference pictures, e.g. from the most recent (in decoding order) reference picture. In H.265, the MV predictor can use predictors from the same reference picture as used by neighboring blocks. SUMMARY

[0023] Aspects of the disclosure provide cross-component methods and apparatuses for selecting and applying a transform of transform coefficients of a cross-component level modification in video processing. In some example implementations, a method for video decoding is disclosed. The method can include receiving a bitstream of a video block comprising a first transform block of a first color component and a second transform block of a second color component, wherein the first transform block and the second transform block are co-located blocks; obtaining the first transform block of the first color component and the second transform block of the second color component from the bitstream of the video block; determining a first flag indicating that all transform coefficients in the first transform block are zero; determining a second flag indicating that a cross-component level reconstruction (CCLR) is applied to the first transform block; and in response to determining that the CCLR is applied to the first transform block; modifying one or more of the transform coefficients in the first transform block by adding one or more offset values to obtain a modified first transform block, the one or more offset values being derived based on transform coefficients in the second transform block and co-located with the one or more of the transform coefficients in the first transform block; determining a target transform kernel for the modified first transform block; performing an inverse transform on the modified first transform block based on the target transform kernel to obtain a target block; and reconstructing the first color component of the video block based at least on the target block.

[0024] Aspects of the disclosure also provide a video encoding or decoding device or apparatus that includes circuitry configured to perform any of the above method implementations.

[0025] Aspects of the disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding and / or encoding, cause the computer to perform a method for video decoding and / or encoding. BRIEF DESCRIPTION OF DRAWINGS

[0026] Further features, the nature, and various advantages of the disclosed subject matter will be more apparent from the following detailed description and the accompanying drawings, in which:

[0027] Figure 1A A diagram shows an exemplary subset of intra prediction directional modes.

[0028] Figure 1B A diagram shows an illustration of exemplary intra prediction directions.

[0029] Figure 2 A diagram shows a spatial merge candidate around a current block and itself for motion vector prediction in one example.

[0030] Figure 3 A diagram shows a simplified block diagram of a communication system (300) in accordance with an example embodiment.

[0031] Figure 4 A diagram shows a simplified block diagram of a communication system (400) in accordance with an example embodiment.

[0032] Figure 5 A diagram shows a simplified block diagram of a video decoder in accordance with an example embodiment.

[0033] Figure 6 A diagram shows a simplified block diagram of a video encoder in accordance with an example embodiment.

[0034] Figure 7 A diagram shows a block diagram of a video encoder in accordance with another example embodiment.

[0035] Figure 8 A diagram shows a block diagram of a video decoder in accordance with another example embodiment.

[0036] Figure 9 A scheme of coding block partitioning in accordance with an example embodiment of the disclosure is shown.

[0037] Figure 10 Another scheme of coding block partitioning in accordance with an example embodiment of the disclosure is shown.

[0038] Figure 11 Another scheme of coding block partitioning in accordance with an example embodiment of the disclosure is shown.

[0039] Figure 12 Another scheme of coding block partitioning in accordance with an example embodiment of the disclosure is shown.

[0040] Figure 13 A scheme of coding block partitioning into multiple transform blocks and coding order of transform blocks in accordance with an example embodiment of the disclosure is shown.

[0041] Figure 14 Another scheme for partitioning an encoding block into multiple transform blocks and the encoding order of transform blocks is shown in accordance with example embodiments of the present disclosure.

[0042] Figure 15 Another scheme for partitioning an encoding block into multiple transform blocks is shown in accordance with example embodiments of the present disclosure.

[0043] Figure 16 A planar rotation transform is illustrated in accordance with example embodiments of the present disclosure.

[0044] Figure 17 Various DCT-2, DCT-4 partial butterfly lookup tables are shown in accordance with example embodiments of the present disclosure.

[0045] Figure 18 A DST-7 partial butterfly lookup table is shown in accordance with example embodiments of the present disclosure.

[0046] Figure 19 An example line graph transform (LGT) is shown in accordance with example embodiments of the present disclosure.

[0047] Figure 20 A flowchart of a method is shown in accordance with example embodiments of the present disclosure.

[0048] Figure 21 A schematic diagram of a computer system is shown in accordance with example embodiments of the present disclosure. DETAILED DESCRIPTION

[0049] Figure 3 A simplified block diagram of a communication system (300) according to an embodiment of the present disclosure is illustrated. The communication system (300) includes multiple terminal devices that can communicate with each other, via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) that are interconnected via the network (350). The terminal devices (310) and (320) can be configured to transmit and / or receive data. Figure 3 In an example, the first pair of terminal devices (310) and (320) can perform unidirectional transmission of data. For example, the terminal device (310) can code video data (e.g., of a stream of video pictures that are captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data can be transmitted in the form of one or more coded video bitstreams. The terminal device (320) can receive the coded video data from the network (350), decode the coded video data to recover the video pictures, and display video pictures according to the recovered video data. Unidirectional data transmission can be implemented in media serving applications etc.

[0050] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that performs bidirectional transmission of encoded video data that can implement, for example, a videoconferencing application. For bidirectional transmission of data, in an example, each terminal device of the terminal devices (330) and (340) can code video data, e.g., of a stream of video pictures captured by the terminal device, for transmission to the other terminal device of the terminal devices (330) and (340) via the network (350). Each terminal device of the terminal devices (330) and (340) also can receive the encoded video data transmitted by the other terminal device of the terminal devices (330) and (340) and can decode the encoded video data to recover the video pictures and can display the video pictures at an accessible display device according to the recovered video data.

[0051] In Figure 3 In an example, the terminal devices (310), (320), (330) and (340) can be implemented as servers, personal computers and smart phones, but the applicability of the principles of the present disclosure can not be limited thereto. Embodiments of the present disclosure can be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, and / or dedicated video conferencing equipment, etc. The network (350) represents any number or type of networks that transmit encoded video data among the terminal devices (310), (320), (330) and (340), including, for example, cable (wired) and / or wireless communication networks. The communication network (350) can exchange data in circuit- switched, packet-switched, and / or other types of channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of the present discussion, the architecture and topology of the network (350) can not be important and will be largely irrelevant to the operation of the present disclosure, except as explicitly noted herein.

[0052] Figure 4 As an example of the application of the disclosed subject matter, the arrangement of a video encoder and a video decoder in a video streaming environment is illustrated. The disclosed subject matter can equally apply to other video applications, including, for example, video conferencing, digital TV broadcast, gaming, virtual reality, storage of compressed video on digital media including CD, DVD, memory stick, etc.

[0053] A video streaming system can include a video capture subsystem (413) that can include a video source (401) such as a digital camera for creating a stream of uncompressed video pictures or images (402). In an example, the stream of video pictures (402) includes samples recorded by the digital camera of video source 401. The stream of video pictures (402), depicted as bold lines to emphasize the high data volume when compared to encoded video data (404) (or coded video bitstreams), can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted as thin lines to emphasize the lower data volume when compared to the stream of uncompressed video pictures (402), can be stored on a streaming server (405) for future use or directly to a downstream video Figure 4 One or more streaming client subsystems (406) and (408) in clients (420) and (430) can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). Client subsystem (406) can include a video decoder (410) in an electronic device (430), for example. Video decoder (410) decodes the incoming copy (407) of encoded video data and creates an outgoing stream (411) of video pictures that are uncompressed and can be presented on a display (412) (e.g., a display screen) or other presentation device (not depicted). The video decoder 410 can be configured to perform some or all of the various functions described in this disclosure. In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) can be encoded according to certain video coding / compression standards. Examples of those standards include ITU-T Recommendation H.265. In an example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter can be used in the context of VVC and other video coding standards.

[0054] It is noted that the electronic devices (420) and (430) can include other components (not shown). For example, the electronic device (420) can include a video decoder (not shown) and the electronic device (430) can include a video encoder (not shown) as well.

[0055] Figure 5A block diagram illustrating a video decoder (510) in accordance with any of the embodiments of the present disclosure is shown. The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., receiving circuitry). The video decoder (510) can be used instead of the video decoder (410) in the example of Figure 4

[0056] The receiver (531) can receive one or more coded video sequences to be decoded by the video decoder (510). In the same or another embodiment, one coded video sequence at a time can be decoded, where the decoding of each coded video sequence is independent of the others. Each video sequence can be associated with a plurality of video frames or pictures. The coded video sequences can be received from a channel (501), which can be a hardware / software link to a storage device that stores the coded video data or to a streaming source that sends the coded video data. The receiver (531) can receive the coded video data along with other data, such as coded audio data and / or ancillary data streams, which can be forwarded to their respective processing circuitries (not depicted). The receiver (531) can separate the coded video sequence from the other data. To combat network jitter, a buffer memory (515) can be placed between the receiver (531) and the entropy decoder / pars er (520) (hereafter “parser (520)”). In certain applications, the buffer memory (515) can be implemented as part of the video decoder (510). In other applications, the buffer memory can be external to and separate from the video decoder (510). In still other applications, there can be a buffer memory (not depicted) external to the video decoder (510), e.g., for the purpose of combating network jitter, and another additional buffer memory (515) internal to the video decoder (510), e.g., for the purpose of handling playback timing. When the receiver (531) receives data from a store / forward device or from a isochronous network with sufficient bandwidth and controllability, the buffer memory (515) can not be needed, or can be small. For use on best effort packet networks, such as the Internet, the buffer memory (515) can need to be of adequate size, and can be relatively large. Such a buffer memory can be implemented with an adaptive size, and can be at least partially implemented in an operating system or a similar element (not depicted) external to the video decoder (510).

[0057] ​The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. These symbols may or may not be part of the electronic device (530), but may be coupled to the electronic device (530), such as… Figure 5 As shown in the diagram. Control information for one or more presentation devices may be in the form of supplemental enhancement information (SEI messages) or video availability information (VUI) parameter set fragments (not depicted). The parser (520) can parse / entropy decode the encoded video sequence received by the parser (520). The entropy coding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a set of subgroup parameters for at least one subgroup of pixels in the encoded video sequence based on at least one parameter corresponding to the subgroup. Subgroups may include picture groups (GOPs), pictures, tiles, stripes, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (520) may also extract information such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc. from the encoded video sequence information.

[0058] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) in order to create symbols (521).

[0059] The reconstruction of the symbol (521) can involve multiple different processing or functional units, depending on the type of the encoded video picture or its portions (such as inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors. The units involved and the manner in which they are involved can be controlled by subgroup control information, which is parsed from the encoded video sequence by the parser (520). For simplicity, such subgroup control information flow between the parser (520) and the multiple processing or functional units is not depicted below.

[0060] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into multiple functional units as described below. In a practical implementation operating under commercial constraints, many of these functional units interact closely with each other and can be at least partially integrated with each other. However, for the sake of clearly describing the various functions of the disclosed subject matter, a conceptual subdivision into functional units is adopted in the following disclosure.

[0061] The first unit can include a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) can receive quantized transform coefficients from the parser (520) as well as control information in the form of one or more symbols (521) including information indicating which type of inverse transform to use, block size, quantization factor / parameter, quantization scaling matrices, etc. The scaler / inverse transform unit (551) can output a block comprising sample values that can be input into the aggregator (555).

[0062] In some cases, the output samples of the scaler / inverse transform (551) can be related to an intra coded block, i.e. a block that is not using prediction information from previously reconstructed pictures, but can use prediction information from previously reconstructed parts of the current picture. Such prediction information can be provided by an intra picture prediction unit (552). In some cases, the intra picture prediction unit (552) can use surrounding block information that has been reconstructed and stored in a current picture buffer (558) to generate a block of the same size and shape as the block under reconstruction. The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator (555) can add the prediction information generated by the intra prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per sample basis.

[0063] In other cases, the output samples of the scaler / inverse transform unit (551) can be related to an inter coded and possibly motion compensated block. In this case, a motion compensated prediction unit (553) can access a reference picture memory (557) to fetch samples for inter prediction. After motion compensation of the fetched samples according to the symbols (521) pertaining to the block, these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (the output of unit 551 can be referred to as residual samples or residual signal) in order to generate output sample information. The addresses within the reference picture memory (557) from which the prediction samples are fetched by the motion compensated prediction unit (553) can be controlled by motion vectors that are available to the motion compensated prediction unit (553) in the form of symbols (521) that can have, for example, X, Y components (shift) and reference picture component (time). Motion compensation can also include interpolation of sample values fetched from the reference picture memory (557) when sub-pixel precise motion vectors are used, and can also be associated with motion vector prediction mechanisms, etc.

[0064] The output samples of the aggregator (555) can be subject to various loop filtering techniques in the loop filter unit (556). Video compression technologies can include in-loop filter technologies that are controlled by parameters included in the coded video sequence (also referred to as coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but can also be responsive to meta-information obtained during the decoding of previous (in decoding order) parts of the coded picture or coded video sequence, as well as responsive to previously reconstructed and loop-filtered sample values. Several types of loop filters can be included as part of the loop filter unit 556 in various orders, as will be described in further detail below.

[0065] The output of the loop filter unit (556) can be a stream of samples that can be output to the rendering device (512) and stored in the reference picture memory (557) for use in future inter prediction.

[0066] Some coded pictures, once fully reconstructed, can be used as reference pictures for future inter prediction. The current picture buffer (558), for example, can become part of the reference picture memory (557) once the corresponding coded picture has been fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser (520), and a new current picture buffer can be reallocated before commencing the reconstruction of a subsequent coded picture.

[0067] The video decoder (510) can perform decoding operations according to a predetermined video compression technology adopted in a standard, such as ITU-T Rec. H.265. The coded video sequence can conform to a syntax specified by the video compression technology or standard being used, in the sense that it complies with the video compression technology or standard’s syntax, as well as a profile and level of the video compression technology or standard specified by the video compression technology or standard. Specifically, a profile can select certain tools made available under that profile from all tools available in the video compression technology or standard, as a function of level, as some tools can be available in a level of a profile only. To be compliant with the standard, the complexity of the coded video sequence can be within bounds defined by the level. In some cases, the level restricts the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the bounds set by the level can be further restricted by Hypothetical Reference Decoder (HRD) specifications and metadata for HRD buffer management that are signaled in the coded video sequence.

[0068] In some example embodiments, the receiver (531) can receive additional (redundant) data with the encoded video. The additional data can be included as part of the coded video sequence(s). The additional data can be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data can be in the form of temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, in the form of redundant slices, in the form of redundant pictures, in the form of forward error correction codes, or the like.

[0069] Figure 6 A block diagram of a video encoder (603) according to an example embodiment of the present disclosure is shown. The video encoder (603) can be included in an electronic device (620). The electronic device (620) can further include a transmitter (640) (e.g., transmitting circuitry). The video encoder (603) can be used in the place of the video encoder (403) in the example of Figure 4

[0070] The video encoder (603) can receive video samples from a video source (601) (that is not part of the electronic device (620) in the example of Figure 6 which is to be encoded by the video encoder (603). In another example, the video source (601) can be implemented as part of the electronic device (620).

[0071] The video source (601) can provide the source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream that can be of any suitable bit depth (for example: 8 bit, 10 bit, 12 bit,...), any color space (for example, BT.601 Y CrCB, RGB, XYZ,...), and any appropriate sampling structure (for example, Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) can be a storage device capable of storing previously prepared video. In a videoconferencing system, the video source (601) can be a camera that captures local image information as a video sequence. Video data can be provided as a plurality of individual pictures or images that impart motion when viewed in sequence. The pictures themselves can be organized as a spatial array of pixels, wherein each pixel can include one or more samples depending on the sampling structure, color space, etc. in use. A person having ordinary skill in the art can readily understand the relationship between pixels and samples. The following description focuses on samples.

[0072] ​According to some example embodiments, the video encoder (603) can encode and compress images of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate constitutes a function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units as described below. For simplicity, coupling is not depicted. Parameters set by the controller (650) may include parameters related to rate control (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of images (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions related to the video encoder (603) optimized for a particular system design.

[0073] In some example embodiments, the video encoder (603) may be configured to operate within an encoding loop. As an oversimplification, in this example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols such as a symbol stream based on the input image to be encoded and one or more reference images), and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to that created by the (remote) decoder, even though the embedded decoder 633 processes the encoded video stream through the source encoder 630 without entropy coding (because any compression between the symbols and the encoded video stream in entropy coding is likely lossless in video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is fed into a reference image memory (634). Since the decoding of the symbol stream results in bit-accurate results regardless of the decoder location (local or remote), the contents of the reference image memory (634) are also bit-accurate between the local and remote encoders. In other words, when prediction is used during decoding, the encoder's prediction portion, as a reference picture sample, "sees" the exact same sample values ​​as the decoder. The fundamental principle of reference picture synchronization (and, if synchronization cannot be maintained, such as due to channel errors, drift) is used to improve coding quality.

[0074] The operation of the "local" decoder (633) can be the same as that of the "remote" decoder, as has been combined above. Figure 5 A detailed description of the video decoder (510). However, a brief reference is also available. Figure 5Since symbols are available and encoding / decoding symbols into an encoded video sequence via an entropy encoder (645) and a parser (520) can be lossless, the entropy decoding portion of the video decoder (510) including the buffer (515) and the parser (520) may not be fully implemented in the local decoder (633) within the encoder.

[0075] At this point, it can be observed that any decoder technique other than parsing / entropy decoding, which may only exist in the decoder, may necessarily need to exist in the corresponding encoder in essentially the same functional form. For this reason, the topics disclosed may sometimes focus on decoder operations related to the decoding part of the encoder. The descriptions of encoder techniques can thus be simplified, in contrast to the comprehensive descriptions of decoder techniques. A more detailed description of the encoder is provided below only in certain areas or aspects.

[0076] During operation in some example implementations, the source encoder (630) may perform motion-compensated predictive coding, which predictively encodes the input image by referencing one or more previously encoded images from the video sequence designated as "reference images." In this way, the encoding engine (632) encodes the differences (or residuals) in the color channels between pixel blocks of the input image and pixel blocks of one or more reference images that can be selected as one or more predictive references to the input image. The term "residual" and its adjective form "residual" are used interchangeably.

[0077] The local video decoder (633) can decode encoded video data of a picture that can be designated as a reference picture based on symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the video decoder (633) Figure 6 When decoding encoded video data at a location (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process performed by the video decoder on the reference picture and can store the reconstructed reference picture in a reference picture cache (634). In this way, the video encoder (603) can locally store copies of the reconstructed reference pictures that have the same content (without transmission errors) as the reconstructed reference pictures that will be obtained by the remote video decoder.

[0078] The predictor (635) can perform a search to predict the new pictures to be coded by the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, and so on, that can be used as appropriate prediction references for the new pictures. The predictor (635) can operate on a pixel block-by-pixel block basis to find appropriate prediction references. In some cases, as determined by search results obtained by the predictor (635), an input picture can have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0079] The controller (650) can manage the encoding operations of the source coder (630), including, for example, setting of parameters and sub-group parameters used in the encoding of the video data.

[0080] The outputs of all the above-described functional units can be entropy encoded in the entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into an encoded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, and so on.

[0081] The transmitter (640) can buffer the one or more coded video sequences created by the entropy coder (645) to prepare them for transmission via a communication channel (660), which can be a hardware / software link to a storage device that will store the encoded video data. The transmitter (640) can merge coded video data from the video coder (603) with other data (e.g., coded audio data and / or ancillary data streams (sources not shown)) to be transmitted.

[0082] The controller (650) can manage operation of the video encoder (603). During coding, the controller (650) can assign to each coded picture a certain coded picture type, which can affect the coding techniques that can be applied to the respective picture. For example, pictures often can be assigned to one of the following picture types:

[0083] An Intra Picture (I picture) can be one that can be coded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of Intra pictures, including, for example Independent Decoder Refresh (“IDR”) pictures. A person of ordinary skill in the art is aware of these variants of I pictures and their respective applications and features.

[0084] A Predictive Picture (P picture) can be one that can be coded and decoded using either intra or inter prediction, which uses at most one motion vector and reference index to predict sample values for each block.

[0085] Bidirectional predictive pictures (B-pictures) can be pictures that can be encoded and decoded using intra prediction or inter prediction that uses up to two motion vectors and reference indices to predict sample values of each block. Similarly, multi-predicted pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

[0086] Source pictures can typically be spatially subdivided into a plurality of sample- encoded blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded on a block-by-block basis. A block can be predictively encoded with reference to other (already encoded) blocks determined by the encoding assignment of the respective picture to which the block is applied. For example, blocks of I-pictures can be non-predictively encoded, or they can be predictively encoded with reference to already encoded blocks of the same picture (spatial or intra prediction). Blocks of P-pictures can be predictively encoded with reference to one previously encoded reference picture either by spatial prediction or by temporal prediction. Blocks of B-pictures can be predictively encoded with reference to one or two previously encoded reference pictures either by spatial prediction or by temporal prediction. Source pictures or pictures under intermediate processing can be subdivided into other types of blocks for other purposes. The partitioning of the encoding blocks and other types of blocks can or can not follow the same ways, as described in further detail below.

[0087] The video encoder (603) can perform encoding operations in accordance with a predetermined video coding technology or standard, such as ITU-T Rec. H.265. In its operation, the video encoder (603) can perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. Accordingly, the encoded video data can conform to a syntax specified by the video coding technology or standard in use.

[0088] In some example embodiments, the transmitter (640) can transmit additional data with the encoded video. The source coder (630) can include such data as part of the coded video sequence. The additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and so on.

[0089] A video can be captured as a time sequence of multiple source pictures (video pictures). Intra prediction, often abbreviated as I-picture, exploits spatial correlation in a given picture, and inter prediction exploits temporal or other correlation between pictures. For example, a particular picture under encoding / decoding, referred to as a current picture, can be partitioned into blocks. When a block in the current picture is similar to a reference block in a reference picture that has been previously encoded and is still buffered, the block in the current picture can be encoded by a vector, referred to as a motion vector. In case multiple reference pictures are used, the motion vector points to the reference block in a reference picture, and can have a third dimension identifying the reference picture.

[0090] In some example embodiments, a bi-prediction technique can be used for inter prediction. According to such bi-prediction technique, two reference pictures, such as a first reference picture and a second reference picture, are used, both preceding the current picture in the video in decoding order (but can be in the past or future respectively in display order). A block in the current picture can be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block can be jointly predicted by a combination of the first reference block and the second reference block.

[0091] Further, a merge mode technique can be used in inter prediction to improve coding efficiency.

[0092] According to some example embodiments of the disclosure, predictions such as inter-prediction and intra-prediction are performed in units of blocks. For example, a picture in a video sequence is partitioned into coding tree units (CTUs) for compression, the CTUs in a picture can have the same size such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU can include three parallel coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree- split into one or more coding units (CUs). For example, a 64x64-pixel CTU can be split into one 64x64-pixel CU, or 4 32x32-pixel CUs. Each of the one or more 32x32 blocks can be further split into 4 16x16-pixel CUs. In some example embodiments, each CU can be analyzed during encoding to determine a prediction type for the CU among various prediction types such as an inter-prediction type or an intra-prediction type. Depending on the temporal and / or spatial predictability, the CU can be split into one or more prediction units (PUs). Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. The split of CUs into PUs (or PBs of different color channels) can be performed in various spatial patterns. For example, a luma or chroma PB can include a matrix of sample values (e.g., luma values) such as 8x8 pixels, 16x16 pixels, 8x16 samples, 16x8 samples, and so on.

[0093] Figure 7 A diagram of a video encoder (703) according to another example embodiment of the disclosure is shown. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values within a current video picture in a video sequence, and encode the processing block into a coded picture that is part of a coded video sequence. The example video encoder (703) can be used in place of the video encoder (403) in the example. Figure 4 The video encoder (403) in the example.

[0094] For example, a video encoder (703) receives a sample value matrix of a processing block (such as an 8×8 sample prediction block). The video encoder (703) then determines whether the processing block is best encoded using intra-frame mode, inter-frame mode, or bidirectional prediction mode such as Rate-Distortion Optimization (RDO). When the processing block is determined to be encoded in intra-frame mode, the video encoder (703) can encode the processing block into an encoded picture using intra-frame prediction techniques; and when the processing block is determined to be encoded in inter-frame mode or bidirectional prediction mode, the video encoder (703) can encode the processing block into an encoded picture using inter-frame prediction or bidirectional prediction techniques, respectively. In some example embodiments, a merged mode can be used as a sub-mode of inter-frame prediction, where motion vectors are derived from one or more motion vector predictors without the benefit of encoded motion vector components outside the predictors. In some other example embodiments, motion vector components applicable to subject blocks may exist. Accordingly, the video encoder (703) may include... Figure 7 Components not explicitly shown in the document (such as the pattern decision module) are used to determine the conventional pattern of the processing block.

[0095] exist Figure 7 In the example, the video encoder (703) includes an inter-frame encoder (730), an intra-frame encoder (722), a residual encoder (723), a switch (726), a residual encoder (724), a general-purpose controller (721), and an entropy encoder (725) coupled together, as... Figure 7 The example layout is shown in the image.

[0096] The inter-frame encoder (730) is configured to receive samples of the current block (e.g., the processing block), compare that block with one or more reference blocks in a reference image (e.g., blocks in previous and subsequent images in display order), generate inter-frame prediction information (e.g., descriptions of redundancy information based on inter-frame coding techniques, motion vectors, merging mode information), and compute inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some examples, the reference image is used using embedded... Figure 6 Example encoder 620 (shown for) Figure 7 The decoding unit 633 in the residual decoder 728 (described in further detail below) decodes a decoded reference image based on encoded video information.

[0097] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block to blocks that have been encoded in the same picture, and generate quantized coefficients after a transform, and in some cases also intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). The intra encoder (722) can calculate an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same picture.

[0098] The general controller (721) can be configured to determine general control data and control other components of the video encoder (703) based on the general control data. In an example, the general controller (721) determines a prediction mode of a block, and provides a control signal to the switch (726) based on the prediction mode. For example, when the prediction mode is an intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select and include the intra prediction information in the bitstream; and when the prediction mode of the block is an inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select and include the inter prediction information in the bitstream.

[0099] The residual calculator (723) can be configured to calculate a difference (residual data) between a received block and a prediction result of the block selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) can be configured to encode the residual data to generate transform coefficients. For example, the residual encoder (724) can be configured to convert the residual data from a spatial domain to a frequency domain to generate transform coefficients. The transform coefficients are then subject to quantization processing to obtain quantized transform coefficients. In various example embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform inverse transform, and generate decoded residual data. The decoded residual data can be used by the intra encoder (722) and the inter encoder (730) as appropriate. For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and intra prediction information. The decoded block is processed as appropriate to generate a decoded picture, and the decoded picture can be buffered in a memory circuit (not shown) and used as a reference picture.

[0100] The entropy encoder (725) can be configured to format the bitstream to include the coded blocks and perform entropy encoding. The entropy encoder (725) is configured to include various information in the bitstream. For example, the entropy encoder (725) can be configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other suitable information in the bitstream. When a block is coded in inter mode or merge submode of bi-predictive mode, there can be no residual information.

[0101] Figure 8 A diagram illustrates an example video decoder (810) according to another embodiment of the disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence, and decode the coded pictures to generate reconstructed pictures. In an example, the video decoder (810) can be used in place of the video decoder (410) in the example of FIG. 4. Figure 4

[0102] In the example of FIG. 4, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residue decoder (873), a reconstruction module (874), and an intra decoder (872) coupled together as shown in the example arrangement of FIG. 8. Figure 8 Figure 8 In the example of FIG. 4, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residue decoder (873), a reconstruction module (874), and an intra decoder (872) coupled together as shown in the example arrangement of FIG. 8.

[0103] The entropy decoder (871) can be configured to reconstruct, from the coded picture, certain symbols that represent syntax elements that make up the coded picture. Such symbols can include, for example, modes in which blocks are coded (e.g., intra mode, inter mode, bi-predictive mode, merge submode, or another submode), prediction information (e.g., intra prediction information or inter prediction information) that can identify certain samples or metadata that an intra decoder (872) or inter decoder (880) uses for prediction, residual information in the form of, for example, quantized transform coefficients, and the like. In an example, when the prediction mode is inter or bi-predictive mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can be subject to inverse quantization and provided to the residue decoder (873).

[0104] The inter decoder (880) can be configured to receive the inter prediction information and generate inter prediction results based on the inter prediction information.

[0105] The intra decoder (872) can be configured to receive the intra prediction information and generate prediction results based on the intra prediction information.

[0106] ​​The residue decoder (873) can be configured to perform inverse quantization to extract de-quantized transform coefficients and process the de-quantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residue decoder (873) can also utilize certain control information (including the quantizer parameter (QP)) that can be provided by the entropy decoder (871) (data paths not depicted as this can be low data volume control information only).

[0107] The reconstruction module (874) can be configured to combine, in the spatial domain, the prediction results (output by the inter or intra prediction modules, as the case can be) and the residual (output by the residue decoder (873)) to form a reconstructed block that forms a part of a reconstructed picture that forms part of a reconstructed video. It should be noted that other appropriate operations, such as a de-blocking operation, etc., can also be performed to provide visual quality.

[0108] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented using any suitable technique. In some example embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) can be implemented using one or more processors that execute software instructions.

[0109] For coding block partitioning, and in some example implementations, a predetermined pattern can be applied. As shown in Figure 9 An example 4-way partitioning tree can be employed starting from a first predetermined level (e.g., 64x64 block level) down to a second pre-defined level (e.g., 4x4 level). For example, a base block can depend on four partitioning options indicated by 902, 904, 906, and 908, where a partition designated as R is allowed for recursive partitioning in which the same partitioning tree as Figure 9 indicated in can be repeated at lower levels until the lowest level (e.g., 4x4 level). In some implementations, additional restrictions can be applied to the partitioning scheme of Figure 9 In implementations of Figure 9 , rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) can be allowed, but can not be allowed to be recursive, but square partitions are allowed to be recursive. If desired, the recursive partitioning of Figure 9 generates a final set of coding blocks. Such a scheme can be applied to one or more color channels.

[0110] Figure 10 Another example pre-defined partitioning pattern is shown that allows recursive partitioning to form a partitioning tree. As shown inFigure 10 As shown in FIG. 10, an example 10-way partition structure or pattern can be predefined. The root block can start at a predefined level (e.g., from a 128x128 level, or a 64x64 level). Figure 10 The example partition structure of FIG. 10 includes various 2: 1 / 1:2 and 4: 1 / 1:4 rectangular partitions. Figure 10 The partition types with 3 sub-partitions indicated as 1002, 1004, 1006, and 1008 in the second row of FIG. 10 can be referred to as "T-shaped" partitions. The "T-shaped" partitions 1002, 1004, 1006, and 1008 can be referred to as left T-shaped, top T-shaped, right T-shaped, and bottom T-shaped. In some implementations, Figure 10 None of the rectangular partitions of FIG. 10 are allowed to be further subdivided. A coding tree depth can be further defined to indicate the depth of partitioning from the root node or root block. For example, the coding tree depth of the root node or root block (e.g., a 128x128 block) can be set to 0, and the root block can follow Figure 10 The coding tree depth is increased by 1 after the square partition in 1010 is further partitioned once. In some implementations, only the square partition in 1010 can be allowed to follow Figure 10 The pattern recursive partitioning to the next level of the partition tree. In other words, for the square partitions with patterns 1002, 1004, 1006, and 1006, no recursive partitioning can be allowed. If needed, the square partitions with patterns 1002, 1004, 1006, and 1006 can follow Figure 10 The recursive partitioning generates a final set of coding blocks. Such a scheme can be applied to one or more color channels.

[0111] After the base block is divided or partitioned following any of the above partitioning processes or other processes, likewise, a final set of partitioned or coded blocks can be obtained. Each of these partitions can be at one of various partition levels. Each of the partitions can be referred to as a coding block (CB). For the above various example partitioning implementations, each resulting CB can have any of the allowed sizes and partition levels. They are referred to as coding blocks because they can form the units on which some basic encoding / decoding decisions can be made, and the encoding / decoding parameters can be optimized, determined, and signaled in the coded video bitstream. The highest level in the final partitions represents the depth of the coding block partition tree. The coding blocks can be luma coding blocks or chroma coding blocks.

[0112] In some other example implementations, a quad-tree structure can be used to recursively partition the base luma and chroma blocks into coding units. Such a partitioning structure can be referred to as a coding tree unit (CTU), which is partitioned into coding units (CUs) using a quad-tree structure to adapt the partitioning to various local characteristics of the base CTU. In such implementations, an implicit quad-tree partitioning can be performed at picture boundaries so that the blocks will remain quad-tree partitioned until the size fits the picture boundary. The term CU is used to refer to the unit of luma and chroma coding blocks (CBs).

[0113] In some implementations, a CB can be further partitioned. For example, a CB can be further partitioned into a plurality of prediction blocks (PBs) for the purpose of intra- or inter-prediction during the encoding and decoding processes. In other words, a CB can be further divided into different sub-partitions, where separate prediction decisions / configurations can be made. At the same time, a CB can be further partitioned into a plurality of transform blocks (TBs) for delineating the stages where transforms or inverse transforms of video data are performed. The partitioning schemes of CBs to PBs and TBs can be the same or different. For example, each partitioning scheme can be performed using its own procedure based on, for example, various characteristics of the video data. In some example implementations, the PB and TB partitioning schemes can be independent. In some other example implementations, the PB and TB partitioning schemes and boundaries can be related. In some implementations, for example, TBs can be partitioned after PB partitioning, and specifically, each PB can be further partitioned into one or more TBs after the partitioning of the coding block is determined. For example, in some implementations, a PB can be split into one, two, four, or other number of TBs.

[0114] In some implementations, the luma channel and the chroma channels can be treated differently for partitioning of a base block into coding blocks and further partitioning into prediction blocks and / or transform blocks. For example, in some implementations, partitioning of coding blocks into prediction blocks and / or transform blocks can be allowed for the luma channel, while partitioning of coding blocks into prediction blocks and / or transform blocks can not be allowed for one or more chroma channels. In such implementations, the transform and / or prediction of luma blocks can thus be performed only at the coding block level. For another example, the minimum transform block size for the luma channel and one or more chroma channels can be different, e.g., coding blocks of the luma channel can be partitioned into transform blocks and / or prediction blocks that are smaller than the chroma channels. For yet another example, the maximum depth of partitioning of coding blocks into transform blocks and / or prediction blocks can be different between the luma channel and the chroma channels, e.g., coding blocks of the luma channel can be partitioned into transform blocks and / or prediction blocks that are deeper than the one or more chroma channels. For a particular example, luma coding blocks can be partitioned into transform blocks of multiple sizes, which can be represented by recursive partitioning down to up to 2 levels, and transform block shapes such as square, 2: 1 / 1:2, and 4: 1 / 1:4, and transform block sizes from 4x4 to 64x64 can be allowed. However, for chroma blocks, only the maximum possible transform blocks specified for luma blocks are allowed.

[0115] In some example implementations for partitioning of coding blocks into PBs, the depth, shape, and / or other characteristics of the PB partitioning can depend on whether the PB is intra- or inter-coded.

[0116] Partitioning of a coding block (or prediction block) into transform blocks can be implemented in various example schemes, including but not limited to recursive or non-recursive quadtree partitioning and pre-defined mode partitioning, and additional considerations are made for transform blocks at the boundaries of the coding block or prediction block. In general, the resulting transform blocks can be at different partition levels, can not have the same size, and can not need to be square in shape (e.g., they can be rectangular with some allowed sizes and aspect ratios).

[0117] In some implementations, a coding partition tree scheme or structure can be used. The coding partition tree scheme for the luma and chroma channels can not necessarily be the same. In other words, the luma channel and the chroma channels can have separate coding tree structures. Further, whether the luma channel and the chroma channels use the same or different coding partition tree structures and the actual coding partition tree structure to use can depend on whether the slice being coded is a P, B, or I slice. For example, for I slices, the chroma and luma channels can have separate coding partition tree structures or coding partition tree structure modes, while for P or B slices, the luma and chroma channels can share the same coding partition tree scheme. When separate coding partition tree structures or modes are applied, the luma channel can be partitioned into CBs by one coding partition tree structure and the chroma channel can be partitioned into chroma CBs by another coding partition tree structure.

[0118] The following describes a particular example implementation of coding block and transform block partitioning. In such an example implementation, the base coding block can be partitioned into coding blocks using the recursive quadtree partitioning described above. At each level, whether further quadtree partitioning of a particular partition should continue can be determined by local video data characteristics. The resulting CBs can be at various quadtree partitioning levels of various sizes. A decision can be made at the CB level (or CU level, for all three color channels) as to whether to use inter-picture (temporal) or intra-picture (spatial) prediction to code the picture region. Each CB can be further partitioned into one, two, four, or other number of PBs according to the PB partition type. Within a PB, the same prediction process can be applied, and the relevant information is sent to the decoder on a PB basis. After obtaining the residual block by applying the prediction process based on the PB partition type, the CB can be partitioned into TBs according to another quadtree structure similar to the coding tree of the CB. In this particular implementation, the CBs or TBs can but need not be limited to square shape. Further, in this particular example, the PBs can be square or rectangular in shape for inter prediction and only square for intra prediction. The coding blocks can be further partitioned into, for example, four square TBs. Each TB can be further recursively partitioned (using quadtree partitioning) into smaller TBs, referred to as a residual quadtree (RQT).

[0119] The following describes another specific example of partitioning a base-coded block into CBs and other PBs and / or TBs. For example, instead of using, etc. Figure 10 Instead of the multi-partition unit type shown, a quadtree with nested multi-type trees can be used, utilizing binary and ternary partitioning segmentation structures. The separation of the concepts CB, PB, and TB (i.e., partitioning a CB into PB and / or TB, and partitioning a PB into TB) can be abandoned unless a CB of excessive size for the maximum transform length is required, where such a CB might need further partitioning. This example partitioning scheme can be designed to support greater flexibility in the shape of CB partitions, allowing prediction and transform to be performed at the CB level without further partitioning. In such a coding tree structure, the CB can have a square or rectangular shape. Specifically, the coding tree block (CTB) can first be partitioned by a quadtree structure. Then, the quadtree leaf nodes can be further partitioned by a multi-type tree structure. An example of a multi-type tree structure is shown in... Figure 11 shown in . Specifically, Figure 11 The example multi-type tree structure includes four partition types, referred to as vertical binary partition (SPLIT_BT_VER) (1102), horizontal binary partition (SPLIT_BT_HOR) (1104), vertical ternary partition (SPLIT_TT_VER) (1106), and horizontal ternary partition (SPLIT_TT_HOR) (1108). The CB then corresponds to the leaves of the multi-type tree. In this example implementation, the partition is used for prediction and transformation processing without any further partitioning unless the CB is too large for the maximum transform length. This means that, in most cases, the CB, PB, and TB have the same block size in a quadtree with a nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is less than the width or height of the color component of the CB.

[0120] Figure 12 The image shows an example of a quadtree with a nested multi-type tree-encoded block structure for block partitioning of a CTB. More detailed... Figure 12 The diagram shows the CTB 1200 partitioned into four square partitions, 1202, 1204, 1206, and 1208, by a quadtree. Further uses are described for each quadtree partition. Figure 11 The decision to split is based on multiple types of tree structures. Figure 12 In the example, partition 1204 was not further divided. Partitions 1202 and 1208 were each divided using another quadtree. For partition 1202, the top-left, top-right, bottom-left, and bottom-right partitions of the second-level quadtree were divided using the third-level quadtree, respectively. Figure 11 1104, non-segmentation and Figure 11Partition 1108. Partition 1208 is partitioned using another quadtree, and the top-left, top-right, bottom-left, and bottom-right partitions of the second-level quadtree are respectively partitioned using... Figure 11 The third-level segmentation 1106, non-segmentation, non-segmentation and Figure 11 Partition 1104. The two sub-partitions within the third-level upper-left partition 1208 are further divided based on 1104 and 1108. Partition 1206 follows... Figure 11 The second-level partitioning pattern of 1102 is divided into two partitions, which are based on... Figure 11 1108 and 1102 were further divided into three levels. According to... Figure 11 Of 1104, the fourth-level segmentation was further applied to one of them.

[0121] For the specific example above, the maximum luma transform size can be 64×64, and the maximum supported chroma transform size can differ from, for example, the luma at 32×32. When the width or height of the luma-coded block or chroma-coded block is greater than the maximum transform width or height, the luma-coded block or chroma-coded block can be automatically segmented in the horizontal and / or vertical directions to meet the transform size limit in that direction.

[0122] In the specific example used to partition the base coding block into the above CBs, the coding tree scheme can support the ability for luma and chroma to have separate block tree structures. For example, for P and B stripes, the luma and chroma CTBs in a CTU can share the same coding tree structure. For example, for I stripes, luma and chroma can have separate coding block tree structures. When the separate block tree mode is applied, the luma CTB can be partitioned into luma CBs using one coding tree structure, and the chroma CTB can be partitioned into chroma CBs using another coding tree structure. This means that a CU in an I stripe can consist of coding blocks for the luma component or coding blocks for the two chroma components, and a CU in a P or B stripe always consists of coding blocks for all three color components, unless the video is monochrome.

[0123] Example implementations for partitioning a coding block or prediction block into transform blocks and the coding order of transform blocks are described in further detail below. In some example implementations, transform partitioning can support transform blocks of multiple shapes (e.g., 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1) with transform block sizes ranging from, for example, 4x4 to 64x64. In some implementations, if a coding block is smaller than or equal to 64x64, transform block partitioning can only be applied to the luma component, such that for chroma blocks, the transform block size is the same as the coding block size. Otherwise, if the coding block width or height is larger than 64, both the luma coding block and the chroma coding block can be implicitly partitioned into multiple transform blocks of min(W, 64)xmin(H, 64) and min(W, 32)xmin(H, 32), respectively.

[0124] In some example implementations, for both intra-coded blocks and inter-coded blocks, a coding block can be further partitioned into multiple transform blocks with a partition depth of up to a pre-defined number of levels (e.g., 2 levels). The transform block partition depth and size can be related. An example mapping from a transform size of a current depth to a transform size of a next depth is shown in Table 1 below.

[0125] Table 1: Transform partition size settings

[0126]

[0127] Based on the example mapping of Table 1, for a 1:1 square block, a next level transform split can create four 1:1 square sub-transform blocks. The transform partitioning can stop, for example, at 4x4. In this way, a transform size of a current depth of 4x4 corresponds to a next depth of the same size of 4x4. In the example of Table 1, for a 1:2 / 2:1 non-square block, a next level transform split will create two 1:1 square sub-transform blocks, and for a 1:4 / 4:1 non-square block, a next level transform split will create two 1:2 / 2:1 sub-transform blocks.

[0128] In some example implementations, for the luma component of an intra-coded block, additional restrictions can be applied. For example, for each level of transform partitioning, all sub-transform blocks can be restricted to have equal sizes. For example, for a 32x16 coding block, a level 1 transform split creates two 16x16 sub-transform blocks, and a level 2 transform split creates eight 8x8 sub-transform blocks. In other words, a second level split must be applied to all first level sub-blocks to maintain equal transform unit sizes. In some example implementations, the transform partitioning can be applied to the luma component of an intra-coded block only if the coding block size is larger than 64x64. Otherwise, the transform block size is the same as the coding block size. Figure 13An example of transform block partitioning for an intra coded square block of Table 1 below is shown in 1300 along with the coding order illustrated by the arrows. Specifically, 1302 shows a square coded block. The first level partitioning into 4 equal size transform blocks according to Table 1 is shown in 1304 in the coding order indicated by the arrows. In 1306, the second level partitioning of all first level equal size blocks into 16 equal size transform blocks according to Table 1 is shown in the coding order indicated by the arrows.

[0129] In some example implementations, for the luma component of an inter coded block, the above restrictions for intra coding can not be applied. For example, after the first level transform partitioning, any of the sub-transform blocks can be further independently partitioned one level. Thus, the resulting transform blocks can or can not have the same size. Figure 14 An example of partitioning an inter coded block into transform blocks with their coding order is shown in 1400. In this example, the inter coded block 1402 is partitioned into two levels of transform blocks according to Table 1. At the first level, the inter coded block is partitioned into four equal size transform blocks. Then, as shown in 1404, only one (not all) of the four transform blocks is further partitioned into four sub-transform blocks, resulting in a total of 7 transform blocks having two different sizes. An example coding order of these 7 transform blocks is shown by the arrows in 1404. Figure 14 Figure 14

[0130] In some example implementations, for one or more chroma components, some additional restrictions on transform blocks can be applied. For example, for one or more chroma components, the transform block size can be as large as the coded block size, but not smaller than a predefined size, e.g., 8x8.

[0131] In some other example implementations, for coded blocks with width (W) or height (H) larger than 64, both the luma and chroma coded blocks can be implicitly partitioned into multiple min(W, 64)xmin(H, 64) and min(W, 32)xmin(H, 32) transform units, respectively.

[0132] Figure 15 A further example scheme for partitioning a coded block or prediction block into transform blocks is shown in 1500. As shown in 1500, instead of using recursive transform partitioning, a predefined set of partition types can be applied to the coded block according to the transform type of the coded block. In the specific example shown in 1500, one of six example partition types can be applied to partition the coded block into various numbers of transform blocks. Such a scheme can be applied to coded blocks or prediction blocks. Figure 15 Figure 15 In more detail, the partitioning scheme of 1500 can be applied to a coded block or prediction block as follows. In 1502, a coded block or prediction block is partitioned into one or more transform blocks according to the transform type of the coded block or prediction block. In 1504, a partition type is selected from a predefined set of partition types according to the transform type of the coded block or prediction block. In 1506, the coded block or prediction block is partitioned into one or more transform blocks according to the selected partition type.

[0133] Figure 15 Figure 15 ​​​​​Up to 6 partition types are provided for any given transform type as illustrated in In this scheme, a transform type can be assigned to each coding or prediction block based on, for example, rate-distortion cost. In an example, the partition type assigned to a coding or prediction block can be determined based on the transform partition type of the coding or prediction block. A particular partition type can correspond to a transform block partition size and mode (or partition type) as illustrated in Figure 15 The 4 partition types illustrated in An example correspondence is shown below, where the capital letters indicate the transform type that can be assigned to a coding or prediction block based on rate-distortion cost:

[0134] • PARTITION_NONE: Assign a transform size equal to the block size.

[0135] • PARTITION_SPLIT: Assign a transform size with width equal to 1 / 2 of the block size and height equal to 1 / 2 of the block size.

[0136] • PARTITION_HORZ: Assign a transform size with width equal to the block size and height equal to 1 / 2 of the block size.

[0137] • PARTITION_VERT: Assign a transform size with width equal to 1 / 2 of the block size and height equal to the block size.

[0138] • PARTITION_HORZ4: Assign a transform size with width equal to the block size and height equal to 1 / 4 of the block size.

[0139] • PARTITION_VERT4: Assign a transform size with width equal to 1 / 4 of the block size and height equal to the block size.

[0140] In the above example, all the partition types as illustrated in Figure 15 The partition types as illustrated in

[0141] Turning to primary transforms, an example 2-D (two-dimensional) transform process can involve using a hybrid transform kernel (e.g., which can be composed of different 1-D (one-dimensional) transforms for each dimension of the coded residual block), as well as using the same transform kernel for both dimensions. Example primary 1-D transform kernels can include, but are not limited to: a) 4-point (4p), 8-point (8p), 16-point (16p), 32-point (32p), and 64-point (64p) DCT-2; b) 4-point, 8-point, 16-point asymmetric DST and their flipped versions (DST denotes Discrete Sine Transform); c) 4-point, 8-point, 16-point, or 32-point identity transform; d) incremental distance transform (IDT). Thus, a 2-D transform process can involve using a hybrid transform or transform kernel (different transform for each dimension of the coded residual block), where the selection of the transform or transform kernel to use for each dimension can be based on rate-distortion (RD) criteria. The term "transform kernel" can be alternatively referred to as a transform basis function. For example, basis functions for a hybrid of 1-D DCT-2, DST-4, and DST-7 that can be implemented as a 2-D transform are listed in Table 2 (where DCT denotes Discrete Cosine Transform).

[0142] Table 2 Example primary transform basis functions (DCT-2, DST-4, and DST-7 for N-point input)

[0143]

[0144] For example, DCT-2 (4p-64p), DST-4 (8p, 16p), and DST-7 (4p) transforms exhibit symmetric / anti-symmetric properties, and thus in some example implementations a "partial butterfly" implementation can be supported to reduce the number of operation counts (multiplications, additions / subtractions, shifts). A partial butterfly implementation can involve using planar rotations of trigonometric sine and cosine functions at various angles as described in Figure 16 "Fast Algorithms for Digital Signal Processing," by D. H. Dudgeon and R. M. Merserau, 1983, Prentice-Hall, Inc., pages 112-114. Example 12-bit lookup tables are shown in Figure 17 and Figure 18 and can be used to generate values of the trigonometric functions. Specifically, Figure 17 example DCT-2 (4p-64p) / DST-4 (8p, 16p) partial butterfly lookup tables are shown, and Figure 18 example DST-7 (4p) partial butterfly lookup tables are shown.

[0145] In some example implementations, a transform can include a line graph transform (LGT) as described in Figure 19A graph can be a general mathematical structure consisting of a set of vertices and edges that are used to model similarity relationships between objects of interest. In fact, a weighted graph, to which a set of weights is assigned to edges and potentially to vertices, can provide a sparse representation for robust modeling of signals / data. LGTs can improve coding efficiency by providing better adaptation to different block statistics. Separable LGTs can be designed and optimized by learning line graphs from data to model row- and column-wise statistics of block residual signals, where the associated generalized graph Laplacian (GGL) matrix is used to derive LGTs.

[0146] In one implementation, given a weighted graph G(W, V), the GGL matrix can be defined as LE = D - W + V, where W can be an adjacency matrix including non-negative edge weights Wc, D can be a diagonal matrix, and V can be a diagonal matrix representing weighted self-loops Vc1, Vc2. The matrix L c can be represented as:

[0147]

[0148] The LGT can then be derived by eigen-decomposition of the GGL Lc.

[0149] L c = UΦU T (2)

[0150] where the columns of the orthogonal matrix U are the basis vectors of the LGT, and Φ is a diagonal eigenvalue matrix. In fact, DCTs and DSTs (including DCT-2, DCT-8, and DST-7) are LGTs derived from certain forms of GGL. DCT-2 is derived by setting Vc1 = 0; DST-7 is derived by setting Vc = Wc; DCT-8 is derived by setting Vc2 = Wc; DCT-4 is derived by setting Vc1 = 2w c DST-4 is derived by setting Vc2 = 2w c DCT-4 is derived by setting Vc2 = wc.

[0151] The LGT can be implemented as a matrix multiplication. A 4p LGT kernel can be derived by setting vc1 = 2w c in Lc, which means it is DST-4. An 8p LGT kernel can be derived by setting vc1 = 1.5w c in Lc, and 16p, 32p, and 64p LGT kernels can be derived by setting vc1 = wc in Lc, which means it is DST-7.

[0152] Turning to some example implementations of signaling for a particular type of coding block / unit, for each intra and inter coded unit, a flag (i.e., a skip_txfm flag) can be signaled in the coded bitstream as shown in the example syntax of Table 3 below and represented by the read_skip() function for obtaining these flags from the bitstream. This flag can indicate whether the transform coefficients are all zero in the current coding unit. In some example implementations, if this flag is signaled with, for example, a value of 1, then no further transform coefficient related syntax (e.g., EOB (End of Block)) needs to be signaled for any color coding block in the coding unit and can be derived as a value or data structure that is predefined for and associated with a zero transform coefficient block. For inter coded blocks, as shown in the example of Table 3, this flag can be signaled after the skip mode flag, which indicates that the coding unit can be skipped for various reasons. When skip mode is true, the coding unit should be skipped and no skip_txfm flag needs to be signaled and the skip_txfm flag is inferred to be 1. Otherwise, if skip mode is false, more information about the coding unit will be included in the bitstream and the skip_txfm flag will be additionally signaled to indicate whether the coding unit is all zero or not.

[0153] Table 3: Skip mode and skip syntax intra mode information syntax

[0154]

[0155] Intra mode information syntax

[0156]

[0157]

[0158] Skip syntax

[0159]

[0160] Turning to the encoding and decoding (entropy coding) of the transform coefficients of the residuals in each color component, for each transform block, the transform coefficient encoding can start with the signaling of a skip symbol, followed by the transform kernel type and the end of block (EOB) position when the skip symbol is zero (indicating the presence of non-zero coefficients). Each coefficient value is then mapped to a multi-level mapping (amplitude mapping) and a sign.

[0161] After the EOB position is coded, the lower level mapping and the mid level mapping can be coded in a reverse scan order, the former indicating whether the coefficient magnitude is within the lower level (e.g., between 0 and 2), and the latter indicating whether the range is within the mid level (e.g., between 3 and 14). In the forward scan order, the next step encodes the sign of the coefficient and the residual value of the coefficient larger than the high level (e.g., 14) by, for example, an exponential Golomb code.

[0162] For the use of context modeling, the lower level mapping coding can incorporate the transform size and direction as well as up to five neighboring coefficient information. On the other hand, the mid level mapping coding can follow a similar approach as the lower level magnitude coding, except that the number of neighboring coefficients is reduced to a smaller number (e.g., 2). The example exponential Golomb code for the residual level as well as the sign of the AC coefficients are coded without any context model, while the sign of the DC coefficients is coded using the dc sign of its neighboring transform block.

[0163] In some example implementations, chroma residuals can be jointly coded. Such coding schemes can be based on some statistical correlation between the chroma channels. For example, in many cases, the Cr and Cb chroma coefficients can be similar in magnitude and opposite in sign, and thus can be jointly coded, e.g., at the transform block level (at which transform coefficients are signaled), to improve coding efficiency by introducing only a small color distortion. The use (activation) of the joint chroma coding mode can be indicated, e.g., by a joint chroma coding flag (e.g., a TU-level flag tu_joint_cbcr_residual_flag), and the selected joint mode can be implicitly indicated by the chroma CBF.

[0164] In particular, if either or both of the chroma CBFs of a TU (transform block) is equal to 1, there can be a flag tu_joint_cbcr_residual_flag. In the PPS and slice header, chroma quantization parameter (QP) offset values can be signaled for the joint chroma residual coding mode to be distinguished from the chroma QP offset values signaled for the regular chroma residual coding mode. These chroma QP offset values can be used to derive the chroma QP values for those blocks coded using the joint chroma residual coding mode. When the corresponding joint chroma coding mode (mode 2 in Table 4 below) is activated in a TU, this chroma QP offset can be added to the luma-derived chroma QP applied during quantization and decoding of that TU. For other modes (modes 1 and 3 in Table 4), the chroma QP can be derived in the same way as for regular Cb or Cr blocks. The process of reconstructing the chroma residual (resCb and resCr) from the transmitted transform block is described in Table 4. When this mode (mode 2) is activated, one single joint chroma residual block (resJointC[x][y] in Table 4) can be signaled and the residual block for Cb (resCb) and the residual block for Cr (resCr) can be derived taking into account information such as tu_cbf_cb, tu_cbf_cr and CSign, a sign value specified in the slice header, for example, and not at the transform block level. In some implementations, CSign can be -1 most of the time.

[0165] The three example joint chroma coding modes described above can be supported only in intra coded CUs. In inter coded CUs, only mode 2 is supported. Thus, for inter coded CUs, the syntax element tu_joint_cbcr_residual_flag only exists if both chroma CBFs are 1.

[0166] Table 4. Reconstruction of chroma residual. Value CSign is the sign value (+1 or -1) specified in the slice header, resJointC[][] is the transmitted residual.

[0167]

[0168] The above joint chroma coding schemes assume some correlation between the transform coefficients of co-located Cr and Cb transform blocks. These assumptions are typically statistical and thus can lead to distortion in some cases. In particular, when one of the color coefficients in a transform block is non-zero while the other color component has zero coefficients, then some of the assumptions made in the joint chroma coding schemes will certainly be off and such coding will not save any coded bits (as one of the chroma coefficients is zero anyway).

[0169] In various example implementations described below, a coefficient-level (i.e., transform coefficient by transform coefficient) cross-component coding scheme is described that exploits some correlation between the co-located transform coefficients (co-located in the frequency domain) of the color components. Such a scheme is particularly useful for transform blocks (or units) in which the coefficients of one color component are zero while the corresponding transform coefficients of another color component are non-zero. For those pairs of zero and non-zero color coefficients, either before or after dequantization, the non-zero color coefficient can be used to estimate or derive the original smaller value of the zero coded coefficient of the other color component (which can have originally been a non-zero but smaller value before quantization during the encoding process), thereby potentially recovering some information lost in, for example, the quantization process during encoding. Due to the statistically existing inter-color correlation, some information lost during quantization to zero can be recovered. Such cross-component coding recovers some of the lost information at no significant (zero coefficient) coding cost. Such a coefficient information recovery process can also be referred to as a transform coefficient correction process (or, for simplicity, a coefficient correction process) because a coefficient with a value of zero can be recovered to, for example, a small non-zero value. In one implementation, during the coefficient correction process, a zero transform coefficient in a second transform block can be corrected by adding an offset value. The offset value can be derived based on a corresponding (e.g., co-located) transform coefficient in a first transform block. The first transform block can be associated with a first color component and the second transform block can be associated with a second color component different from the first color component. The color components can be any of the luminance and chrominance components.

[0170] In example implementations, a cross-component coefficient sign coding method can be implemented that utilizes the coefficient sign values of a first color component to code the coefficient signs of a second color component. In one more specific example, the sign values of Cb transform coefficients can be used as contexts to code the signs of other Cr transform coefficients. Such cross-component coding can be implemented on a transform coefficient by transform coefficient basis on pairs of color components. The underlying principles of such implementations and other implementations described in further detail below are not limited to Cr and Cb components. They apply to any two of the three color components. In this regard, the luminance channel is considered to be one of the color components.

[0171] The method of utilizing transform coefficients in a first transform block in a first component to derive and correct co-located transform coefficients in a second transform block in a second component can be referred to as cross-component level reconstruction (CCLR). In this method, CCLR is applied to the first transform block using the second transform block as a reference. For example, the level values of Cb transform coefficients can be used to derive the level values of corresponding (e.g., co-located) Cr transform coefficients and vice versa. Using CCLR, at the decoder end, information of one color component can be referenced to correct or recover the information of another color component.

[0172] In the following example implementations, the term chroma channel can refer to both the Cb color component and the Cb color component (or channel), or both the U color component and the V color component (or channel). The term luma channel can include the luma component or Y component. The luma component or luma channel can be referred to as the luma color component or luma color channel. The following uses Y, U, and V to represent the three color components. Furthermore, the terms “coded block” and “encoded block” are used interchangeably to refer to a block to be encoded or a block that has been encoded. They can be blocks of any of the three color components. The three corresponding color encoded / coded blocks can be used for a coded / encoded unit.

[0173] In the following example implementations, the transform set refers to a set of transform kernel (or candidate) options. The transform set can include one or more of DCT, ADST, FLIPADST, IDT, LGT, KLT, or RCT type of kernel (or candidate) options.

[0174] In the following example implementations, the transform type refers to the type of primary transform and / or secondary transform. Examples of primary transform types can include, but are not limited to, DCT, ADST, FLIPADST, IDT, LGT, KLT, and RCT. Examples of secondary transform types include, but are not limited to, KLT using different input size, different kernel.

[0175] In the following example implementations, the term transform can refer to a primary transform, or a secondary transform, or a combination of a primary transform and a secondary transform. The term inverse transform can refer to an inverse primary transform, or an inverse secondary transform, or a combination of an inverse primary transform and an inverse secondary transform.

[0176] The following example implementations can be used individually or in any order in combination. The term block size can refer to the block width or height of a block, or the maximum of the width and height, or the minimum of the width and height, or the area size (width*height), or the aspect ratio (width:height, or height:width). The term “level value” or “level” can refer to the magnitude of a transform coefficient value.

[0177] In some example implementations, the level values and / or sign values of the transform coefficients of a first color component can be used to derive an offset value added to the transform coefficient level values of a second color component.

[0178] In some further implementations, the transform coefficients of a first color component used to generate the offset value and transform coefficients of a second color component are co-located (have the same coordinates in the frequency domain, e.g., the estimates are not cross-frequency).

[0179] While the first color component and the second color component described above can not be limited to specific color components, in some example implementations, the first color component can be Cb (or Cr), and the second color component is Cr (or Cb).

[0180] In some particular example implementations, the first color component can be luma, and the second color component can be one of Cb and Cr.

[0181] In some particular example implementations, the first color component can be one of Cb and Cr, and the second color component can be luma.

[0182] In some example implementations, the quantized transform coefficients of the first color component can be non-zero, and the quantized transform coefficients of the second color component can be zero. In this way, the originally relatively small non-zero information of the original transform coefficients of the second component can be lost due to quantization during the encoding process, and the example implementations described herein help recover some of the lost information using the corresponding non-zero color component that can be statistically related to the zero coefficient color component.

[0183] In some example implementations, the sign value of the transform coefficients of the first color component can be used to derive an offset value that is added to the dequantized transform coefficient level values of the second color component.

[0184] In some example implementations, the sign value of the transform coefficients of the first color component is used to derive an offset value that is added to the transform coefficient level values of the second color component before dequantization.

[0185] In some example implementations, if the sign value of the transform coefficients of the first color component is positive (or negative), a negative (or positive) offset value is added to the dequantized transform coefficient level values of the second color component for reconstructing the transform coefficient values of the second color component. In other words, the sign value of the transform coefficients of the first color component and the sign value of the offset value that is added to the transform coefficient level values of the second color component have different sign values. Such an implementation can be consistent with the statistical observation that two chroma components typically have opposite signs of transform coefficients.

[0186] In some example implementations, whether the sign value of the transform coefficients of the first color component and the sign value of the offset value that is added to the transform coefficient level values of the second color component have opposite sign values is signaled in high level syntax, including but not limited to: SPS, VPS, PPS, APS, picture header, frame header, slice header, tile header, CTU header. This is similar to the signaling scheme described above for the joint chroma coding scheme.

[0187] In some example implementations, the offset value that is added to the transform coefficient level values of the second color component can depend on both the sign and the level of the transform coefficients of the first color component.

[0188] In some example implementations, the magnitude of the offset value that is added to the transform coefficient level values of the second color component can depend on the level of the transform coefficients of the first color component.

[0189] In some example implementations, the magnitude of the offset value added to the transform coefficient level values of the second color component can be predefined for each input value of the coefficient level of the transform coefficient of the first color component.

[0190] In some example implementations, the offset value added to the transform coefficient level values of the second color component can depend on the frequency at which the transform coefficient is located. For example, for higher frequency coefficients, the offset value can be smaller.

[0191] In some example implementations, the offset value added to the transform coefficient level values of the second color component can depend on the block size of the block to which the transform coefficient belongs. For example, for larger block sizes, the offset value can generally be smaller.

[0192] In some example implementations, the offset value added to the transform coefficient level values of the second color component can depend on whether the second component is a luma (Y) component or a chroma (Cb or Cr) component. For example, if the second color component is luma, the offset value can be smaller.

[0193] In some example implementations, for a transform block of the second color component, the selection of the transform kernel for this transform block can depend on whether the CCLR method is applied to modify the transform coefficients in this transform block. The selected transform kernel can be used for the primary transform or the secondary transform. Using the primary transform as an example, at the encoding (encoder) side, the selected transform kernel can be used to perform the transform on the prediction residual to obtain the transform block; at the decoding (decoder) side, if the CCLR method is applied to the transform block, the selected transform kernel can be used to perform the inverse transform on the modified transform block to obtain the prediction residual. Note that the CCLR modification process is performed on the transform block before the inverse transform to obtain the modified transform block.

[0194] In some example implementations, whether CCLR is applied or enabled for a transform block can be signaled, for example, by a syntax value or a flag.

[0195] In some example implementations, when the EOB used to signal the relative block end position associated with a transform block of the second color component is zero, it indicates that all transform coefficients in this transform block are zero, and if CCLR is applied to this transform block, the CCLR modification process can be applied to each transform coefficient in this transform block, for example, by adding an offset value corresponding to each transform coefficient. The offset value can be derived based on the co-located transform coefficient in the transform block of the first color component. As a result of the modification process, the modified transform coefficients in the transform block are no longer all zero. After the modification process, the inverse transform can be performed on the CCLR modified transform block.

[0196] In some example implementations, the CCLR correction process can target only a portion of the transform coefficients in a transform block, rather than the entire transform block.

[0197] In some example implementations, when applying inverse transform on the CCLR corrected transform block of the second color component, the same transform kernel as used for inverse transforming the co-located transform block of the first color component can be selected. Note that the co-located transform block of the first color component is used as a basis (or reference) for deriving the offset value used in the CCLR correction process applied on the transform block of the second color component.

[0198] In some example implementations, when CCLR is applied on a transform block, the transform kernel used to perform inverse transform on the corrected transform block (from which the transform block is corrected) can be explicitly signaled. For example, an index indicating the selected transform kernel from a set of transforms (i.e., a set of candidate transform kernels) can be signaled. The set of transforms can be pre-configured, pre-defined, derived, or signaled.

[0199] In some example implementations, when CCLR is applied to a transform block of a second color component and the coded block associated with the transform block is an intra-predicted block, then the transform kernel used to perform inverse transform on the corrected transform block (from which the transform block is corrected) can be implicitly derived based on the intra-prediction mode. In one implementation, additional constraints can be imposed on the selection of the transform kernel, such that whether CCLR is applied on the transform block should also be considered. For example, for one transform block on which CCLR is applied, the selected transform kernel must be different from the selected transform kernel of another transform block on which CCLR is not applied. The other transform block can be associated with the same coded block or a different coded block that is intra-predicted.

[0200] In some example implementations, when CCLR is applied to a transform block of a color component (e.g., Cb or Cr), then the transform kernel used to perform inverse transform on the corrected transform block (from which the transform block is corrected) can be the same as the transform kernel applied to the co-located transform block of the luma component.

[0201] In some example implementations, the transform kernel used to perform inverse transform on the corrected transform block can be selected from a set of transforms, and the selection can be based on the block size of the corrected transform block. The set of transforms can be pre-configured, pre-defined, derived, or signaled.

[0202] In some example implementations, there can be further constraints on whether the CCLR method can be applied to a transform block. The constraints can be based on the transform type. In one implementation, in order to apply the CCLR method, the transform type of the primary transform or the secondary transform must be limited to certain types or combinations of types. For example, when the primary transform is a 2-D transform, then both 1-D transforms of the 2-D transform must be either DCTs, or IDTs, or another combination of transform types.

[0203] In some example implementations, the derivation of the offsets during the CCLR correction process can depend on the transform type selected for the transform block to be corrected. The transform type can be applied to the primary transform or the secondary transform.

[0204] While the cross-component zero coefficient correction is described for the case where EOB is indicated as zero (or the relative position of the end of block is zero, meaning that all the transform coefficients of the corresponding block are zero), the various implementations of the cross-component correction and the selection and signaling of the transform kernel type and / or the specific kernel are not limited thereto. For example, a specific transform block of one color component can have only a small number of non-zero coefficients. However, the transform coefficients in another color component can be used to correct the other zero transform coefficients. Additionally, the correction can be further based on the non-zero coefficients in the same transform block being corrected. The selection of the transform kernel type and / or the kernel can be made similarly to the above implementations.

[0205] Figure 20 A flowchart 2000 of an example video decoding method is shown that follows the principles based on which the above implementations are based. The method can include some or all of the following steps: step 2010, receiving a bitstream of a video block including a first transform block of a first color component and a second transform block of a second color component, wherein the first transform block and the second transform block are co-located blocks; step 2020, obtaining the first transform block of the first color component and the second transform block of the second color component from the bitstream of the video block; step 2030, determining a first flag indicating that all transform coefficients in the first transform block are zero; step 2040, determining a second flag indicating that cross-component level reconstruction (CCLR) is applied to the first transform block; and step 2050, in response to determining that CCLR is applied to the first transform block:

[0206] correcting one or more of the transform coefficients in the first transform block by adding one or more offset values, the one or more offset values being derived based on transform coefficients in the second transform block that are co-located with the one or more of the transform coefficients in the first transform block;

[0207] determining a target transform kernel for the corrected first transform block;

[0208] performing an inverse transform on the corrected first transform block based on the target transform kernel to obtain a target block; and

[0209] reconstructing the first color component of the video block based on the target block.

[0210] In embodiments and implementations of the disclosure, any of the steps and / or operations can be combined or arranged in any number or order as desired. Two or more of the steps and / or operations can be performed in parallel. Embodiments and implementations in the disclosure can be used individually or in any combination in any order. Further, each of the methods (or embodiments), encoders, and decoders can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute programs stored in a non-volatile computer-readable medium.

[0211] The techniques described above, can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 21 A computer system (2800) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0212] Computer software can be coded using any suitable machine code or computer language that can be subject to assembly, compilation, linking, or like mechanisms to create code that can be executed by one or more computers CPUs, GPUs, etc.

[0213] These instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0214] Figure 21 The components shown in the computer system (2800) are exemplary and not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. Neither should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of a computer system (2800).

[0215] Computer system (2800) can include certain human interface input devices. Such a human interface input device can be responsive to user input data being generated anywhere within the vicinity of the computer system (2800). The computer system (2800) can also include certain human interface output devices. Such a human interface output device can be responsive to

[0216] Input human interface devices can include one or more of: a keyboard (2801), a mouse (2802), a touchpad (2803), a touchscreen (2810), a data glove (not shown), a joystick (2805), a microphone (2806), a scanner (2807), a camera (2808).

[0217] Computer system (2800) can also include certain human interface output devices. Such a human interface output device can be responsive to

[0218] Computer system (2800) can also include human accessible storage devices and their associated media and user input / output interfaces. Examples of human

[0219] Those skilled in the art will further appreciate that the term "computer-readable medium" as used herein does not include transitory mainlines, carriers, or other transitory signals.

[0220] The computer system (2800) can also include an interface (2854) to one or more communication networks (2855). Networks can for example be wireless, wireline, optical. Networks can further be local, wide-area, metropolitan, vehicular and industrial, real-time, delay-tolerant, and so on. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks to include GSM, 3G, 4G, 5G, LTE and the like, TV wireline or wireless wide area digital networks to include cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial to include CANBus, and so forth. Certain networks commonly require external network interface adapters that attached to certain general purpose data ports or peripheral buses (2849) (such as USB ports of the computer system (2800)); others are commonly integrated into computer systems by attachment to system buses as described below (for example Ethernet interface into PC computer systems or cellular network interface into smartphone computer systems). Using any of these networks, the computer system (2800) can communicate with other entities. Such communication can be uni-directional, receive only (for example, broadcast TV), uni-directional send-only (for example CANbus to certain CANbus devices), or bi-directional, for example to other computer systems using local or wide-area digital networks. Certain protocols and protocol stacks can be used on each of those networks and network interfaces as described above.

[0221] The above human interface devices, human-accessible storage devices, and network interfaces can be attached to the core (2840) of the computer system (2800).

[0222] The core (2840) can include one or more Central Processing Units (CPU) (2841), Graphics Processing Units (GPU) (2842), specialized programmable processing units in the form of Field Programmable Gate Areas (FPGA) (2843), hardware accelerators for certain tasks (2844), a graphics adapter (2850), and so forth. These devices, along with Read-only memory (ROM) (2845), Random-access memory (2846), internal mass storage such as internal non-user accessible hard drives, SSDs, and the like (2847), can be connected to a system bus (2848). In some computer systems, the system bus (2848) can be implemented as a Motherboard

[0223] The CPU(s) (2841), GPU(s) (2842), FPGA(s) (2843), and accelerator(s) (2844) can execute certain instructions that are part of computer programs. The combination of these

[0224] These computer programs, in turn, can be stored in a non-transitory computer- readable storage medium, such as the ROM (2845) or the RAM (2846). The computer programs can also be received from a computer-readable storage medium such as the internal mass storage (2847) or the removable storage drive (2855). The execution of the computer programs can be managed by the CPU(s) (2841), GPU(s) (2842), FPGA(s) (2843), accelerator(s) (2844), and / or the like. The computer programs can also be up-loadable to the computer system, from a computer-readable storage medium such as the removable storage drive (2855), the internal mass storage (2847), or the ROM (2845), via the network (2852), the peripheral bus (2849), and / or the like.

[0225] By way of example, and not by way of limitation, a computer system having architecture (2800), and specifically the core (2840) thereof, can provide functionality as a result of processor(s) (including CPUs, GPUs, FPGA, accelerators, and the like) executing software embodied in one or more tangible, computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as introduced above, as well as certain storage of the core (2840) that have non-volatile properties such that they are not erased or modified by powered down states or rebooting (including, for example, non-volatile memory external to the core

[0226] Although several exemplary embodiments have been described in detail above, many alternatives, modifications, permutations, and equivalents will be apparent to those skilled in the art. Accordingly, it is intended that the application be taken only by fair scope of the appended claims and that those skilled in the art specifically adapt ideas and forms of what is described herein to specific applications. It is to be understood that not only the preferred but other embodiments can be practiced. Therefore, these and all other

[0227] APPENDIX A: ABBREVIATIONS

[0228] JEM: Joint Exploration Model

[0229] VVC: Versatile Video Coding

[0230] BMS: Base Set

[0231] MV: Motion Vector

[0232] HEVC: High Efficiency Video Coding

[0233] SEI: Supplemental Enhancement Information

[0234] VUI: video usability information

[0235] GOP: group of pictures

[0236] TU: transform unit

[0237] PU: prediction unit

[0238] CTU: coding tree unit

[0239] CTB: coding tree block

[0240] PB: prediction block

[0241] HRD: hypothetical reference decoder

[0242] SNR: signal-to-noise ratio

[0243] CPU: central processing unit

[0244] GPU: graphics processing unit

[0245] CRT: cathode ray tube

[0246] LCD: liquid crystal display

[0247] OLED: organic light emitting diode

[0248] CD: compact disc

[0249] DVD: digital video disc

[0250] ROM: read-only memory

[0251] RAM: random access memory

[0252] ASIC: application-specific integrated circuit

[0253] PLD: programmable logic device

[0254] LAN: local area network

[0255] GSM: global system for mobile communications

[0256] LTE: long term evolution

[0257] CANBus: controller area network bus

[0258] USB: universal serial bus

[0259] PCI: peripheral component interconnect

[0260] FPGA: field-programmable gate array

[0261] SSD: solid state drive

[0262] IC: Integrated Circuit

[0263] HDR: High Dynamic Range

[0264] SDR: Standard Dynamic Range

[0265] JVET: Joint Video Exploration Team

[0266] MPM: Most Probable Mode

[0267] WAIP: Wide Angle Intra Prediction

[0268] CU: Coding Unit

[0269] PU: Prediction Unit

[0270] TU: Transform Unit

[0271] CTU: Coding Tree Unit

[0272] PDPC: Position Dependent Prediction Combination

[0273] ISP: Intra Sub Partition

[0274] SPS: Sequence Parameter Set

[0275] PPS: Picture Parameter Set

[0276] APS: Adaptive Parameter Set

[0277] VPS: Video Parameter Set

[0278] DPS: Decoding Parameter Set

[0279] ALF: Adaptive Loop Filter

[0280] SAO: Sample Adaptive Offset

[0281] CC-ALF: Cross Component Adaptive Loop Filter

[0282] CDEF: Constrained Directional Enhancement Filter

[0283] CCSO: Cross Component Sample Offset

[0284] LSO: Local Sample Offset

[0285] LR: Loop Restoration Filter

[0286] AV1: AOMedia Video 1

[0287] AV2: AOMedia Video 2

[0288] DCT: Discrete Cosine Transform

[0289] DST: Discrete Sine Transform

[0290] ADST: asymmetric DST

[0291] FLIPADST: flipped ADST

[0292] IDT: incremental distance transform

[0293] LGT: line graph transform

[0294] KLT: K-L transform

[0295] RCT: row-column transform

Claims

1. A method for video processing, characterized in that, The method includes: The video block bitstream is received, the video block including a first transform block of a first color component and a second transform block of a second color component, wherein the first transform block and the second transform block are co-occurring blocks; A first transform block for the first color component and a second transform block for the second color component are obtained from the bitstream of the video block; A first flag is determined, which indicates that all transformation coefficients in the first transform block are zero; A second flag is determined, indicating that Cross Component Level Reconstruction (CCLR) is applied to the first transform block; and In response to determining that CCLR is applied to the first transform block: One or more transformation coefficients in the first transformation block are corrected by adding one or more offset values ​​to obtain a corrected first transformation block, wherein the one or more offset values ​​are derived based on transformation coefficients in the second transformation block that are in the same position as one or more transformation coefficients in the first transformation block; Determine the target transform kernel for the first transform block used in the correction; Based on the target transformation kernel, an inverse transformation is performed on the modified first transformation block to obtain the target block; and The first color component of the video block is reconstructed based at least on the target block.

2. The method according to claim 1, characterized in that, The first color component includes a chromaticity component, while the second color component includes another chromaticity component; The first color component includes a luminance component, while the second color component includes a chrominance component; or The first color component includes a chromaticity component, while the second color component includes a luminance component.

3. The method according to any one of claims 1 to 2, characterized in that, Determining the target transformation kernel includes: The same transform kernel used for the second transform block is selected as the target transform kernel for the modified first transform block.

4. The method according to any one of claims 1 to 2, characterized in that, Determining the target transformation kernel includes: Extract the indicator signaled in the bitstream, wherein the indicator specifies the target transform kernel, and the indicator is signaled in response to determining that the CCLR is applied to the first transform block; and The target transformation kernel is selected based on the indicator.

5. The method according to any one of claims 1 to 2, characterized in that, Determining the target transformation kernel includes: In response to the fact that the video block is predicted using intra-frame prediction, the target transform kernel is derived based on the intra-frame prediction pattern.

6. The method according to claim 5, characterized in that, When CCLR is not applied to the second transform block, the target transform core is different from the transform core used for the second transform block.

7. The method according to claim 1, characterized in that, Determining the target transformation kernel includes: In response to the video block being inter-frame predicted, the target transform kernel is selected based on the luminance transform block that is in the same position as the first transform block.

8. The method according to any one of claims 1 to 2, characterized in that, Determining the target transformation kernel includes: The target transform kernel is selected from a kernel list based on the block size of the first transform block, wherein the kernel list is predefined or signaled in the bitstream.

9. The method according to any one of claims 1 to 2, characterized in that, CCLR is allowed to be applied to the first transform block only if the first transform block is associated with a predefined set of primary transform types.

10. The method according to claim 9, characterized in that, The transformation associated with each of the predefined master transform types in the set of master transform types is a two-dimensional transform, which is formed by two one-dimensional transforms, wherein both one-dimensional transforms are either discrete cosine transforms (DCT) or incremental distance transforms (IDT).

11. The method according to any one of claims 1 to 2, characterized in that, Further includes: The one or more offset values ​​are derived based on the following: 1) the transform coefficients in the second transform block, which are in the same position as the one or more transform coefficients in the first transform block, and 2) the target transform kernel.

12. An apparatus for video processing, the apparatus comprising a memory for storing computer instructions and a processor in communication with the memory, characterized in that, When the processor executes the computer instructions, the processor is configured to cause the device to perform the method as described in any one of claims 1-11.

13. A non-volatile storage medium for storing computer-readable instructions, characterized in that, When executed by a processor of a device for processing video data, the computer-readable instructions cause the processor to perform the method as described in any one of claims 1-11.

Images