Method and related device for video decoding

Abstract

The present disclosure relates to a cross-component method for refining decoded transform coefficients before or after dequantization in video decoding. For example, a method for video decoding is disclosed. The method may include: extracting a first transform coefficient of a first color component from a bitstream of an encoded video; extracting a second transform coefficient of a second color component from the bitstream of the encoded video; deriving an offset value based on a magnitude or sign value of the first transform coefficient; adding the offset value to the magnitude of the second transform coefficient to generate a modified second transform coefficient of the second color component; and reconstructing the encoded video based on at least the first transform coefficient of the first color component and the modified second transform coefficient of the second color component.

Description

[0001] Incorporated by Reference

[0002] This application is based on and claims the benefit of priority of U.S. Non-Provisional Patent Application No. 17 / 575,075 filed on January 13, 2022, which claims the benefit of priority of U.S. Provisional Application No. 63 / 224,046, entitled “CROSS-COMPONENT TRANSFORM COEFFICIENT LEVEL RECONSTRUCTION”, filed on July 21, 2021. Both of the above applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to video encoding / decoding techniques, and more particularly to methods and apparatus for video decoding. Background Art

[0004] The background art provided herein is not necessarily prior art to the present disclosure.

[0005] Video encoding and video decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video may include a series of pictures, each of which has a spatial dimension of, for example, 1920×1080 luminance samples and associated full chrominance samples or sub-sampled chrominance samples. A series of pictures may have a fixed or variable picture rate (alternatively referred to as a frame rate), such as 60 pictures per second or 60 frames per second. Uncompressed video has specific bit rate requirements for streaming or data processing. For example, at 8 bits per pixel per color channel, a video with a pixel resolution of 1920×1080, a frame rate of 60 frames per second, and a chrominance subsampling of 4:2:0 requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0006] One purpose of video encoding and video decoding can be to reduce redundancy in an uncompressed input video signal by compression. In some cases, compression can help reduce the above-mentioned bandwidth and / or storage space requirements by two orders of magnitude or more. Both lossless compression and lossy compression and their combinations can be used. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal via a decoding process. Lossy compression refers to an encoding / decoding process in which the original video information is not fully retained during encoding and cannot be fully restored during decoding. When lossy compression is used, the reconstructed signal may not be the same as the original signal, but the distortion between the original signal and the reconstructed signal is small enough so that the reconstructed signal is useful for the intended application despite some information loss. In the case of video, lossy compression is widely used in many applications. The amount of distortion allowed depends on the application. For example, users of certain consumer video streaming applications can tolerate higher distortion than users of movie or television broadcast applications. The compression ratio that can be achieved by a specific encoding algorithm can be selected or adjusted to reflect various distortion tolerances: higher tolerable distortion generally allows encoding algorithms that produce higher losses and higher compression ratios.

[0007] Video encoders and video decoders may utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transforms, quantization, and entropy coding.

[0008] Video codec techniques may include techniques known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into blocks of samples. When all sample blocks are encoded in intra-mode, the picture may be referred to as an intra-picture. Intra-pictures and their derivatives (e.g., independent decoder refresh pictures) may be used to reset decoder states, and may therefore be used as the first picture in a coded video bitstream and video session or as a still image. Samples of the blocks after intra-prediction may then be transformed into the frequency domain, and the generated transform coefficients may be quantized before entropy coding. Intra-prediction represents a technique for minimizing sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after transformation, and the smaller the AC coefficient, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0009] Conventional intra-frame coding, such as known from, for example, MPEG-2 generation coding techniques, does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to encode / decode blocks based on, for example, metadata and / or surrounding sample data obtained during encoding and / or decoding that is spatially adjacent and precedes the data block being intra-coded or decoded in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, and does not use reference data from other reference pictures.

[0010] There may be a variety of different forms of intra-frame prediction. When more than one such technique may be used in a given video coding technique, the techniques used may be referred to as intra-frame prediction modes. One or more intra-frame prediction modes may be provided in a particular codec. In some cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information of a video block and the intra-frame coding parameters may be encoded separately or collectively included in a mode codeword. The codeword to be used for a given mode, sub-mode, and / or parameter combination may have an impact on the coding efficiency gain through intra-frame prediction, and may therefore have an impact on the entropy coding technique used to convert the codeword into a bitstream.

[0011] Certain modes of intra prediction were introduced by H.264, refined in H.265, and further refined in newer coding techniques such as the Joint Exploitation Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Typically, for intra prediction, a block of prediction values ​​can be formed using neighboring sample values ​​that have become available. For example, available values ​​for a particular set of neighboring samples along a particular direction and / or line can be copied into a block of prediction values. A reference to the direction used can be encoded in the bitstream, or it can itself be predicted.

[0012] Reference Figure 1A , depicted at the bottom right is a subset of 9 prediction value directions specified in the 33 possible intra-frame prediction value directions of H.265 (corresponding to 33 angular modes of the 35 intra-frame modes specified in H.265). The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which neighboring samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more neighboring samples at a 45 degree angle to the horizontal direction at the upper right. Similarly, arrow (103) indicates that sample (101) is predicted from one or more neighboring samples at a 22.5 degree angle to the horizontal direction at the lower left of sample (101).

[0013] Still refer to Figure 1A, depicted at the top left is a square block (104) of 4×4 samples (indicated by the dashed bold line). The square block (104) includes 16 samples, each of which is labeled by "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in block (104) in both the Y dimension and the X dimension. Since the size of the block is 4×4 samples, S44 is at the bottom right. Example reference samples following a similar numbering scheme are further shown. The reference samples are labeled by R, their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, prediction samples that are adjacently adjacent to the block being reconstructed are used.

[0014] The intra picture prediction of block 104 can start by copying reference sample values ​​from neighboring samples according to the prediction direction signaled by the signal. For example, assume that the encoded video bitstream includes the following signaling, which indicates the prediction direction of the arrow (102) for this block 104, that is, the samples are predicted according to one or more prediction samples at a 45 degree angle to the horizontal direction at the upper right. In this case, samples S41, S32, S23 and S14 are predicted according to the same reference sample R05. Then, sample S44 is predicted according to the reference sample R08.

[0015] In some cases, the values ​​of multiple reference samples may be combined, such as by interpolation, in order to calculate the reference sample; in particular when the direction is not divisible by 45 degrees.

[0016] As video coding techniques continue to advance, the number of possible directions is also increasing. For example, in H.264 (2003), nine different directions were available for intra prediction. This was increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions when disclosed. Experimental studies have been conducted to help identify the most appropriate intra prediction directions, and certain techniques in entropy coding can be used to encode those most appropriate directions with a small number of bits, accepting a specific bit penalty for the direction. In addition, the direction itself can sometimes be predicted based on the neighboring directions used in intra prediction of decoded neighboring blocks.

[0017] Figure 1B A schematic diagram (180) depicting 65 intra prediction directions according to the JEM is shown to illustrate the increasing number of prediction directions in various encoding techniques developed over time.

[0018] The manner in which the bits representing the intra-prediction direction are mapped to the prediction direction in the coded video bitstream may vary from one video coding technique to another; and the mapping may vary, for example, from a simple direct mapping of prediction direction to intra-prediction mode to codeword to complex adaptive schemes involving most probable modes and similar techniques. However, in all cases, there may be certain intra-prediction directions that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, in a well-functioning video coding technique, those less probable directions may be represented by a larger number of bits than more probable directions.

[0019] Inter-picture prediction or inter-prediction may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or a portion thereof (reference picture) may be used to predict a newly reconstructed picture or picture portion (e.g., block) after spatial shifting in the direction indicated by a motion vector (hereinafter referred to as MV). In some cases, the reference picture may be the same as the currently reconstructed picture. MV may have two dimensions, X and Y, or three dimensions, where the third dimension is an indication of the reference picture in use (similar to the time dimension).

[0020] In some video compression techniques, the current MV applicable to a specific area of ​​sample data can be predicted from other MVs, such as the following other MVs, which are related to other areas of sample data that are spatially adjacent to the area being reconstructed and precede the current MV in decoding order. Doing so can significantly reduce the total amount of data required to encode the MV by relying on removing redundancy in the associated MV, thereby improving compression efficiency. MV prediction can work effectively, for example, because when encoding an input video signal derived from a camera (called natural video), there is a statistical possibility that areas larger than the area applicable to a single MV in a video sequence move in similar directions, and therefore in some cases similar motion vectors derived from MVs of neighboring areas can be used for prediction. This results in the actual MV for a given area being similar or identical to the MV predicted from the surrounding MVs. Such an MV can be represented after entropy coding with less bits than when the MV is encoded directly from neighboring MVs rather than predicted from neighboring MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from an original signal (i.e., a sample stream). In other cases, the MV prediction itself may be lossy, for example due to round-off errors when computing the prediction value from several surrounding MVs.

[0021] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding", December 2016). Among the multiple MV prediction mechanisms specified by H.265, described below is a technique hereinafter referred to as "spatial merging".

[0022] Specifically, refer to Figure 2 , the current block (201) includes samples that have been discovered by the encoder during the motion search process, which are predictable from a previous block of the same size that has been spatially offset. Instead of encoding the MV directly, the MV can be derived from metadata associated with one or more reference pictures, for example, using the MV associated with any of the five surrounding samples denoted A0, A1 and B0, B1, B2 (corresponding to 202 to 206), the MV is derived from metadata of the nearest reference picture (in decoding order). In H.265, MV prediction can use the prediction value of the same reference picture that the neighboring blocks are also using.

[0023] Some joint chroma coding schemes assume some correlation between transform coefficients of co-located Cr and Cb transform blocks. These assumptions are usually statistical and thus introduce distortions in some cases. In particular, when one of the color coefficients in the transform block is non-zero while the coefficient of the other color component is zero, then some of the assumptions made in the joint chroma coding scheme are incorrect and such coding will not save any coding bits (since one of the chroma coefficients is zero anyway). Summary of the invention

[0024] Aspects of the present disclosure provide a method for video decoding. The method may include: extracting a first transform coefficient of a first color component from a bitstream of an encoded video; extracting a second transform coefficient of a second color component from the bitstream of the encoded video; obtaining an offset value based on a magnitude or a sign value of the first transform coefficient; adding the offset value to the magnitude of the second transform coefficient to generate a modified second transform coefficient of the second color component; and reconstructing the encoded video based on at least the first transform coefficient of the first color component and the modified second transform coefficient of the second color component.

[0025] Aspects of the present disclosure also provide a video decoding device, including: a first extraction module, configured to extract a first transform coefficient of a first color component from a bit stream of an encoded video; a second extraction module, configured to extract a second transform coefficient of a second color component from the bit stream of the encoded video; a calculation module, configured to obtain an offset value based on the amplitude or sign value of the first transform coefficient; an addition module, configured to add the offset value to the amplitude of the second transform coefficient to generate a modified second transform coefficient of the second color component; and a reconstruction module, configured to reconstruct the encoded video based on at least the first transform coefficient of the first color component and the modified second transform coefficient of the second color component.

[0026] Aspects of the present disclosure also provide an apparatus for video decoding, including: at least one memory configured to store program codes; and at least one processor configured to read the program codes to execute the above method.

[0027] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform the above-described method.

[0028] According to the above method of the present disclosure, an offset value is obtained based on the amplitude or sign value of the first transform coefficient of the first color component, and the offset value is added to the amplitude of the second transform coefficient of the second color component to generate a modified second transform coefficient, thereby utilizing some correlation between the transform coefficients of the color components. Such a scheme is particularly useful for transform blocks (or units) in which the coefficients of one color component are zero and the corresponding transform coefficients of another color component are non-zero. For those zero and non-zero color coefficient pairs, before or after dequantization, the non-zero color coefficient can be used to estimate the original small value of the zero-coded coefficient of the other color component (which may initially be a non-zero but smaller value before quantization in the encoding process), thereby possibly recovering some information lost during the quantization process during encoding, for example. Due to the statistically existing correlation between colors, the information lost during quantization to zero can be recovered. Such cross-component coding recovers the lost information to a certain extent without significant coding cost (of zero coefficients). BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Additional features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0030] Figure 1A A schematic diagram of an exemplary subset of intra prediction directional modes is shown.

[0031] Figure 1B A diagram of exemplary intra prediction directions is shown.

[0032] Figure 2 A schematic diagram of a current block and its surrounding spatial merging candidates for motion vector prediction in one example is shown.

[0033] Figure 3 A schematic illustration of a simplified block diagram of a communication system (300) is shown according to some example embodiments.

[0034] Figure 4 A schematic illustration of a simplified block diagram of a communication system (400) is shown according to some example embodiments.

[0035] Figure 5 A schematic illustration of a simplified block diagram of a video decoder according to some example embodiments is shown.

[0036] Figure 6 A schematic illustration of a simplified block diagram of a video encoder is shown according to some example embodiments.

[0037] Figure 7 A block diagram of a video encoder according to further example embodiments is shown.

[0038] Figure 8 A block diagram of a video decoder according to further example embodiments is shown.

[0039] Fig. 9 A scheme of coding block partitioning according to some example embodiments of the present disclosure is shown.

[0040] Fig.10 Another scheme of coding block partitioning according to some example embodiments of the present disclosure is shown.

[0041] Fig.11 Another scheme of coding block partitioning according to some example embodiments of the present disclosure is shown.

[0042] Fig.12 Another scheme of coding block partitioning according to some example embodiments of the present disclosure is shown.

[0043] Fig.13 A scheme of dividing a coding block into a plurality of transform blocks and an encoding order of the transform blocks according to some example embodiments of the present disclosure are shown.

[0044] Fig.14 Another scheme of dividing a coding block into a plurality of transform blocks and an encoding order of the transform blocks according to some example embodiments of the present disclosure are shown.

[0045] Fig.15 Another scheme of dividing a coding block into a plurality of transform blocks according to some example embodiments of the present disclosure is shown.

[0046] Fig.16 A flow chart illustrating a method according to some example embodiments of the present disclosure is shown.

[0047] Fig.17 Schematic diagrams of computer systems according to some example embodiments of the present disclosure are shown. DETAILED DESCRIPTION

[0048] Figure 3 A simplified block diagram of a communication system (300) according to some embodiments of the present disclosure is shown. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and a terminal device (320) interconnected via the network (350). Figure 3 In the example of , a first pair of terminal devices (310) and a terminal device (320) can perform unidirectional data transmission. For example, the terminal device (310) can encode video data (e.g., a video picture stream captured by the terminal device (310)) for transmission to another terminal device (320) via a network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to restore the video picture, and display the video picture based on the restored video data. The unidirectional data transmission can be implemented in a media service application, etc.

[0049] In another example, the communication system (300) includes a second pair of terminal devices (330) and terminal devices (340) that perform bidirectional transmission of encoded video data, which can be implemented, for example, during a video conferencing application. For the bidirectional transmission of data, in the example, each of the terminal devices (330) and the terminal devices (340) can encode video data (e.g., a video picture stream captured by the terminal device) for transmission to the other of the terminal devices (330) and the terminal devices (340) via a network (350). Each of the terminal devices (330) and the terminal devices (340) can also receive the encoded video data transmitted by the other of the terminal devices (330) and the terminal devices (340), and can decode the encoded video data to restore the video picture, and can display the video picture at an accessible display device based on the restored video data.

[0050] exist Figure 3In the example of , terminal devices (310), (320), (330) and (340) can be implemented as servers, personal computers and smart phones, but the applicability of the basic principles of the present disclosure may not be limited to this. Implementations of the present disclosure can be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, etc. Network (350) represents any number or any type of network that transmits encoded video data between terminal devices (310), (320), (330) and (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) can exchange data in circuit switching, packet switching and / or other types of channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purposes of this discussion, unless explicitly stated herein, the architecture and topology of the network (350) may be irrelevant to the operation of the present disclosure.

[0051] As examples of applications of the disclosed subject matter, Figure 4 The placement of the video encoder and video decoder in a video streaming environment is shown. The disclosed subject matter may be equally applicable to other video applications, including, for example: video conferencing, digital television broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc., etc.

[0052] The video streaming system may include a video capture subsystem (413) that may include a video source (401) such as a digital camera for creating an uncompressed video picture or image stream (402). In an example, the video picture stream (402) includes samples recorded by the digital camera of the video source 401. The video picture stream (402) is depicted as a thick line to emphasize the high amount of data when compared to the encoded video data (404) (or encoded video bitstream), and the video picture stream (402) may be processed by an electronic device (420) coupled to the video source (401) including a video encoder (403). The video encoder (403) may include hardware, software, or a combination thereof to implement or embody aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is depicted as a thin line to emphasize the lower data volume when compared to the uncompressed video picture stream (402), and the encoded video data (404) can be stored on the streaming server (405) for future use or directly stored to a downstream video device (not shown). One or more streaming client subsystems, such as Figure 4The client subsystems (406) and (408) in the video transmission system can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in the electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and creates an outgoing video picture stream (411) that is uncompressed and can be presented on a display (412) (e.g., a display screen) or other rendering device (not depicted). The video decoder 410 can be configured to perform some or all of the various functions described in the present disclosure. In some streaming systems, the encoded video data (404), (407) and (409) (e.g., video bitstreams) can be encoded according to certain video coding standards / video compression standards. Examples of these standards include ITU-T Recommendation H.265. In the example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed subject matter can be used in the context of VVC and other video coding standards.

[0053] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0054] Figure 5 A block diagram of a video decoder (510) according to any of the following embodiments of the present disclosure is shown. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used instead of Figure 4 A video decoder (410) in an example of FIG.

[0055] The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510). In the same embodiment or another embodiment, one encoded video sequence can be decoded at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Each video sequence can be associated with multiple video frames or images. The encoded video sequence can be received from a channel (501), which can be a hardware / software link to a storage device storing the encoded video data or a streaming source sending the encoded video data. The receiver (531) can receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams that can be forwarded to their respective processing circuit systems (not depicted). The receiver (531) can separate the encoded video sequence from the other data. In order to combat network jitter, a buffer memory (515) can be set between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) can be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be external to the video decoder (510) and separate from the video decoder (510) (not depicted). In still other applications, there may be a buffer memory (not depicted) external to the video decoder (510) for purposes such as combating network jitter, and there may also be another other buffer memory (515) internal to the video decoder (510) to, for example, handle playback timing. When the receiver (531) is receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may not be required, or the buffer memory (515) may be small. For use on a best effort packet network such as the Internet, a buffer memory (515) of sufficient size may be required, and the size of the buffer memory (515) may be relatively large. Such a buffer memory may be implemented to have an adaptive size and may be implemented at least in part in an operating system or similar element (not depicted) external to the video decoder (510).

[0056] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The categories of symbols include information for managing the operation of the video decoder (510) and may include information for controlling a presentation device such as a display (512) (e.g., a display screen), which may or may not be part of the electronic device (530) but may be coupled to the electronic device (530), such as Figure 5As shown. The control information for the rendering device may take the form of a supplementary enhancement information (SEI (Supplemental Enhancement Information) message) or a video usability information (VUI) parameter set fragment (not depicted). The parser (520) may parse / entropy decode the coded video sequence received by the parser (520). The entropy coding of the coded video sequence may comply with a video coding technique or a video coding standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a subgroup parameter set for at least one subgroup of the pixel subgroups in the video decoder from the coded video sequence based on at least one parameter corresponding to the subgroup. The subgroup may include a group of pictures (Group of Pictures, GOP), a picture, a tile, a slice, a macroblock, a coding unit (Coding Unit, CU), a block, a transform unit (Transform Unit, TU), a prediction unit (Prediction Unit, PU), etc. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients (eg, Fourier transform coefficients), quantizer parameter values, motion vectors, and the like.

[0057] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515), thereby creating symbols (521).

[0058] Depending on the type of coded video pictures or parts thereof (such as: inter- and intra-pictures, inter- and intra-blocks) and other factors, the reconstruction of the symbol (521) may involve a number of different processing units or functional units. Which units are involved and how they are involved may be controlled by sub-group control information parsed from the coded video sequence by the parser (520). For the sake of brevity, the flow of such sub-group control information between the parser (520) and the following multiple processing units or functional units is not depicted.

[0059] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these functional units interact closely with each other and may be at least partially integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed subject matter, the conceptual subdivision into functional units is adopted in the following disclosure.

[0060] The first unit may include a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) may receive quantized transform coefficients as symbols (521) and control information from the parser (520), including information indicating which type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrix, etc. The sealer / inverse transform unit (551) may output blocks including sample values, which may be input into the aggregator (555).

[0061] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to intra-coded blocks, i.e., blocks that do not use predictive information from a previously reconstructed picture, but instead use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may use surrounding block information that has been reconstructed and stored in a current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator (555) may add the prediction information that has been generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0062] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-coded and possibly motion compensated block. In such a case, the motion compensated prediction unit (553) may access the reference picture memory (557) to obtain samples for inter-picture prediction. After motion compensation of the obtained samples according to the symbol (521) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (551) by the aggregator (555) (the output of unit 551 may be referred to as residual samples or residual signal) to generate output sample information. The address within the reference picture memory (557) from which the motion compensated prediction unit (553) obtains the predicted samples may be controlled by a motion vector, which may be obtained by the motion compensated prediction unit (553) in the form of a symbol (521) that may have, for example, an X component, a Y component (offset) and a reference picture component (time). Motion compensation may also include interpolation of sample values ​​retrieved from a reference picture memory (557) when using sub-sample accurate motion vectors, and may also be associated with motion vector prediction mechanisms, etc.

[0063] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and obtained by the loop filter unit (556) as symbols (521) from the parser (520), but may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of the coded picture or coded video sequence, as well as to previously reconstructed and loop filtered sample values. As will be described in further detail below, several types of loop filters may be included as part of the loop filter unit 556 in various orders.

[0064] The output of the loop filter unit (556) may be a sample stream that may be output to a rendering device (512) and stored in a reference picture memory (557) for future inter-picture prediction.

[0065] Certain coded pictures, once fully reconstructed, can be used as reference pictures for future inter-picture prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by a parser (520)), the current picture buffer (558) can become part of the reference picture memory (557) and a new current picture buffer can be reallocated before starting to reconstruct a subsequent coded picture.

[0066] The video decoder (510) may perform decoding operations according to a predetermined video compression technique adopted in a standard such as ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence follows both the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence may be within limits such as defined by the level of the video compression technology or standard. In some cases, the level may limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (measured in, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the encoded video sequence.

[0067] In some example embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The video decoder (510) may use the additional data to correctly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0068] Figure 6 A block diagram of a video encoder (603) according to an example implementation of the present disclosure is shown. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may also include a transmitter (640) (e.g., a transmission circuit system). The video encoder (603) may be used instead of Figure 4 A video encoder (403) in an example.

[0069] The video encoder (603) can capture video images to be encoded by the video encoder (603) from a video source (601) (in Figure 6 In an example, the video source (601) is not part of the electronic device (620) to receive the video samples. In another example, the video source (601) can be implemented as part of the electronic device (620).

[0070] The video source (601) can provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (603), and the digital video sample stream can have any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits, ...), any color space (e.g., BT.601YCrCB, RGB, XYZ, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) can be a storage device capable of storing previously prepared videos. In a video conferencing system, the video source (601) can be a camera that captures local image information as a video sequence. The video data can be provided as multiple individual pictures or images that are given motion when viewed sequentially. The picture itself can be organized as a spatial pixel array, where each pixel can include one or more samples, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples can be easily understood by a person skilled in the art. The following description focuses on samples.

[0071] According to some example embodiments, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed constitutes a function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units as described below. For the sake of brevity, the coupling is not depicted. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technique, etc.), picture size, picture group (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other suitable functions, which are related to the video encoder (603) optimized for a specific system design.

[0072] In some example embodiments, the video encoder (603) may be configured to operate in an encoding loop. As an extremely simplified description, in an example, the encoding loop may include: a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture); and a (local) decoder (633), which is embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder would create sample data, even if the embedded decoder 633 is processing the encoded video stream of the source encoder 630 without entropy coding (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video bitstream in entropy coding may be lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream results in a bit-exact result that is independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" as reference picture samples exactly the same sample values ​​as the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the offset in case synchronization cannot be maintained, e.g. due to channel errors) is used to improve encoding quality.

[0073] The operation of the "local" decoder (633) may be the same as the operation of a "remote" decoder, such as the video decoder (510), described above in conjunction with Figure 5 The video decoder (510) is described in detail. However, reference is also briefly made to Figure 5, since the symbols are available and encoding of the symbols into a coded video sequence by the entropy encoder (645) and decoding of the symbols by the parser (520) can be lossless, the entropy decoding portion of the video decoder (510) including the buffer memory (515) and the parser (520) may not be fully implemented in the local decoder (633) in the encoder.

[0074] At this point, it can be observed that any decoder technology, except for the parsing / entropy decoding that can only be present in the decoder, must also be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter may sometimes focus on the decoder operation, which is partially associated with the decoding of the encoder. Since the encoder technology is contrary to the decoder technology that has been fully described, the description of the encoder technology can be simplified. A more detailed description of the encoder is provided below only in certain areas or aspects.

[0075] In some example implementations, during operation, the source encoder (630) may perform motion compensated predictive coding that predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as "reference pictures". In this manner, the encoding engine (632) encodes the differences (or residuals) of color channels between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture. The term "residual" and its adjective form "residual" may be used interchangeably.

[0076] The local video decoder (633) can decode the encoded video data of the picture that can be designated as the reference picture based on the symbol created by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the video decoder ( Figure 6 When the encoded video data is decoded at a local video decoder (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors therein. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture, and may cause the reconstructed reference picture to be stored in the reference picture memory (634). In this way, the video encoder (603) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture that will be obtained by the far-end (remote) video decoder.

[0077] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may be used as suitable prediction references for the new picture. The predictor (635) may operate on a sample-block-by-pixel-block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references obtained from multiple reference pictures stored in the reference picture memory (634).

[0078] The controller (650) may manage encoding operations of the source encoder (630), including, for example, setting of parameters and sub-group parameters for encoding video data.

[0079] The outputs of all the above functional units may be entropy encoded in an entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing them according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0080] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which can be a hardware / software link to a storage device that can store the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown).

[0081] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a specific encoding picture type to each encoded picture, which may affect the encoding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned one of the following picture types:

[0082] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those of ordinary skill in the art are aware of those variations of I pictures and their respective applications and features.

[0083] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, where inter prediction uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0084] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, the inter prediction uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0085] The source picture can typically be spatially subdivided into a plurality of sample coding blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples, respectively) and encoded block by block. The blocks can be predictively encoded with reference to other (already encoded) blocks determined by the coding allocation applied to the corresponding picture of the block. For example, a block of an I picture can be non-predictively encoded, or a block of an I picture can be predictively encoded (spatial prediction or intra-frame prediction) with reference to an already encoded block of the same picture. A pixel block of a P picture can be predictively encoded via spatial prediction or via temporal prediction with reference to a previously encoded reference picture. A block of a B picture can be predictively encoded via spatial prediction or via temporal prediction with reference to one or two previously encoded reference pictures. For other purposes, the source picture or the intermediate processed picture can be subdivided into other types of blocks. The division of coding blocks and other types of blocks may or may not follow the same approach, as described in further detail below.

[0086] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In its operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard being used.

[0087] In some example embodiments, the transmitter (640) may send additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0088] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often simplified to intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits temporal or other correlations between pictures. For example, a particular picture being encoded / decoded, referred to as the current picture, can be divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that was previously encoded in the video and is still buffered, the block in the current picture can be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case of using multiple reference pictures, the motion vector may have a third dimension that identifies the reference picture.

[0089] In some example embodiments, bidirectional prediction techniques may be used for inter-picture prediction. According to such bidirectional prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture that are both before the current picture in the video in decoding order (but may be in the past and in the future, respectively, in display order). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block may be jointly predicted by a combination of the first reference block and the second reference block.

[0090] In addition, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0091] According to some example embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, a picture in a video picture sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the picture may have the same size, for example, 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU may include three parallel coding tree blocks (CTBs): a luminance CTB and two chrominance CTBs. Each CTU may be recursively split into one or more coding units (CUs) in a quadtree. For example, a 64×64 pixel CTU may be split into a 64×64 pixel CU, or four 32×32 pixel CUs. Each of one or more of the 32×32 blocks may also be split into four 16×16 pixel CUs. In some example embodiments, each CU may be analyzed during encoding to determine the prediction type of the CU among various prediction types, such as an inter prediction type or an intra prediction type. Depending on temporal and / or spatial predictability, a CU may be split into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma prediction blocks. In an embodiment, prediction operations in encoding (encoding / decoding) are performed in units of prediction blocks. Splitting a CU into PUs (or PBs for different color channels) may be performed in various spatial modes. A luma or chroma PB may, for example, contain a matrix of values ​​(e.g., luma values) of samples of, for example, 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0092] Figure 7 A diagram of a video encoder (703) according to another example implementation of the present disclosure is shown. The video encoder (703) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures and encode the processed block into an encoded picture that is part of an encoded video sequence. The example video encoder (703) may be used instead of Figure 4 The video encoder (403) in the example.

[0093] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) then uses, for example, rate-distortion optimization (RDO) to determine whether to use intra mode, inter mode, or bidirectional prediction mode to best encode the processing block. When it is determined that the processing block is to be encoded in intra mode, the video encoder (703) may encode the processing block into a coded picture using intra prediction techniques; and when it is determined that the processing block is to be encoded in inter mode or bidirectional prediction mode, the video encoder (703) may encode the processing block into a coded picture using inter prediction techniques or bidirectional prediction techniques, respectively. In some example embodiments, a merge mode may be used as a sub-mode of inter-picture prediction, in which a motion vector is derived from one or more motion vector predictors without the aid of an encoded motion vector component external to the prediction value. In some other example embodiments, there may be a motion vector component applicable to the object block. Thus, the video encoder (703) may include Figure 7 Components not explicitly shown, such as a mode decision module for determining a prediction mode for a processing block.

[0094] exist Figure 7 In the example of FIG. 7 , the video encoder ( 703 ) includes Figure 7 An inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together are shown in the example arrangement of.

[0095] The inter-frame encoder (730) is configured to: receive samples of a current block (e.g., a processing block); compare the block to one or more reference blocks in a reference picture (e.g., a block in a previous picture and a block in a subsequent picture in display order); generate inter-frame prediction information (e.g., motion vectors, merge mode information, description of redundant information according to an inter-frame coding technique); and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some examples, the reference picture is based on the encoded video information using an embedded Figure 6 The decoding unit 633 (shown as Figure 7 The decoded reference picture is decoded by the residual decoder 728, as described in further detail below.

[0096] The intra encoder (722) is configured to: receive samples of a current block (e.g., a processing block); compare the block with an already encoded block in the same picture; and generate quantization coefficients after transformation; and in some cases also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). The intra encoder (722) can calculate an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same picture.

[0097] The general controller (721) can be configured to determine general control data and control other components of the video encoder (703) based on the general control data. In an example, the general controller (721) determines a prediction mode of a block and provides a control signal to a switch (726) based on the prediction mode. For example, when the prediction mode is an intra-frame mode, the general controller (721) controls the switch (726) to select an intra-frame mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select intra-frame prediction information and include the intra-frame prediction information in the bitstream; and when the prediction mode of the block is an inter-frame mode, the general controller (721) controls the switch (726) to select an inter-frame prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select inter-frame prediction information and include the inter-frame prediction information in the bitstream.

[0098] The residual calculator (723) may be configured to calculate the difference (residual data) between the received block and the prediction result of the block selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) may be configured to encode the residual data to generate a transform coefficient. For example, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain to generate a transform coefficient. Then, the transform coefficient is quantized to obtain a quantized transform coefficient. In various example embodiments, the video encoder (703) further includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) may generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) may generate a decoded block based on the decoded residual data and the intra prediction information. The decoded blocks are appropriately processed to generate decoded pictures, and these decoded pictures may be buffered in a memory circuit (not shown) and used as reference pictures.

[0099] The entropy encoder (725) may be configured to format the bitstream to include the coded blocks and perform entropy coding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. When the block is encoded in the merge sub-mode of the inter-frame mode or the bidirectional prediction mode, the residual information may not be present.

[0100] Figure 8 A diagram of an example video decoder (810) according to another embodiment of the present disclosure is shown. The video decoder (810) is configured to receive an encoded picture as part of an encoded video sequence and decode the encoded picture to generate a reconstructed picture. In an example, the video decoder (810) may be used instead of Figure 4 A video decoder (410) in an example of FIG.

[0101] exist Figure 8 In the example of FIG. 8 , the video decoder ( 810 ) includes Figure 8 An entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together are shown in the example arrangement of.

[0102] The entropy decoder (871) may be configured to reconstruct certain symbols from the coded picture, which represent syntax elements constituting the coded picture. Such symbols may include, for example, a mode for encoding a block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, merge sub-mode, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that may identify certain samples or metadata used by an intra-frame decoder (872) or an inter-frame decoder (880) for prediction, residual information in the form of, for example, quantized transform coefficients, etc. In an example, when the prediction mode is an inter-frame mode or a bidirectional prediction mode, the inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is an intra-frame prediction type, the intra-frame prediction information is provided to the intra-frame decoder (872). The residual information may be inverse quantized and provided to the residual decoder (873).

[0103] The inter-frame decoder (880) may be configured to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0104] The intra decoder (872) may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0105] The residual decoder (873) can be configured to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) can also utilize certain control information (to include quantizer parameters (Quantizer Parameter, QP)), which can be provided by the entropy decoder (871) (since this may only be low data volume control information, the data path is not depicted).

[0106] The reconstruction module (874) can be configured to combine the residual output by the residual decoder (873) with the prediction result (output by the inter-frame prediction module or the intra-frame prediction module, as the case may be) in the spatial domain to form a reconstructed block, which forms part of a reconstructed picture, which is part of the reconstructed video. It should be noted that other suitable operations such as deblocking operations can also be performed to improve visual quality.

[0107] It should be noted that the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using any suitable technology. In some example embodiments, the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using one or more processors that execute software instructions.

[0108] Turning to coding block partitioning, in some example implementations, a predetermined pattern may be applied. Fig. 9 As shown, an example 4-way partitioning tree starting from a first predefined level (e.g., 64×64 block level) to a second predefined level (e.g., 4×4 level) may be employed. For example, a basic block may undergo four partitioning options indicated by 902, 904, 906, and 908, where the partition designated as R is allowed for recursive partitioning because Fig. 9 The same partitioning tree indicated in may be repeated at lower levels until the lowest level (e.g., 4×4 level). Fig. 9 Additional restrictions apply to the partitioning scheme. Fig. 9 In the implementation of , rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed, but these rectangular partitions are not allowed to be recursive, while square partitions are allowed to be recursive. If necessary, follow Fig. 9 The partitioning is performed in a recursive manner to generate the final set of coding blocks.Such a scheme can be applied to one or more of the color channels.

[0109] Fig.10 Another example predefined partitioning pattern that allows recursive partitioning to form a partitioning tree is shown. Fig.10 As shown, an example 10-way partitioning structure or pattern may be predefined. A root block may start at a predefined level (eg, from a 128×128 level or a 64×64 level). Fig.10 Example partitioning structures include various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitionings. Fig.10 The partition type with 3 sub-partitions indicated as 1002, 1004, 1006, and 1008 in the second row of can be referred to as a "T-type" partition. The "T-type" partitions 1002, 1004, 1006, and 1008 can be referred to as a left T-type, a top T-type, a right T-type, and a bottom T-type. In some implementations, Fig.10 The rectangular partitions are not allowed to be further subdivided. The coding tree depth can also be defined to indicate the split depth starting from the root node or root block. For example, the coding tree depth of the root node or root block (e.g., for a 128×128 block) can be set to 0, and the coding tree depth of the root block can be set to 0. Fig.10 After being further split once, the coding tree depth increases by 1. In some implementations, only full square partitions in 1010 may be allowed according to Fig.10 In other words, for the square partitions with patterns 1002, 1004, 1006, and 1008, recursive partitioning may not be allowed. If necessary, follow Fig.10 The partitioning is performed in a recursive manner to generate the final set of coding blocks.Such a scheme can be applied to one or more of the color channels.

[0110] After the basic block is segmented or partitioned according to any of the above-mentioned partitioning processes or other processes, a final set of partitions or coding blocks can be obtained again. Each of these partitions can be at one of the various partitioning levels. Each of the partitions can be referred to as a coding block (CB). For the various example partitioning implementations above, each resulting CB can have any allowed size and partitioning level. These partitions are called coding blocks because they can form the following units, for which some basic encoding / decoding decisions can be made and encoding / decoding parameters can be optimized, determined and signaled in the encoded video bitstream. The highest level in the final partition represents the depth of the coding block partition tree. The coding block can be a luminance coding block or a chrominance coding block.

[0111] In some other example implementations, a quadtree structure may be used to recursively split the base luma and chroma blocks into coding units. Such a split structure may be referred to as a coding tree unit (CTU), which is split into coding units (CUs) using a quadtree structure to adapt the partitioning to the various local characteristics of the base CTU. In such an implementation, an implicit quadtree split may be performed at a picture boundary so that the block will remain quadtree split until the size fits the picture boundary. The term CU is used to collectively refer to units of luma coding blocks and chroma coding blocks (CBs).

[0112] In some implementations, CB can be further divided. For example, for the purpose of intra-frame or inter-frame prediction during the encoding process and the decoding process, CB can be further divided into multiple prediction blocks (PB). In other words, CB can be further divided into different sub-divisions, in which separate prediction decisions / configurations can be made. In parallel, for the purpose of defining the level of transformation or inverse transformation of video data, CB can be further divided into multiple transform blocks (TB). The schemes for dividing CB into PB and TB can be the same or different. For example, each division scheme can be performed using its own process based on various features of video data, for example. In some example implementations, PB division schemes and TB division schemes can be independent. In some other example implementations, PB division schemes and TB division schemes can be related to boundaries. In some implementations, for example, TB can be divided after PB division, and in particular, each PB can be further divided into one or more TBs after being determined according to the division of the coding block. For example, in some implementations, PB can be split into one, two, four or other number of TBs.

[0113] In some implementations, in order to divide the basic block into coding blocks and further into prediction blocks and / or transform blocks, the luma channel and the chroma channel may be processed differently. For example, in some implementations, the division of the coding block into prediction blocks and / or transform blocks may be allowed for the luma channel, while such division of the coding block into prediction blocks and / or transform blocks may not be allowed for the chroma channel. In such an implementation, the transformation and / or prediction of the luma block can therefore be performed only at the coding block level. For another example, the minimum transform block size of the luma channel and the chroma channel may be different, for example, the coding block of the luma channel may be allowed to be divided into smaller transform blocks and / or prediction blocks than the chroma channel. For yet another example, the maximum depth of dividing the coding block into transform blocks and / or prediction blocks may be different between the luma channel and the chroma channel, for example, the coding block of the luma channel may be allowed to be divided into deeper transform blocks and / or prediction blocks than the chroma channel. For a specific example, a luma coding block may be partitioned into transform blocks of various sizes, which may be represented by recursive partitioning down to up to 2 levels, and may allow transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, and transform block sizes from 4×4 to 64×64. However, for chroma blocks, only the largest possible transform block specified for luma blocks may be allowed.

[0114] In some example implementations of partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partition may depend on whether the PB is intra-coded or inter-coded.

[0115] The division of the coding block (or prediction block) into transform blocks may be implemented in various example schemes, including but not limited to recursively or non-recursively performing quadtree splitting and predefined pattern splitting, and additionally taking into account transform blocks at the boundaries of the coding block or prediction block. In general, the resulting transform blocks may be at different split levels, may not have the same size, and may not need to be square in shape (e.g., the resulting transform blocks may be rectangular with certain allowed sizes and aspect ratios).

[0116] In some implementations, a coding partition tree scheme or structure may be used. The coding partition tree schemes for the luma channel and the chroma channel may not need to be the same. In other words, the luma channel and the chroma channel may have independent coding tree structures. In addition, whether the luma channel and the chroma channel use the same or different coding partition tree structures and the actual coding partition tree structure to be used may depend on whether the slice being encoded is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma channel and the luma channel may have separate coding partition tree structures or coding partition tree structure patterns, while for a P slice or a B slice, the luma channel and the chroma channel may share the same coding partition tree scheme. When a separate coding partition tree structure or pattern is applied, the luma channel may be divided into CBs by one coding partition tree structure, and the chroma channel may be divided into chroma CBs by another coding partition tree structure.

[0117] The following describes a specific example implementation of coding block and transform block partitioning. In such an example implementation, the above-mentioned recursive quadtree splitting can be used to split the basic coding block into coding blocks. At each level, whether to continue to further quadtree splitting for a specific partition can be determined by local video data characteristics. The resulting CB can be at different quadtree splitting levels and can have different sizes. A decision can be made at the CB level (or CU level, for all three color channels) about whether to use inter-frame picture (time) prediction or intra-frame picture (space) prediction to encode the picture area. Each CB can be further split into one, two, four or other number of PBs according to the PB split type. Within a PB, the same prediction process can be applied, and the relevant information is transmitted to the decoder based on the PB. After obtaining the residual block by applying the prediction process based on the PB split type, the CB can be divided into TBs according to another quadtree structure similar to the coding tree for the CB. In this specific implementation, the CB or TB can be square in shape, but it is not necessarily limited to the square shape. In addition, in this specific example, for inter-frame prediction, the PB can be square or rectangular in shape, but for intra-frame prediction, the PB can be only square. The coding block can be further split into, for example, four square-shaped TBs. Each TB can be further recursively split (using quadtree splitting) into smaller TBs, which are called residual quadtrees (RQTs).

[0118] Another specific example of dividing a basic coding block into CB and other PBs and / or TBs is described below. For example, instead of using a multi-partition unit type such as Fig.10The multi-partition unit type shown can utilize a quadtree of nested multi-type trees using binary and ternary split partitioning structures. The separation of CB, PB, and TB concepts (i.e., dividing CB into PB and / or TB, and dividing PB into TB) can be abandoned unless it is needed for a CB whose size is too large for the maximum transform length, where such a CB may need to be further split. The example partitioning scheme can be designed to support greater flexibility in the CB partition shape, so that both prediction and transformation can be performed at the CB level without further partitioning. In such a coding tree structure, the CB can have a square shape or a rectangular shape. Specifically, the coding tree block (CTB) can first be divided by a quadtree structure. Then, the quadtree leaf nodes can be further divided by the multi-type tree structure. An example of a multi-type tree structure is shown in FIG. Fig.11 shown in . Specifically, Fig.11 The example multi-type tree structure includes four split types, which are called vertical binary split (SPLIT_BT_VER) (1102), horizontal binary split (SPLIT_BT_HOR) (1104), vertical ternary split (SPLIT_TT_VER) (1106) and horizontal ternary split (SPLIT_TT_HOR) (1108). The CB then corresponds to the leaves of the multi-type tree. In this example implementation, unless the CB is too large for the maximum transform length, the partition is used for both prediction and transform processing without any further partitioning. This means that in most cases, CB, PB and TB have the same block size in a quadtree with a nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is less than the width or height of the color component of the CB.

[0119] Fig.12 An example of a quadtree with a nested multi-type tree coding block structure for block partitioning of a CTB is shown. In more detail, Fig.12 The CTB 1200 is shown to be split into four square partitions 1202, 1204, 1206 and 1208 by a quadtree split. Further use is made for each of the quadtree split partitions. Fig.11 The decision of splitting is made on the multi-type tree structure. Fig.12 In the example of , partition 1204 is not further split. Partitions 1202 and 1208 each use another quadtree split. For partition 1202, the second level quadtree splits the upper left, upper right, lower left, and lower right partitions using the third level of the quadtree split, Fig.11 1104, no split and Fig.11 1108. The partition 1208 uses another quadtree split, and the second level quadtree split upper left, upper right, lower left and lower right partitions are respectively used Fig.11The third level split, no split, no split and Fig.11 1104. Two of the subdivisions of the third level upper left division 1208 are further split according to 1104 and 1108. Division 1206 adopts the following Fig.11 The second level splitting mode of 1102 is divided into two partitions, and the two partitions are divided according to Fig.11 1108 and 1102 are further split at the third level. Fig.11 The 1104 further applies a fourth level split to one of them.

[0120] For the above specific example, the maximum luma transform size may be 64×64, and the maximum supported chroma transform size may be different from luma, for example, 32×32. When the width or height of a luma coding block or a chroma coding block is greater than the maximum transform width or height, the luma coding block or the chroma coding block may be automatically split in the horizontal direction and / or vertical direction to meet the transform size limit in that direction.

[0121] In the above specific example of dividing the basic coding block into CBs, the coding tree scheme can support the ability of luma and chroma to have separate block tree structures. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU can share the same coding tree structure. For example, for I slices, luma and chroma can have separate coding block tree structures. When a separate block tree mode is applied, the luma CTB can be divided into luma CBs by one coding tree structure, and the chroma CTB can be divided into chroma CBs by another coding tree structure. This means that the CU in the I slice can include coding blocks for the luma component or coding blocks for two chroma components, and unless the video is monochrome, the CU in the P slice or B slice always includes coding blocks for all three color components.

[0122] Example implementations of partitioning a coding block or prediction block into transform blocks and the encoding order of the transform blocks are further described in detail below. In some example implementations, transform partitioning can support transform blocks of various shapes, such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, where the transform block size ranges from, for example, 4×4 to 64×64. In some implementations, if the coding block is less than or equal to 64×64, then the transform block partitioning can be applied only to the luma component, so that for the chroma block, the transform block size is the same as the coding block size. In addition, if the coding block width or height is greater than 64, both the luma coding block and the chroma coding block can be implicitly split into multiple min(W,64)×min(H,64) and min(W,32)×min(H,32) transform blocks, respectively.

[0123] In some example implementations, for both intra-coded blocks and inter-coded blocks, the coded blocks may be further divided into multiple transform blocks, where the division depth reaches a predefined number of levels (e.g., 2 levels). The transform block division depth and size may be related. An example mapping from the transform size of the current depth to the transform size of the next depth is shown below in Table 1.

[0124] Table 1: Transform partition size settings

[0125]

[0126] Based on the example mapping of Table 1, for a 1:1 square block, the next level transform split can create four 1:1 square sub-transform blocks. The transform partitioning can stop at, for example, 4×4. Therefore, the transform size of 4×4 at the current depth corresponds to the same size of 4×4 at the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform split will create two 1:1 square sub-transform blocks, and for a 1:4 / 4:1 non-square block, the next level transform split will create two 1:2 / 2:1 sub-transform blocks.

[0127] In some example implementations, for the luma component of an intra-coded block, additional restrictions may be applied. For example, for each level of transform partitioning, all sub-transform blocks may be restricted to have the same size. For example, for a 32×16 coding block, a level 1 transform split creates two 16×16 sub-transform blocks and a level 2 transform split creates eight 8×8 sub-transform blocks. In other words, the second level split must be applied to all first level sub-blocks to keep the transform unit sizes equal. Fig.13 1 shows an example of transform block partitioning of an intra-coded square block according to Table 1, and the coding order indicated by the arrows. Specifically, 1302 shows a square coding block. A first level splitting into 4 equal-sized transform blocks according to Table 1, and the coding order indicated by the arrows are shown in 1304. A second level splitting into 16 equal-sized transform blocks according to Table 1, and the coding order indicated by the arrows are shown in 1306.

[0128] In some example implementations, the above restrictions for intra-coding may not apply to the luma component of inter-coded blocks. For example, after the first level of transform splitting, any of the sub-transform blocks may be further split independently at more than one level. Therefore, the resulting transform blocks may or may not have the same size. Fig.14 An example splitting of inter-coded blocks into transform blocks and their coding order is shown in FIG. Fig.14In the example of FIG. 1402, an inter-coded block 1402 is split into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is split into four transform blocks of equal size. Then, only one of the four transform blocks (not all four transform blocks) is further split into four sub-transform blocks, resulting in a total of 7 transform blocks with two different sizes as shown by 1404. The example coding order of these 7 transform blocks is as follows: Fig.14 The arrow in 1404 is shown.

[0129] In some example implementations, some additional restrictions on transform blocks may apply to chroma components. For example, for chroma components, the transform block size may be as large as the coding block size, but not smaller than a predefined size, such as 8×8.

[0130] In some other example implementations, for coding blocks whose width (W) or height (H) is greater than 64, both the luma coding blocks and the chroma coding blocks may be implicitly split into multiple min(W, 64)×min(H, 64) and min(W, 32)×min(H, 32) transform units, respectively.

[0131] Fig.15 Another alternative example scheme for dividing a coding block or prediction block into transform blocks is further shown. Fig.15 As shown, instead of using recursive transform partitioning, a predefined set of partitioning types can be applied to the coding block according to the transform type of the coding block. Fig.15 In the specific example shown in , one of 6 example partition types can be applied to split the coding block into various numbers of transform blocks. Such a scheme can be applied to coding blocks or prediction blocks.

[0132] In more detail, Fig.15 As shown, Fig.15 The partitioning scheme of provides up to 6 partitioning types for any given transform type. In this scheme, a transform type can be assigned to each coding block or prediction block based on, for example, rate-distortion cost. In an example, the partitioning type assigned to a coding block or prediction block can be determined based on the transform partitioning type of the coding block or prediction block. A specific partitioning type can correspond to a transform block split size and mode (or partitioning type), such as Fig.15 As shown in the 4 partition types described in . The correspondence between various transform types and various partition types can be predefined. An example correspondence is shown below, where the uppercase label indicates the transform type that can be assigned to the coding block or prediction block based on the rate-distortion cost:

[0133] PARTITION_NONE: Allocate a transform size equal to the block size.

[0134] PARTITION_SPLIT: Allocate a transform size of 1 / 2 the block size in width and 1 / 2 the block size in height.

[0135] PARTITION_HORZ: Allocates a transform size with the same width as the block size and 1 / 2 the height of the block size.

[0136] PARTITION_VERT: Allocates a transform size with a width of 1 / 2 the block size and the same height as the block size.

[0137] PARTITION_HORZ4: Allocates a transform size with the same width as the block size and 1 / 4 the height of the block size.

[0138] PARTITION_VERT4: Allocates a transform size with a width of 1 / 4 the block size and the same height as the block size.

[0139] In the above example, Fig.15 The partition types shown all include uniform transform sizes for the partitioned transform blocks. This is only an example and not a limitation. In some other implementations, mixed transform block sizes may be used for the partitioned transform blocks of a particular partition type (or mode).

[0140] Turning to some example implementations of specific types of signaling for coding blocks / units, for each intra-coded unit and inter-coded unit, a flag, namely the skip_txfm flag, may be signaled in the coded bitstream, as shown in the example syntax of Table 2 and represented by the read_skip() function for retrieving these flags from the bitstream. This flag may indicate whether the transform coefficients in the current coding unit are all zero. In some example implementations, if this flag is signaled using, for example, a value of 1, then another transform coefficient related syntax such as EOB (end of block) does not need to be signaled for any color coded block in the coding unit and may be derived as a value or data structure predefined for and associated with zero transform coefficient blocks. For inter-coded blocks, as shown in the example of Table 2, this flag may be signaled after the skip_mode flag, which indicates that the coding unit may be skipped for various reasons. When skip_mode is true, the coding unit should be skipped and no skip_txfm flag needs to be signaled, and the skip_txfm flag is inferred to be 1. Otherwise, if skip_mode is false, more information about the coding unit will be included in the bitstream, and the skip_txfm flag will be additionally signaled to indicate whether the coding unit is all zeros.

[0141] Table 2: Skip patterns and skip syntax

[0142] Intra-mode information syntax:

[0143]

[0144] Inter-frame mode information syntax:

[0145]

[0146] Skip Syntax

[0147]

[0148] In the following example implementations, the term chrominance channel may generally refer to both Cb and Cr color components (or channels), or both U and V color components (or channels). The term luma channel may include a luma component or Y component. A luma component or channel may be referred to as a luma color component or channel. Y, U, and V are used below to represent the three color components. In addition, the terms "coding block" and

[0149] "Encoded" blocks are used interchangeably to refer to blocks to be encoded or blocks that have been encoded. They can be blocks of any of the three color components. Three corresponding color encoded / encoded blocks can be used for an encoded / encoded unit.

[0150] Turning to the encoding and decoding (entropy coding) of the transform coefficients of the residual in each of the color components, for each transform block, the transform coefficient encoding may first signal a skip symbol, followed by a transform kernel type and an end-of-block (EOB) position when the skip symbol is zero (indicating the presence of non-zero coefficients). Each coefficient value is then mapped into multiple level maps (amplitude maps) and signs.

[0151] After encoding the EOB position, the low level map and the mid level map may be encoded in reverse scanning order, the former indicating the case where the coefficient magnitude is within the low level (e.g., between 0 and 2), and the latter indicating the case where the range is within the mid level (e.g., between 3 and 14). The next step is to encode the signs of the coefficients and the residual values ​​of the coefficients greater than the high level (e.g., 14) in a forward scanning order by, for example, Exp-Golomb coding.

[0152] With regard to the use of context modeling, the low-level mapping coding can contain the transform size and direction and up to five neighboring coefficient information. On the other hand, the medium-level mapping coding can follow a similar approach to the low-level amplitude coding, except that the number of neighboring coefficients is reduced to a smaller number (e.g., two). Without any context model, the example Exp-Golomb code of the residual level and the sign of the AC coefficient are encoded, while the sign of the DC coefficient is encoded using the dc sign of its neighboring transform block.

[0153] In some exemplary implementations, the chroma residual may be jointly encoded. Such a coding scheme may be based on some statistical correlation between the chroma channels. For example, in many cases, the Cr and Cb chroma coefficients may be similar in magnitude but opposite in sign, and thus, for example, in the case of signaling transform coefficients, the transform block level may be jointly encoded to improve coding efficiency by introducing only small color distortions. The use (activation) of the joint chroma coding mode may be indicated, for example, by a joint chroma coding flag (e.g., a TU level flag tu_joint_cbcr_residual_flag), and the selected joint mode may be implicitly indicated by the chroma CBF.

[0154] Specifically, the flag tu_joint_cbcr_residual_flag may be present when either or both of the chroma CBFs of a TU (transform block) are equal to 1. In the PPS and slice header, chroma quantization parameter (QP) offset values ​​may be signaled for the joint chroma residual coding mode to distinguish from the chroma QP offset values ​​signaled for the conventional chroma residual coding mode. These chroma QP offset values ​​may be used to derive chroma QP values ​​for those blocks encoded using the joint chroma residual coding mode. When the corresponding joint chroma coding mode (mode 2 in Table 3) is active in a TU, the chroma QP offset may be added to the chroma QP derived by applying luma during quantization and decoding of the TU. For other modes (modes 1 and 3 in Table 3), the chroma QP may be derived in the same manner as for conventional Cb or Cr blocks. The reconstruction process of the chroma residuals (resCb and resCr) from the transmitted transform blocks is depicted in Table 3. When this mode is activated (mode 2), a single joint chroma residual block (resJointC[x][y] in Table 3) may be signaled, and the residual block for Cb (resCb) and the residual block for Cr (resCr) may be derived taking into account information such as tu_cbf_cb, tu_cbf_cr, and CSign as a specified symbol value in, for example, the slice header rather than in the transform block level. In some implementations, CSign may be -1 most of the time.

[0155] The above three example joint chroma coding modes may be supported only in intra-coded CUs. In inter-coded CUs, only mode 2 may be supported. Therefore, for inter-coded CUs, the syntax element tu_joint_cbcr_residual_flag is present only when both chroma CBFs are 1.

[0156] Table 1. Reconstruction of chroma residual. The value CSign is the sign value (+1 or -1) specified in the slice header, and resJointC[][] is the transmitted residual.

[0157]

[0158] The above joint chroma coding scheme assumes some correlation between transform coefficients between co-located Cr and Cb transform blocks. These assumptions are usually statistical and thus introduce distortions in some cases. In particular, when one of the color coefficients in the transform block is non-zero while the coefficient of the other color component is zero, then some of the assumptions made in the joint chroma coding scheme are incorrect and such coding will not save any coding bits (since one of the chroma coefficients is zero anyway).

[0159] In various example implementations below, a cross-component coding scheme at the coefficient level (i.e., transform coefficient by transform coefficient) is described that exploits some correlation between co-located transform coefficients (same position in the frequency domain) of color components. Such a scheme is particularly useful for transform blocks (or units) where the coefficients of one color component are zero and the corresponding transform coefficients of another color component are non-zero. For those zero and non-zero color coefficient pairs, before or after dequantization, the non-zero color coefficient can be used to estimate the original small value of the zero-coded coefficient of the other color component (which may originally be a non-zero but smaller value before quantization in the encoding process), thereby potentially recovering some information lost during, for example, quantization during encoding. Due to the statistically existing correlation between colors, the information lost during quantization to zero can be recovered. Such cross-component coding recovers the lost information to some extent without significant coding cost (of the zero coefficients).

[0160] In particular, a cross-component coefficient sign encoding method can be implemented, which encodes the coefficient sign of the second color component using the coefficient sign value of the first color component. In a more specific example, the sign value of the Cb transform coefficient can be used as a context for encoding the sign of other Cr transform coefficients. Such cross-component encoding can be implemented coefficient by coefficient for the transform coefficient pairs of the color components. The basic principles of such implementations and other implementations described in further detail below are not limited to Cr and Cb components. They are applicable between any two of the three color components. In this regard, the brightness channel is considered to be one of the color components.

[0161] The following example implementations may be used individually or combined in any order. The term block size may refer to either the block width or the block height, or the maximum of the width and height, or the minimum of the width and height, or the area size (width*height), or the aspect ratio of the block (width:height, or height:width). The term "horizontal value" or "horizontal" may refer to the magnitude of the transform coefficient value.

[0162] In some example implementations, the horizontal value and / or the sign value of the transform coefficient of the first color component may be used to derive an offset value that is added to the horizontal value of the transform coefficient of the second color component.

[0163] In some further implementations, the transform coefficients of the first color component and the transform coefficients of the second color component used to generate the offset are co-located (same coordinates in the frequency domain, eg, not cross-frequency estimates).

[0164] Although the above-mentioned first color component and second color component may not be limited to specific color components, in some example implementations, the first color component may be Cb (or Cr) and the second color component may be Cr (or Cb).

[0165] In some specific example implementations, the first color component may be luminance, and the second color component may be one of Cb and Cr.

[0166] In some specific example implementations, the first color component may be one of Cb and Cr, and the second color component may be luminance.

[0167] In some example implementations, the quantized transform coefficients of the first color component may be non-zero, and the quantized transform coefficients of the second color component may be zero. Thus, relatively small non-zero information of the original transform coefficients of the second component may be lost during the encoding process due to quantization, and example implementations described herein may use corresponding non-zero color components that are statistically correlated with zero-coefficient color components to recover some of the lost information.

[0168] In some example implementations, the sign value of the transform coefficient of the first color component may be used to derive an offset value that is added to the dequantized transform coefficient horizontal value of the second color component.

[0169] In some example implementations, the sign value of the transform coefficient of the first color component is used to derive an offset value that is added to the horizontal value of the transform coefficient of the second color component prior to dequantization.

[0170] In some example implementations, in the case where the sign value of the transform coefficient of the first color component is positive (or negative), a negative (or positive) offset value is added to the dequantized transform coefficient level value of the second color component for reconstructing the transform coefficient value of the second color component. In other words, the sign value of the transform coefficient of the first color component and the sign value of the offset value added to the transform coefficient level value of the second color component have different sign values. Such an implementation is consistent with some statistical observations that the two chrominance components generally have opposite signs for the transform coefficients.

[0171] In some example implementations, whether the sign value of the transform coefficient of the first color component and the sign value of the offset value added to the transform coefficient horizontal value of the second color component have opposite sign values ​​is signaled in a high-level syntax, including but not limited to: SPS, VPS, PPS, APS, picture header, frame header, slice header, tile header, CTU header. This is similar to the above-mentioned signaling scheme of the joint chroma coding scheme.

[0172] In some example implementations, the offset value added to the transform coefficient level value of the second color component may depend on both the sign and the level of the transform coefficient of the first color component.

[0173] In some example implementations, the magnitude of the offset value added to the transform coefficient level value of the second color component may depend on the transform coefficient level of the first color component.

[0174] In some example implementations, the magnitude of the offset value added to the transform coefficient level value of the second color component may be predefined for each input value of the coefficient level of the transform coefficient of the first color component.

[0175] In some example implementations, the offset value added to the transform coefficient horizontal value of the second color component may depend on the frequency that the transform coefficient is in. For example, for higher frequency coefficients, the offset value may be smaller.

[0176] In some example implementations, the offset value added to the transform coefficient horizontal value of the second color component may depend on the block size of the block to which the transform coefficient belongs. For example, for larger block sizes, the offset value may generally be smaller.

[0177] In some example implementations, the offset value added to the transform coefficient horizontal value of the second color component may depend on whether the second component is a luminance (Y) component or a chrominance (Cb or Cr) component. For example, in the case where the second color component is luminance, the offset value may be smaller.

[0178] Fig.16A flowchart 1600 of an example method that follows the basic principles of the above implementation of cross-component decoding is shown. The example method flow starts at 1601. In S1610, a first transform coefficient of a first color component is extracted from a bitstream of an encoded video. In S1620, a second transform coefficient of a second color component is extracted from the bitstream of the encoded video. In S1630, an offset value is derived based on an amplitude or a sign value of the first transform coefficient. In S1640, the offset value is added to the amplitude of the second transform coefficient to generate a modified second transform coefficient of the second color component. In S1650, the encoded video is reconstructed based on at least the first transform coefficient of the first color component and the modified second transform coefficient of the second color component. The example method flow ends at S1699.

[0179] The embodiments of the present disclosure may be used alone or in any order. In addition, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to luminance blocks or chrominance blocks.

[0180] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Fig.17 A computer system (1700) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0181] Computer software may be encoded using any suitable machine code or computer language, which may be assembled, compiled, linked, or the like to create code comprising instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), or the like, or through interpretation, microcode execution, or the like.

[0182] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smart phones, gaming devices, IoT devices, and the like.

[0183] Fig.17 The components for the computer system (1700) shown in the example are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement related to any one or combination of components shown in the exemplary embodiment of the computer system (1700).

[0184] The computer system (1700) may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), olfactory input (not depicted). The human-machine interface device may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0185] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard (1701), mouse (1702), touchpad (1703), touch screen (1710), data gloves (not shown), joystick (1705), microphone (1706), scanner (1707), camera (1708).

[0186] The computer system (1700) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include: tactile output devices (e.g., tactile feedback through a touch screen (1710), a data glove (not shown), or a joystick (1705), but there may also be tactile feedback devices that are not used as input devices); audio output devices (e.g., speakers (1709), headphones (not depicted)); visual output devices (e.g., screens (1710), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which may be capable of outputting two-dimensional visual outputs or more than three-dimensional outputs through means such as stereoscopic image output; virtual reality glasses (not depicted); holographic displays and smoke generators (not depicted)); and printers (not depicted).

[0187] The computer system (1700) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (1720) with CD / DVD etc. media (1721), thumb drives (1722), removable hard drives or solid-state drives (1723), legacy magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD based devices such as security dongles (not depicted), etc.

[0188] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0189] The computer system (1700) may also include an interface (1754) to one or more communication networks (1755). The network may be, for example, a wireless network, a wired network, an optical network. The network may also be a local area network, a wide area network, a metropolitan area network, an in-vehicle and industrial network, a real-time network, a delay-tolerant network, etc. Examples of networks include: a local area network (e.g., Ethernet, wireless LAN), a cellular network including GSM, 3G, 4G, 5G, LTE, etc., a television wired or wireless wide area digital network including cable television, satellite television, and terrestrial broadcast television, a vehicle and industrial network including a CAN bus, etc. Some networks typically require an external network interface adapter attached to some common data port or peripheral bus (1749) (e.g., a USB port of the computer system (1700)); other networks are typically integrated into the core of the computer system (1700) by attaching to the system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smart phone computer system). Using any of these networks, the computer system (1700) can communicate with other entities. Such communications may be one-way receive only (e.g., broadcast television), one-way send only (e.g., a CAN bus to certain CAN bus devices), or bidirectional (e.g., to other computer systems using a local area digital network or a wide area digital network). Certain protocols and protocol stacks may be used on each of these networks and network interfaces as described above.

[0190] The above-mentioned human-machine interface device, human-accessible storage device, and network interface may be attached to the core ( 1740 ) of the computer system ( 1700 ).

[0191] The core (1740) may include one or more central processing units (CPUs) (1741), graphics processing units (GPUs) (1742), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1743), hardware accelerators (1744) for certain tasks, graphics adapters (1750), and the like. These devices, along with read-only memory (ROM) (1745), random access memory (1746), internal mass storage devices (e.g., internal non-user accessible hard drives, SSDs, etc.) (1747), may be connected via a system bus (1748). In some computer systems, the system bus (1748) may be accessed in the form of one or more physical plugs to allow expansion by additional CPUs, GPUs, and the like. Peripheral devices may be attached to the core's system bus (1748) directly or via a peripheral bus (1749). In an example, a screen (1710) may be connected to a graphics adapter (1750). The architecture of the peripheral bus includes PCI, USB, and the like.

[0192] The CPU (1741), GPU (1742), FPGA (1743) and accelerator (1744) can execute certain instructions, which in combination can constitute the computer code mentioned above. The computer code can be stored in ROM (1745) or RAM (1746). Transient data can also be stored in RAM (1746), while permanent data can be stored in, for example, an internal mass storage device (1747). Fast storage and retrieval of any of the memory devices can be achieved by using a cache memory, which can be closely associated with one or more CPUs (1741), GPUs (1742), mass storage devices (1747), ROMs (1745), RAMs (1746), etc.

[0193] Computer readable media may have computer codes thereon for performing various computer-implemented operations. These media and computer codes may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0194] As a non-limiting example, a computer system (1700) having an architecture and in particular a core (1740) can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software implemented in one or more tangible computer-readable media. Such a computer-readable medium can be a medium associated with a user-accessible mass storage device as described above, as well as certain storage devices of the core (1740) having non-transitory properties, such as a core internal mass storage device (1747) or ROM (1745). Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core (2640). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the core (2640) and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform specific processing or specific parts of specific processing described herein, including defining data structures stored in RAM (2646) and modifying such data structures according to processing defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise implemented in circuits (e.g., accelerators (2644)) that may operate in place of or in conjunction with software to perform specific processing or specific portions of specific processing described herein. Where appropriate, references to software may encompass logic, and vice versa. Where appropriate, references to computer-readable media may include circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits embodying logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0195] Although the present disclosure has described several exemplary embodiments, there are changes, permutations and various replacement equivalents that fall within the scope of the present disclosure. In the above implementations and embodiments, any operation of the process can be combined or arranged in any quantity or order as needed. In addition, two or more operations in the above process can be performed in parallel. Therefore, it should be appreciated that those skilled in the art will be able to conceive many systems and methods, although not explicitly shown or described herein, these systems and methods implement the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

[0196] Appendix A: Acronyms

[0197] JEM: Joint Development Model

[0198] VVC: Versatile Video Coding

[0199] BMS: Benchmark Set

[0200] MV: Motion Vector

[0201] HEVC: High Efficiency Video Coding

[0202] SEI: Supplemental Enhancement Information

[0203] VUI: Video Availability Information

[0204] GOP: Group of Pictures

[0205] TU: Transform Unit

[0206] PU: prediction unit

[0207] CTU: Coding Tree Unit

[0208] CTB: Coding Tree Block

[0209] PB: prediction block

[0210] HRD: Hypothetical Reference Decoder

[0211] SNR: Signal to Noise Ratio

[0212] CPU: Central Processing Unit

[0213] GPU: Graphics Processing Unit

[0214] CRT: cathode ray tube

[0215] LCD: Liquid Crystal Display

[0216] OLED: Organic Light Emitting Diode

[0217] CD: Compact Disc

[0218] DVD: Digital Video Disc

[0219] ROM: Read Only Memory

[0220] RAM: Random Access Memory

[0221] ASIC: Application-Specific Integrated Circuit

[0222] PLD: Programmable Logic Device

[0223] LAN: Local Area Network

[0224] GSM: Global System for Mobile Communications

[0225] LTE: Long Term Evolution

[0226] CANBus: Controller Area Network Bus

[0227] USB: Universal Serial Bus

[0228] PCI: Peripheral Component Interconnect

[0229] FPGA: Field Programmable Gate Array

[0230] SSD: Solid State Drive

[0231] IC: Integrated Circuit

[0232] HDR: High Dynamic Range

[0233] SDR: Standard Dynamic Range

[0234] JVET: Joint Video Experts Team

[0235] MPM: Most Probable Mode

[0236] WAIP: Wide Angle Intra Prediction

[0237] CU: Coding Unit

[0238] PU: prediction unit

[0239] TU: Transform Unit

[0240] CTU: Coding Tree Unit

[0241] PDPC: Position Dependent Prediction Combination

[0242] ISP: Intra-frame sub-partitioning

[0243] SPS: Sequence Parameter Set

[0244] PPS: Picture Parameter Set

[0245] APS: Adaptive Parameter Set

[0246] VPS: Video Parameter Set

[0247] DPS: Decoding Parameter Set

[0248] ALF: Adaptive Loop Filter

[0249] SAO: Sample Adaptive Offset

[0250] CC-ALF: Cross-Component Adaptive Loop Filter

[0251] CDEF: Constrained Directional Enhancement Filter

[0252] CCSO: Cross Component Sample Offset

[0253] LSO: Local Sample Offset

[0254] LR: Loop Restoration Filter

[0255] AV1: AOMedia Video 1

[0256] AV2: AOMedia Video 2

Claims

1. A method for video decoding, characterized in that: include: extracting first transform coefficients of a first color component from a bitstream of encoded video; extracting second transform coefficients for a second color component from a bitstream of the encoded video; deriving an offset value based on a magnitude and a sign of the first transform coefficient; adding the offset value to the magnitude of the second transform coefficient to generate a modified second transform coefficient for the second color component; as well as reconstructing the encoded video based on at least a first transform coefficient of the first color component and a modified second transform coefficient of the second color component; The first transform coefficient and the second transform coefficient are coefficients at the same position in the frequency domain; The first transform coefficient is non-zero after quantization; as well as The second transform coefficient is zero after quantization.

2. The method according to claim 1, characterized in that The first color component includes one chrominance component and the second color component includes another chrominance component.

3. The method according to claim 1, characterized in that The first color component includes a luminance component and the second color component includes a chrominance component.

4. The method according to claim 1, characterized in that: The first color component includes a chrominance component and the second color component includes a luminance component.

5. The method according to claim 1, characterized in that: The first transform coefficient comprises a sign value and an amplitude value; and Deriving the offset value includes deriving the offset value based on a sign value and a magnitude value of the first transform coefficient.

6. The method according to any one of claims 1 to 5, characterized in that Adding the offset value to the magnitude of the second transform coefficient includes adding the offset value to the magnitude of the second transform coefficient after dequantization.

7. The method according to any one of claims 1 to 5, characterized in that Adding the offset value to the magnitude of the second transform coefficient includes adding the offset value to the magnitude of the second transform coefficient prior to dequantization.

8. The method according to any one of claims 1 to 5, characterized in that: The offset value has a sign opposite to a sign of the first transform coefficient; and Adding the offset value to the magnitude of the second transform coefficient includes adding the offset value to the magnitude of the second transform coefficient after dequantization.

9. The method according to any one of claims 1 to 5, characterized in that: signaling whether the sign value of the first transform coefficient and the sign of the offset value are opposite in one of the following: Video parameter set VPS; Sequence parameter set SPS; Picture parameter set PPS; Adaptive parameter set APS; Frame header; Opening a coding tree unit header; or Tile header.

10. A device for video decoding, characterized in that include: A first extraction module configured to extract a first transform coefficient of a first color component from a bit stream of an encoded video; A second extraction module configured to extract a second transform coefficient of a second color component from a bit stream of the encoded video; a calculation module configured to derive an offset value based on the magnitude and sign of the first transform coefficient; an adding module configured to add the offset value to the magnitude of the second transform coefficient to generate a modified second transform coefficient for the second color component; as well as a reconstruction module configured to reconstruct the encoded video based on at least a first transform coefficient of the first color component and a modified second transform coefficient of the second color component; The first transform coefficient and the second transform coefficient are coefficients at the same position in the frequency domain; The first transform coefficient is non-zero after quantization; as well as The second transform coefficient is zero after quantization.

11. A device for video decoding, characterized in that include: at least one memory configured to store program code; as well as At least one processor is configured to read the program code to execute the method according to any one of claims 1 to 9.

12. A non-transitory computer readable medium for storing computer instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 9.

13. A method for processing a video bitstream, characterized in that: The video bit stream is decoded according to the method according to any one of claims 1 to 9.

Images