Video stream decoding method, device and computer readable storage medium

Abstract

The present disclosure relates to adaptive loop filtering (ALF) for cross component sample offset (CCSO), local sample offset (LSO) and generalized sample offset (GSO). The ALF uses reconstructed samples of a first color component as input (e.g., Y or Cb or Cr). Based on syntax values ​​in an encoded video stream, it is determined that at least one of GSO, LSO or CCSO is enabled for a frame. The encoded video stream is decoded by applying at least one of GSO, LSO or CCSO at a block level.

Description

[0001] Incorporated by Reference

[0002] This application is based on and claims the benefit of priority to U.S. non-provisional application No. 17 / 977,160, entitled “ADAPTIVE APPLICATION OF GENERALIZED SAMPLE OFFSET,” filed on October 31, 2022, U.S. Provisional Application No. 63 / 280,503, entitled “ADAPTIVE APPLICATION OF GENERALIZED SAMPLE OFFSET,” filed on November 17, 2021, and U.S. Provisional Application No. 63 / 289,139, entitled “ADAPTIVE APPLICATION OF GENERALIZED SAMPLE OFFSET,” filed on December 13, 2021, the contents of each of the above U.S. non-provisional applications and U.S. provisional applications being incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure describes a set of advanced video coding techniques. More specifically, the disclosed techniques relate to cross-component sample offset (CCSO), local sample offset (LSO) and generalized sample offset (GSO). The present disclosure more particularly relates to methods, apparatus and computer-readable storage media for decoding video streams. Background Art

[0004] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. To the extent the work described in this background section, the work of the presently named inventors and aspects of the description that may not otherwise be qualified as prior art at the time the application was filed are neither explicitly nor implicitly admitted as prior art to the present disclosure.

[0005] Video encoding and video decoding can be performed using inter-picture prediction with motion compensation. An uncompressed digital video may include a series of pictures, each picture having a spatial dimension of, for example, 1920×1080 luminance samples and associated full chrominance samples or subsampled chrominance samples. The series of pictures may have a fixed or variable picture rate (alternatively referred to as a frame rate), such as 60 pictures per second or 60 frames per second. Uncompressed video has specific bit rate requirements for streaming or data processing. For example, a video with 4:2:0 chrominance subsampling, a frame rate of 60 frames per second, and a pixel resolution of 1920×1080 at 8 bits per pixel per color channel requires a bandwidth of nearly 1.5 Gbit / s. One hour of such a video requires more than 600 gigabytes of storage space.

[0006] One purpose of video encoding and decoding can be to reduce redundancy in an uncompressed input video signal by compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by two orders of magnitude or more. Both lossless and lossy compression and their combinations can be used. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal via a decoding process. Lossy compression refers to an encoding / decoding process in which the original video information is not fully retained during the encoding process and cannot be fully restored during the decoding process. When lossy compression is used, the reconstructed signal may not be the same as the original signal, but the distortion between the original signal and the reconstructed signal is small enough that the reconstructed signal is useful for the intended application despite some information loss. In the case of video, lossy compression is widely used in many applications. The amount of distortion allowed depends on the application. For example, users of certain consumer video streaming applications can tolerate higher distortion than users of movie or television broadcast applications. The compression ratio achievable by a specific encoding algorithm can be selected or adjusted to reflect various distortion tolerances: higher tolerable distortion generally allows encoding algorithms that produce higher losses and higher compression ratios.

[0007] Video encoders and decoders may utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transforms, quantization, and entropy coding.

[0008] Video codec techniques may include techniques known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into sample blocks. When all sample blocks are encoded in intra-mode, the picture may be referred to as an intra-picture. Intra-pictures and their derivatives such as independent decoder refresh pictures can be used to reset the decoder state, and can therefore be used as the first picture in an encoded video bitstream and video session or as a still image. The samples of the intra-predicted block can be transformed into the frequency domain, and the generated transform coefficients can be quantized before entropy coding. Intra-prediction represents a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform, and the smaller the AC coefficient, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0009] Conventional intra-frame coding, such as known from, for example, the MPEG-2 generation of coding techniques, does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to encode / decode blocks based on, for example, metadata and / or surrounding sample data obtained during encoding / decoding that is spatially adjacent and precedes the data block being intra-coded or decoded in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, and does not use reference data from other reference pictures.

[0010] There may be many different forms of intra-frame prediction. When more than one such technique may be used in a given video coding technique, the technique used may be referred to as an intra-frame prediction mode. One or more intra-frame prediction modes may be provided in a particular codec. In some cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information of a video block and the intra-frame coding parameters may be encoded separately or included together in a mode codeword. The codeword to be used for a given mode, sub-mode, and / or parameter combination may have an impact on the coding efficiency gain through intra-frame prediction, and therefore may have an impact on the entropy coding technique used to convert the codeword into a bitstream.

[0011] Some mode of intra prediction was introduced by H.264, refined in H.265, and further refined in newer coding techniques such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). Typically, for intra prediction, the predictor block can be formed using the values ​​of neighboring samples that have become available. For example, the available values ​​of a particular set of neighboring samples along a particular direction and / or line can be copied into the predictor block. A reference to the direction in use can be encoded in the bitstream, or the reference to the direction in use can be predicted by itself.

[0012] The loop filtering method may include cross-component sample offset (CCSO) to reduce the distortion of the reconstructed samples. In CCSO, a nonlinear mapping is used to derive an output offset considering the processed input reconstructed samples of a first color component, and the output offset is added to the reconstructed samples of another color component during the filtering process of the proposed CCSO. Local sample offset (LSO) is similar to CCSO, but the output offset is applied to the color component that is the same color component as the color component for which the reconstructed samples are used as input to the filtering process. The art lacks a generalized design that considers both types of offset values. Summary of the invention

[0013] Embodiments of the present disclosure provide methods and apparatus for cross component sample offset (CCSO), local sample offset (LSO), and generalized sample offset (GSO). Adaptive loop filtering (ALF) uses reconstructed samples of a first color component as input (e.g., Y or Cb or Cr). For CCSO, the output is applied to a second color component, which is a different color component of the first color component. For LSO, the output is applied to the first color component. For GSO, the combined ALF can be generalized for CCSO and LSO by considering the difference value between adjacent samples of the collocated (or current) sample and also considering the level value of the collocated (or current) sample. At least one of CCSO, LSO, or GSO can be enabled for a frame based on a syntax value included in the encoded video stream.

[0014] In one embodiment, a method includes: receiving an encoded video stream, the encoded video stream including a syntax element indicating whether a sample offset filtering process is applied to a frame in the encoded video stream at a block level; determining to enable at least one of generalized sample offset (GSO), local sample offset (LSO), or cross-component sample offset (CCSO) for the frame based on the syntax element being a predefined value; and decoding the encoded video stream by applying at least one of GSO, LSO, or CCSO at a block level.

[0015] In another embodiment, a device for decoding a video stream includes: a memory storing a plurality of instructions; and a processor configured to execute the plurality of instructions, and when executing the plurality of instructions, the processor is configured to: receive an encoded video stream, the encoded video stream including a syntax element indicating whether a sample offset filtering process is applied to a frame in the encoded video stream at a block level; determine whether to enable at least one of generalized sample offset (GSO), local sample offset (LSO), or cross-component sample offset (CCSO) for the frame at a block level based on the syntax element being a predefined value; and decode the encoded video stream by applying at least one of GSO, LSO, or CCSO at the block level.

[0016] In another embodiment, a non-transitory computer-readable storage medium stores a plurality of instructions executable by a processor, wherein the plurality of instructions, when executed by the processor, are configured to cause the processor to: receive an encoded video stream, the encoded video stream comprising a syntax element indicating whether a sample offset filtering process is applied to a frame in the encoded video stream at a block level; determine, based on the syntax element being a predefined value, whether to enable at least one of generalized sample offset (GSO), local sample offset (LSO), or cross-component sample offset (CCSO) for the frame; and decode the encoded video stream by applying at least one of GSO, LSO, or CCSO at a block level.

[0017] In some other implementations, a device for processing video information is disclosed. The device may include a circuit system configured to perform any one of the above method implementations.

[0018] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions which, when executed by a computer for video decoding and / or encoding, cause the computer to perform a method for video decoding and / or encoding.

[0019] The generalized sample offset (GSO) proposed in the present invention is a generalized design that considers two types of offset values, including a gradient offset (GO) and a band offset (BO). The gradient offset considers the difference value between adjacent samples of the collocated or current sample, and the band offset (BO) considers the level value of the current sample itself. At least one of CCSO, LSO or GSO can be enabled for a frame based on a syntax value included in an encoded video stream, thereby improving the decoding efficiency of the video stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which:

[0021] Figure 1A A schematic diagram showing an exemplary subset of intra prediction directional modes.

[0022] Figure 1B A diagram of exemplary intra prediction directions is shown.

[0023] Figure 2 A schematic diagram showing spatial merging candidates of a current block and its surroundings for motion vector prediction in an example is shown.

[0024] Figure 3 A schematic diagram showing a simplified block diagram of a communication system according to an example embodiment.

[0025] Figure 4 A schematic diagram showing a simplified block diagram of a communication system according to another example implementation.

[0026] Figure 5 A schematic diagram showing a simplified block diagram of a video decoder according to an example implementation.

[0027] Figure 6 A schematic diagram showing a simplified block diagram of a video encoder according to an example implementation.

[0028] Figure 7 A block diagram of a video encoder according to another example embodiment is shown.

[0029] Figure 8 A block diagram of a video decoder according to another example embodiment is shown.

[0030] Fig. 9 A scheme for dividing coding blocks according to an exemplary embodiment of the present disclosure is shown;

[0031] Fig.10 Another scheme of coding block partitioning according to an example implementation of the present disclosure is shown.

[0032] Fig.11Another scheme of coding block partitioning according to an example implementation of the present disclosure is shown.

[0033] Fig.12 An example of partitioning a basic block into coding blocks according to an example partitioning scheme is shown;

[0034] Fig.13 An example ternary tree partitioning scheme is shown;

[0035] Fig.14 An example quadtree binary tree coding block partitioning scheme is shown;

[0036] Fig.15 A scheme for dividing a coding block into a plurality of transform blocks and a coding order of the transform blocks according to an example implementation of the present disclosure is shown;

[0037] Fig.16 Another scheme for dividing a coding block into a plurality of transform blocks and the coding order of the transform blocks according to an example implementation of the present disclosure is shown;

[0038] Fig.17 Another scheme for dividing a coding block into a plurality of transform blocks according to an example implementation of the present disclosure is shown;

[0039] Fig.18 An example adaptive loop filter (ALF) shape is shown;

[0040] Fig.19a shows the sub-sampling positions in the Laplacian calculation of the vertical gradient;

[0041] Fig.19b shows the sub-sampling locations in the Laplacian calculation of the horizontal gradient;

[0042] Fig.19c shows the sub-sampling positions in the Laplacian calculation of the diagonal gradient;

[0043] Fig.19d shows the sub-sampling positions in the Laplacian calculation of another diagonal gradient;

[0044] Fig. 20 An example of modified block classification at a virtual boundary is shown;

[0045] Fig.21 An example of modified adaptive loop filtering of the luma component at a virtual boundary is shown;

[0046] Fig. 22An example of a picture quadtree partitioning aligned with the largest coding unit (LCU) is shown;

[0047] Fig.23 An example of a quadtree split flag encoded in z-order is shown;

[0048] Fig.24 An example of a cross-component adaptive loop filter (CC-ALF) placement is shown;

[0049] Fig.25 An example of a diamond filter is shown;

[0050] Fig.26 Example positions of chroma samples relative to luma samples are shown;

[0051] Fig. 27 An example of a direction search is shown;

[0052] Fig.28 An example of subspace projection is shown;

[0053] Fig.29 An example of a filter support region is shown;

[0054] Fig.30 An example loop filter circuit is shown;

[0055] Fig.31 An example input of a cross-component sample offset (CCSO) is shown;

[0056] Fig.32 An example filter shape in cross-component sample offset (CCSO) is shown;

[0057] Fig.33 An example pixel pattern is shown;

[0058] Fig.34 A flowchart showing a method according to an example implementation of the present disclosure;

[0059] Fig.35 A flowchart illustrating another method according to an example implementation of the present disclosure; and

[0060] Fig.36 A schematic diagram of a computer system according to an example implementation of the present disclosure is shown. DETAILED DESCRIPTION

[0061] Throughout the specification and claims, terms may have slightly different meanings that are suggested or implied in the context beyond the explicitly stated meaning. The phrases "in one embodiment" or "in some embodiments" as used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" as used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" as used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" as used herein do not necessarily refer to different implementations. For example, it is intended that the claimed subject matter includes a combination of the whole or part of the exemplary embodiments / implementations.

[0062] Typically, terms can be understood at least in part from usage in context. For example, terms such as "and", "or" or "and / or" as used herein can include a variety of meanings depending at least in part on the context in which such terms are used. Typically, if "or" is used for an association list, such as A, B, or C, "or" is intended to mean A, B, and C, which are used here in an inclusive sense; and A, B, or C, which are used here in an exclusive sense. In addition, depending at least in part on the context, the terms "one or more" or "at least one" as used herein can be used to describe any feature, structure, or characteristic in a singular sense, or can be used to describe a combination of features, structures, or characteristics in a plural sense. Similarly, terms such as "one", "an", or "the" can also be understood to convey singular usage or to convey plural usage, which depends at least in part on the context. In addition, the term "based on" or "determined by..." can be understood to not necessarily be intended to convey an exclusive set of factors, and can alternatively allow the presence of additional factors that are not necessarily explicitly described, which also depends at least in part on the context.

[0063] Reference Figure 1A , depicted at the bottom right is a subset of nine predictor directions specified in the 33 possible intra-frame predictor directions of H.265 (corresponding to the 33 angular modes of the 35 intra-frame modes specified in H.265). The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which neighboring samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted based on one or more neighboring samples at a 45 degree angle to the horizontal direction to the upper right. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more neighboring samples at a 22.5 degree angle to the horizontal direction to the lower left of sample (101).

[0064] Still refer to Figure 1A, depicted at the top left is a square block (104) of 4×4 samples (indicated by the dashed bold line). The square block (104) includes 16 samples, each of which is labeled by "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in block (104) in both the Y dimension and the X dimension. Since the size of the block is 4×4 samples, S44 is at the bottom right. Example reference samples are further shown, which follow a similar numbering scheme. The reference samples are labeled by R, their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, prediction samples that are adjacently adjacent to the block in reconstruction are used.

[0065] The intra picture prediction of block 104 may start by copying reference sample values ​​from neighboring samples according to a signaled prediction direction. For example, assume that the encoded video bitstream includes the following signaling, which indicates the prediction direction of arrow (102) for this block 104, i.e., the samples are predicted according to one or more prediction samples at an angle of 45 degrees to the horizontal direction to the upper right. In this case, samples S41, S32, S23 and S14 are predicted according to the same reference sample R05. Then, sample S44 is predicted according to reference sample R08.

[0066] In some cases, the values ​​of multiple reference samples may be combined, such as by interpolation, in order to compute the reference sample; especially if the directions do not divide evenly by 45 degrees.

[0067] As video coding techniques continue to develop, the number of possible directions is also increasing. For example, in H.264 (2003), nine different directions were available for intra prediction. This increased to 33 in H.265 (2013), and at the time of this disclosure JEM / VVC / BMS can support up to 65 directions. Experimental studies have been conducted to help identify the most suitable intra prediction directions, and certain techniques in entropy coding are used to encode those most suitable directions with a small number of bits, accepting some bit penalty for the direction. In addition, the direction itself can sometimes be predicted based on the neighboring directions used in the intra prediction of decoded neighboring blocks.

[0068] Figure 1B A schematic diagram (180) depicting 65 intra prediction directions according to the JEM is shown to illustrate that the number of prediction directions increases in various encoding techniques as they evolve over time.

[0069] The manner in which the bits representing the intra-prediction direction in the coded video bitstream are mapped to the prediction direction may vary from one video coding technique to another; and the mapping may vary, for example, from a simple direct mapping of prediction direction to intra-prediction mode to codeword to complex adaptive schemes involving most probable modes and similar techniques. However, in all cases, there may be certain intra-prediction directions that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, in a well-functioning video coding technique, those less likely directions will be represented by a larger number of bits than more likely directions.

[0070] Inter-picture prediction or inter-prediction can be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or a portion thereof (reference picture) can be used to predict a newly reconstructed picture or picture portion (e.g., block) after spatial shifting in the direction indicated by a motion vector (hereinafter referred to as MV (motion vector)). In some cases, the reference picture may be the same as the picture currently being reconstructed. MV may have two dimensions, X and Y, or three dimensions, the third dimension being an indication of the reference picture in use (similar to the time dimension).

[0071] In some video compression techniques, the current MV applicable to certain areas of sample data can be predicted from other MVs, such as the following other MVs, which are related to other areas of sample data that are spatially adjacent to the area being reconstructed and precede the current MV in decoding order. Doing so can significantly reduce the amount of data required to encode the MV by relying on removing redundancy in the relevant MV, thereby increasing compression efficiency. MV prediction can work effectively, for example, because when encoding an input video signal derived from a camera (called natural video), there is a statistical possibility that a larger area than the area to which a single MV applies moves in a similar direction, and therefore in some cases similar motion vectors derived from MVs of neighboring areas can be used for prediction. This results in the actual MV for a given area being similar or identical to the MV predicted from the surrounding MVs. Such an MV can, in turn, be represented after entropy coding with less than the number of bits used when directly encoding the neighboring MVs instead of predicting from the neighboring MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating predictors from several surrounding MVs.

[0072] Various MV prediction mechanisms are described in H.265 / HEVC (High Efficiency Video Coding, HEVC) (ITU-T (International Telecommunication Union-Telecommunication Standardization Sector, ITU-T) H.265 Recommendation, "High Efficiency Video Coding", December 2016). Among the multiple MV prediction mechanisms specified by H.265, the following is a technique referred to as "spatial merging" hereinafter.

[0073] Specifically, refer to Figure 2 , the current block (201) includes samples that have been found by the encoder during the motion search process, which are predictable from a previous block of the same size that has generated a spatial offset. Instead of encoding the MV directly, the MV associated with any of the five surrounding samples denoted A0, A1 and B0, B1, B2 (corresponding to 202 to 206, respectively) can be used, and the MV can be derived from metadata associated with one or more reference pictures, such as metadata from the most recent (in decoding order) reference picture. In H.265, MV prediction can use the predictor of the same reference picture that the neighboring blocks are also using.

[0074] AOMedia Video 1 (AV1) is an open video coding format designed for video transmission over the Internet. AV1 was developed as a successor to VP9 by building on the VP9 code base and incorporating additional technologies. The AV1 bitstream specification includes a reference video codec, such as H.265 or the High Efficiency Video Coding (HEVC) standard or Versatile Video Coding (VVC).

[0075] Figure 3 A simplified block diagram of a communication system (300) according to an embodiment of the present disclosure is shown. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). Figure 3In an embodiment, a first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, the terminal device (310) can encode video data (e.g., a video picture stream captured by the terminal device (310)) for transmission to another terminal device (320) via a network (350). The encoded video data is transmitted in the form of one or more encoded video bitstreams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to restore the video picture, and display the video picture based on the restored video data. The unidirectional data transmission can be implemented in a media service application, etc.

[0076] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which can be implemented, for example, during a video conferencing application. For the bidirectional transmission of data, each of the terminal devices (330) and the terminal device (340) can encode video data (e.g., a video picture stream captured by the terminal device) for transmission to the other terminal device (330) and the terminal device (340) via the network (350). Each of the terminal devices (330) and the terminal device (340) can also receive the encoded video data transmitted by the other terminal device (330) and the terminal device (340), and can decode the encoded video data to restore the video picture, and can display the video picture on an accessible display device based on the restored video data.

[0077] exist Figure 3 In the example of , terminal devices (310), (320), (330) and (340) can be implemented as servers, personal computers and smart phones, but the applicability of the basic principles of the present disclosure may not be limited to this. Implementations of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing devices, etc. Network (350) represents any number or type of networks that transmit encoded video data between terminal devices (310), (320), (330) and (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) can exchange data in circuit switching channels, packet switching channels and / or other types of channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purposes of this discussion, unless otherwise explicitly explained in this article, the architecture and topology of the network (350) may be irrelevant to the operation of the present disclosure.

[0078] As an example of an application for the disclosed subject matter, Figure 4The placement of the video encoder and video decoder in a video streaming environment is shown. The disclosed subject matter can be equally applicable to other video applications, including, for example, video conferencing, digital TV, broadcasting, gaming, virtual reality, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0079] The video streaming system may include a video capture subsystem (413), which may include a video source (401), such as a digital camera, for creating an uncompressed video picture or image stream (402). In an example, the video picture stream (402) includes samples recorded by the digital camera of the video source 401. The video picture stream (402) is depicted as a thick line to emphasize the high amount of data when compared to the encoded video data (404) (or encoded video bitstream), which may be processed by an electronic device (420) coupled to the video source (401) including a video encoder (403). The video encoder (403) may include hardware, software, or a combination thereof to implement or embody aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is depicted as a thin line to emphasize the lower data volume when compared to the uncompressed video picture stream (402), and the encoded video data (404) can be stored on the streaming server (405) for future use or directly to a downstream video device (not shown). One or more streaming client subsystems, such as Figure 4 The client subsystems (406) and (408) in the video streaming server (405) can access the streaming server (405) to read the copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and creates an output video picture stream (411) that is uncompressed and can be presented on a display (412) (e.g., a display screen) or another rendering device (not depicted). The video decoder 410 can be configured to perform some or all of the various functions described in the present disclosure. In some streaming systems, the encoded video data (404), (407) and (409) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T Recommendation H.265. In the example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed subject matter can be used in the context of VVC and other video coding standards.

[0080] Note that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0081] Figure 5 A block diagram of a video decoder (510) according to any embodiment of the present disclosure below is shown. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used instead of Figure 4 A video decoder (410) is shown in an example.

[0082] The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510). In the same embodiment or another embodiment, one encoded video sequence can be decoded at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Each video sequence can be associated with multiple video frames or images. The encoded video sequence can be received from a channel (501), which can be a hardware / software link to a storage device storing encoded video data or a streaming source that sends encoded video data. The receiver (531) can receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams that can be forwarded to their respective processing circuit systems (not depicted). The receiver (531) can separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) can be set between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) can be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be external to the video decoder (510) and separate from the video decoder (510) (not depicted). In still other applications, there may be a buffer memory (not depicted) external to the video decoder (510) to, for example, combat network jitter, and additional buffer memory (515) may be present within the video decoder (510) to, for example, handle playback timing. When the receiver (531) is receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may not be required, or the buffer memory (515) may be small. For use over a best effort packet network such as the Internet, a buffer memory (515) of sufficient size may be required, and the size of the buffer memory (515) may be relatively large. Such a buffer memory may be implemented to have an adaptive size, and may be implemented at least in part in an operating system or similar element (not depicted) external to the video decoder (510).

[0083] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (510) and may include information for controlling a presentation device such as a display (512) (e.g., a display screen), which may or may not be a component of the electronic device (530) but may be coupled to the electronic device (530), such as Figure 5. The control information for the rendering device may be in the form of a supplementary enhancement information (SEI (Supplemental Enhancement Information, SEI) message) or a video usability information (VUI) parameter set fragment (not depicted). The parser (520) may parse / entropy decode the encoded video sequence received by the parser (520). The entropy encoding of the encoded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a subgroup parameter set for at least one subgroup of the pixel subgroups in the video decoder from the encoded video sequence based on at least one parameter corresponding to the subgroup. The subgroup may include a group of pictures (Group of Pictures, GOP), a picture, a tile, a slice, a macroblock, a coding unit (Coding Unit, CU), a block, a transform unit (Transform Unit, TU), a prediction unit (Prediction Unit, PU), etc. The parser (520) may also extract information such as transform coefficients (eg, Fourier transform coefficients), quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0084] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0085] Depending on the type of coded video picture or part thereof (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of the symbol (521) may involve a number of different processing or functional units. The units involved and the manner in which they are involved may be controlled by sub-group control information parsed from the coded video sequence by the parser (520). For clarity, the flow of such sub-group control information between the parser (520) and the following multiple processing or functional units is not depicted.

[0086] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a number of functional units as described below. In a practical implementation operating under commercial constraints, many of these functional units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed subject matter, the conceptual subdivision into functional units is adopted in the following disclosure.

[0087] The first unit may include a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) may receive quantized transform coefficients as symbols (521) and control information from the parser (520), including information indicating which inverse transform to use, block size, quantization factors / parameters, quantization scaling matrix, etc. The sealer / inverse transform unit (551) may output a block including sample values ​​that may be input into an aggregator (555).

[0088] In some cases, the output samples of the scaler / inverse transform (551) may be from an intra-coded block, i.e., a block that does not use predictive information from a previously reconstructed picture, but instead uses predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may use surrounding block information that has already been reconstructed and stored in a current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator (555) may add the prediction information that has been generated by the intra-prediction unit (552) to the output sample information as provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0089] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-frame coded and possibly motion compensated block. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to obtain samples for inter-frame picture prediction. After motion compensation of the obtained samples according to the symbol (521) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (551) by the aggregator (555) (the output of unit 551 may be referred to as residual samples or residual signal) to generate output sample information. The address in the reference picture memory (557) at which the motion compensated prediction unit (553) obtains the predicted samples may be controlled by a motion vector, which may be used by the motion compensated prediction unit (553) in the form of a symbol (521), which may have, for example, an X component, a Y component (shift) and a reference picture component (time). Motion compensation may also include interpolation of sample values ​​such as obtained from a reference picture memory (557) when using sub-sample accurate motion vectors, and may also be associated with motion vector prediction mechanisms, etc.

[0090] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and obtained by the loop filter unit (556) as symbols (521) from the parser (520), but may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of the coded picture or coded video sequence, as well as to previously reconstructed and loop filtered sample values. Several types of loop filters may be included as part of the loop filter unit 556 in various orders, as will be described in further detail below.

[0091] The output of the loop filter unit (556) may be a sample stream that may be output to a rendering device (512) and stored in a reference picture memory (557) for use in future inter-picture prediction.

[0092] Certain coded pictures, once fully reconstructed, can be used as reference pictures for future inter-picture prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by a parser (520)), the current picture buffer (558) can become part of the reference picture memory (557) and a new current picture buffer can be reallocated before starting to reconstruct a subsequent coded picture.

[0093] The video decoder (510) may perform decoding operations according to a predetermined video compression technique employed in a standard such as ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the encoded video sequence follows both the syntax of the video compression technique or standard and the profile recorded in the video compression technique or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technique or standard as the only tools available for use under the profile. To conform to the standard, the complexity of the encoded video sequence is within the limits defined by the level of the video compression technique or standard. In some cases, the level may limit the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (measured in, for example, megasamples per second), the maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.

[0094] In some example embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0095] Figure 6 A block diagram of a video encoder (603) according to an example implementation of the present disclosure is shown. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may also include a transmitter (640) (e.g., transmission circuitry). The video encoder (603) may be used instead of Figure 4 The video encoder (403) in the example.

[0096] The video encoder (603) can capture video images to be encoded by the video encoder (603) from a video source (601) (in Figure 6 In one example, the video source (601) is not a part of the electronic device (620) and receives the video samples. In another example, the video source (601) can be implemented as a part of the electronic device (620).

[0097] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (603), which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ... any color space (e.g., BT.601 YCrCB, RGB, XYZ, ...) and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) may be a storage device capable of storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures or images that are given motion when viewed sequentially. The pictures themselves may be organized as a spatial pixel array, where each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0098] According to some example embodiments, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed constitutes a function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units as described below. For clarity, the coupling is not depicted. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other suitable functions that relate to the video encoder (603) optimized for a specific system design.

[0099] In some example embodiments, the video encoder (603) may be configured to operate in an encoding loop. As an extremely simplified description, in an example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder would create sample data, although the embedded decoder 633 processes the encoded video stream of the source encoder 630 without entropy coding (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video bitstream in entropy coding is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream results in a bit-exact result that is independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-exact between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the creation of drift when synchronization cannot be maintained, e.g. due to channel errors) is also used to improve encoding quality.

[0100] The operation of the "local" decoder (633) may be the same as the operation of a "remote" decoder, such as the video decoder (510), described above in conjunction with Figure 5 The video decoder (510) is described in detail. However, reference is also briefly made to Figure 5, since the symbols are available and encoding of the symbols into an encoded video sequence by the entropy encoder (645) and decoding of the symbols by the parser (520) can be lossless, the entropy decoding portion of the video decoder (510) including the buffer memory (515) and the parser (520) may not be fully implemented in the local decoder (633) in the encoder.

[0101] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter sometimes focuses on the decoder operation, which is related to the decoding part of the encoder. Therefore, the description of the encoder technology can be simplified because the encoder technology is opposite to the decoder technology described comprehensively. A more detailed description of the encoder is provided only in some areas or aspects below.

[0102] In some example implementations, during operation, the source encoder (630) may perform motion compensated predictive coding, which predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as "reference pictures". In this manner, the encoding engine (632) encodes the differences (or residuals) in color channels between pixel blocks of the input picture and pixel blocks of a reference picture, which may be selected as a prediction reference for the input picture. The term "residual" and its adjective form "residual" may be used interchangeably.

[0103] The local video decoder (633) can decode the encoded video data of the picture that can be designated as the reference picture based on the symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the video decoder ( Figure 6 When the encoded video data is decoded at a local (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors therein. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture, and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the far-end (remote) video decoder.

[0104] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may be used as suitable prediction references for the new picture. The predictor (635) may operate on a pixel block by pixel block basis on a sample block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (634).

[0105] The controller (650) may manage encoding operations of the source encoder (630), including, for example, setting of parameters and sub-group parameters for encoding video data.

[0106] The outputs of all the above mentioned functional units may be subjected to entropy coding in an entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0107] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which can be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0108] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain coded picture type to each coded picture, which may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned one of the following picture types:

[0109] Intra-pictures (I-pictures) are pictures that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh (IDR).

[0110] Those skilled in the art are aware of those variations of I pictures and their respective applications and features.

[0111] A predictive picture (P picture) may be a picture that can be encoded and decoded using inter prediction or intra prediction to predict sample values ​​of each block using at most one motion vector and a reference index.

[0112] Bi-directional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using inter-prediction or intra-prediction that uses up to two motion vectors and reference indices to predict sample values ​​for each block. Similarly, multi-predictive pictures can use more than two reference pictures and associated metadata for reconstructing a single block.

[0113] The source picture may typically be spatially subdivided into a plurality of coding blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and encoded block by block. The blocks may be predictively encoded with reference to other (already encoded) blocks determined by the coding allocation applied to the corresponding picture of the block. For example, blocks of an I picture may be non-predictively encoded, or blocks of an I picture may be predictively encoded (spatial prediction or intra-frame prediction) with reference to already encoded blocks of the same picture. Pixel blocks of a P picture may be predictively encoded via spatial prediction or via temporal prediction with reference to a previously encoded reference picture. Blocks of a B picture may be predictively encoded via spatial prediction or via temporal prediction with reference to one or two previously encoded reference pictures. For other purposes, the source picture or intermediate processed pictures may be subdivided into other types of blocks. The division of coding blocks and other types of blocks may or may not follow the same approach, as will be further described in detail below.

[0114] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In the operation of the video encoder (603), the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard being used.

[0115] In some example embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0116] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often simplified to intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits temporal or other correlations between pictures. For example, a particular picture being encoded / decoded, referred to as the current picture, can be divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that was previously encoded in the video and is still buffered, the block in the current picture can be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case of using multiple reference pictures, the motion vector may have a third dimension that identifies the reference picture.

[0117] In some example embodiments, bidirectional prediction techniques may be used for inter-picture prediction. According to such bidirectional prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture that are both before the current picture in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block may be jointly predicted by a combination of the first reference block and the second reference block.

[0118] In addition, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0119] According to some example embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, a picture in a video picture sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU may include three parallel coding tree blocks (CTBs): a luminance CTB and two chrominance CTBs. Each CTU may be recursively partitioned into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU may be partitioned into a 64×64 pixel CU, or four 32×32 pixel CUs. Each of one or more of the 32×32 blocks may also be partitioned into four 16×16 pixel CUs. In some example embodiments, each CU may be analyzed during encoding to determine the prediction type of the CU among various prediction types, such as an inter-prediction type or an intra-prediction type. Depending on temporal and / or spatial predictability, the CU can be partitioned into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in the encoding (encoding / decoding) is performed in units of prediction blocks. The partitioning of the CU into PUs (or PBs of different color channels) can be performed in various spatial modes. The luma or chroma PB can, for example, contain a matrix of values ​​(e.g., luma values) of samples of, for example, 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0120] Figure 7 A diagram of a video encoder (703) according to another example implementation of the present disclosure is shown. The video encoder (703) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures, and encode the processed block into an encoded picture that is part of an encoded video sequence. The example video encoder (703) may be used instead of Figure 4 The video encoder (403) in the example.

[0121] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) then uses, for example, rate-distortion optimization (RDO) to determine whether to use intra mode, inter mode, or bidirectional prediction mode to best encode the processing block. In the event that it is determined that the processing block is to be encoded in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded picture; and in the event that it is determined that the processing block is to be encoded in inter mode or bidirectional prediction mode, the video encoder (703) may use inter prediction or bidirectional prediction techniques to encode the processing block into an encoded picture, respectively. In some example embodiments, a merge mode may be used as an inter-picture prediction sub-mode, wherein a motion vector is derived from one or more motion vector prediction values ​​without the aid of an encoded motion vector component external to the predictor. In some other example embodiments, there may be a motion vector component applicable to the object block. Therefore, the video encoder (703) may include Figure 7 Components not explicitly shown, such as a mode decision module for determining a prediction mode for a processing block.

[0122] exist Figure 7 In the example of FIG. 7 , the video encoder ( 703 ) includes Figure 7 The example arrangement shown in FIG. 1 includes an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0123] The inter-frame encoder (730) is configured to: receive samples of a current block (e.g., a processing block); compare the block to one or more reference blocks in a reference picture (e.g., a block in a previous picture and a block in a subsequent picture in display order); generate inter-frame prediction information (e.g., motion vectors, merge mode information, description of redundant information according to an inter-frame coding technique); and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some examples, the reference picture is based on the encoded video information using an embedded Figure 6 The decoding unit 633 (shown as Figure 7 The decoded reference picture is decoded by the residual decoder 728, as described in further detail below.

[0124] The intra encoder (722) is configured to: receive samples of a current block (e.g., a processing block); compare the block with an already encoded block in the same picture; and generate quantization coefficients after transformation; and in some cases also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). The intra encoder (722) can calculate an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same picture.

[0125] The general controller (721) can be configured to determine general control data and control other components of the video encoder (703) based on the general control data. In an example, the general controller (721) determines a prediction mode of a block and provides a control signal to a switch (726) based on the prediction mode. For example, when the prediction mode is an intra-frame mode, the general controller (721) controls the switch (726) to select an intra-frame mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select intra-frame prediction information and include the intra-frame prediction information in the bitstream; and when the prediction mode of the block is an inter-frame mode, the general controller (721) controls the switch (726) to select an inter-frame prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select inter-frame prediction information and include the inter-frame prediction information in the bitstream.

[0126] The residual calculator (723) may be configured to calculate the difference (residual data) between the received block and the prediction result for the block selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) may be configured to encode the residual data to generate a transform coefficient. For example, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain to generate the transform coefficient. Then, the transform coefficient is subjected to quantization to obtain a quantized transform coefficient. In various example embodiments, the video encoder (703) further includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) may generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) may generate a decoded block based on the decoded residual data and the intra prediction information. The decoded blocks are appropriately processed to generate decoded pictures, and these decoded pictures may be buffered in a memory circuit (not shown) and used as reference pictures.

[0127] The entropy encoder (725) may be configured to format the bitstream to include the encoded blocks and perform entropy encoding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other appropriate information in the bitstream. When the block is encoded in the merge sub-mode of the inter-frame mode or the bidirectional prediction mode, there is no residual information.

[0128] Figure 8 A diagram of an example video decoder (810) according to another embodiment of the present disclosure is shown. The video decoder (810) is configured to receive an encoded picture as part of an encoded video sequence and decode the encoded picture to generate a reconstructed picture. In an example, the video decoder (810) may be used instead of Figure 4 A video decoder (410) is shown in an example.

[0129] exist Figure 8 In the example of FIG. 8 , the video decoder ( 810 ) includes Figure 8 An entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together are shown in the example arrangement of.

[0130] The entropy decoder (871) can be configured to reconstruct certain symbols from the encoded picture, which represent syntax elements constituting the encoded picture. Such symbols may include, for example, a mode for encoding a block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, merge sub-mode or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can identify certain samples or metadata used by the intra-frame decoder (872) or the inter-frame decoder (880) for prediction, respectively, residual information in the form of, for example, quantized transform coefficients, etc. In an example, when the prediction mode is an inter-frame mode or a bidirectional prediction mode, the inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is an intra-frame prediction type, the intra-frame prediction information is provided to the intra-frame decoder (872). The residual information can be subjected to inverse quantization and provided to the residual decoder (873).

[0131] The inter-frame decoder (880) may be configured to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0132] The intra decoder (872) may be configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0133] The residual decoder (873) can be configured to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) can also utilize certain control information (to include quantizer parameters (Quantizer Parameter, QP)), which can be provided by the entropy decoder (871) (since this may only be a small amount of control information, the data path is not depicted).

[0134] The reconstruction module (874) can be configured to combine the residual output by the residual decoder (873) with the prediction result (in this case, it can be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which forms part of the reconstructed picture as part of the reconstructed video. Note that other suitable operations such as deblocking operations can also be performed to improve visual quality.

[0135] Note that the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using any suitable technology. In some example embodiments, the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) may be implemented using one or more processors executing software instructions.

[0136] (603) and (603) and video decoders (410), (510) and (810).

[0137] Turning to the block partitioning for encoding and decoding, the general partitioning can start from the basic block and can follow a predefined set of rules, a specific pattern, a partitioning tree, or any partitioning structure or scheme. The partitioning can be hierarchical and recursive. After the basic block is segmented or divided according to any process in the example partitioning process or other processes described below or a combination of the example partitioning process and other processes, a final set of partitions or coding blocks can be obtained. Each of these partitions can be in one of the various partitioning levels in the partitioning hierarchy and can have various shapes. Each of the partitions can be referred to as a coding block (CB). For the various example partitioning implementations further described below, each resulting CB can have any allowed size and partitioning level. Such partitions are called coding blocks because they can form the following units, for which some basic encoding / decoding decisions can be made and encoding / decoding parameters can be optimized, determined, and signaled in the encoded video bitstream. The highest or deepest level in the final partition represents the depth of the coding block partitioning structure of the tree. The coding block can be a luminance coding block or a chrominance coding block. The CB tree structure of each color can be referred to as a coding block tree (CBT).

[0138] The coding blocks of all color channels may be collectively referred to as coding units (CUs). The hierarchical structure of all color channels may be collectively referred to as coding tree units (CTUs). The division modes or structures of various color channels in a CTU may be the same or different.

[0139] In some implementations, the partitioning tree schemes or structures for the luma channel and the chroma channel may not necessarily be the same. In other words, the luma channel and the chroma channel may have their own coding tree structures or patterns. In addition, whether the luma channel and the chroma channel use the same or different coding partitioning tree structures and the actual coding partitioning tree structures to be used may depend on whether the slice being encoded is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma channel and the luma channel may have separate coding partitioning tree structures or coding partitioning tree structure patterns, while for a P slice or a B slice, the luma channel and the chroma channel may share the same coding partitioning tree scheme. When a separate coding partitioning tree structure or pattern is applied, the luma channel is divided into CBs by one coding partitioning tree structure, and the chroma channel may be divided into chroma CBs by another coding partitioning tree structure.

[0140] In some example implementations, a predetermined partitioning pattern may be applied to a basic block. Fig. 9As shown, the example 4-way partitioning tree may start from a first predefined level (e.g., 64×64 block level or other size as a basic block size), and the basic blocks may be hierarchically partitioned down to a predefined lowest level (e.g., 4×4 level). For example, the basic blocks may be subjected to four predefined partitioning options or modes indicated by 902, 904, 906, and 908, where the partition designated as R is allowed for recursive partitioning because Fig. 9 The same partitioning options shown in can be repeated at lower scales until the lowest level (e.g., 4×4 level). Fig. 9 Additional restrictions apply to the partitioning scheme. Fig. 9 In the implementation of , rectangular partitions (e.g. 1:2 / 2:1) can be allowed, but they cannot be recursive, while square partitions can be recursive. If necessary, follow Fig. 9 The recursive partitioning generates a final set of coding blocks. The coding tree depth can be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth of the root node or root block (e.g., 64×64 block) can be set to 0, and the root block is divided according to Fig. 9 After being further split once, the coding tree depth increases by 1. For the above scheme, the maximum or deepest level of the minimum partition from 64×64 basic block to 4×4 will be 4 (starting from level 0). Such a partitioning scheme can be applied to one or more of the color channels. Each color channel can follow Fig. 9 Alternatively, two or more of the color channels may share a common Fig. 9 The same hierarchical pattern tree (e.g., the same partitioning pattern or option in the predefined pattern can be selected for two or more color channels at each hierarchical level).

[0141] Fig.10 Another example predefined partitioning pattern that allows recursive partitioning to form a partitioning tree is shown. Fig.10 As shown, an example 10-way partitioning structure or pattern may be predefined. A root block may start at a predefined level (eg, starting from a basic block at a 128×128 level or a 64×64 level). Fig.10 Example partitioning structures include various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitionings. Fig.10The partition type with 3 sub-partitions indicated as 1002, 1004, 1006 and 1008 in the second row of FIG. 1004 may be referred to as a "T-type" partition. The "T-type" partitions 1002, 1004, 1006 and 1008 may be referred to as a left T-type, a top T-type, a right T-type and a bottom T-type. In some example implementations, further subdivision is not allowed. Fig.10 The coding tree depth may be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth of the root node or root block (e.g., 128×128 block) may be set to 0, and the coding tree depth of the root block may be set to Fig.10 After being further split once, the coding tree depth increases by 1. In some implementations, only full square partitions in 1010 may be allowed to follow Fig.10 In other words, for the square partitions within the T-patterns 1002, 1004, 1006, and 1008, recursive partitioning may not be allowed. If necessary, follow Fig.10 The partitioning process in the recursive case of generates a final set of coding blocks. Such a scheme can be applied to one or more color channels. In some implementations, more flexibility can be added to the use of partitions below the 8×8 level. For example, 2×2 chroma inter-frame prediction can be used in some cases.

[0142] In some other example implementations of coding block partitioning, a quadtree structure may be used to partition a basic block or intermediate block into quadtree partitions. Such quadtree partitioning may be applied hierarchically and recursively to any square-shaped partition. Whether a basic block or intermediate block or partition is further quadtree partitioned may be adapted to various local characteristics of the basic block or intermediate block / partition. The quadtree partitioning at the picture boundary may be further adapted. For example, an implicit quadtree partitioning may be performed at the picture boundary so that the block will remain quadtree partitioned until the size fits the picture boundary.

[0143] In some other example implementations, hierarchical binary tree partitioning from basic blocks can be used. For such a scheme, a basic block or an intermediate level block can be divided into two partitions. The binary tree partitioning can be horizontal or vertical. For example, horizontal binary tree partitioning can divide a basic block or an intermediate block into equal right and left partitions. Similarly, vertical binary tree partitioning can divide a basic block or an intermediate block into equal upper and lower partitions. Such binary tree partitioning can be hierarchical and recursive. A decision can be made at each of the basic blocks or intermediate blocks whether the binary tree partitioning scheme should continue, and if the scheme continues further, a decision is made whether horizontal binary tree partitioning or vertical binary tree partitioning should be used. In some implementations, further partitioning can stop at a predefined minimum partition size (in one or two dimensions). Alternatively, once a predefined partitioning level or depth from a basic block is reached, further partitioning can be stopped. In some implementations, the aspect ratio of the partitioning can be limited. For example, the aspect ratio of the partitioning can be no less than 1:4 (or greater than 4:1). In this way, a vertical stripe partition having a vertical to horizontal aspect ratio of 4:1 can be further partitioned by only a vertical binary tree into upper and lower partitions each having a vertical to horizontal aspect ratio of 2:1.

[0144] In yet other examples, a ternary tree partitioning scheme may be used to partition a basic block or any intermediate block, such as Fig.13 The ternary tree pattern can be Fig.13 1302 is implemented vertically, or as shown in Fig.13 The 1304 level is achieved. Fig.13 The example split ratios (vertically or horizontally) in are shown as 1:2:1, but other ratios may also be predefined. In some implementations, two or more different ratios may be predefined. Such a ternary tree partitioning scheme may be used to complement a quadtree or binary tree partitioning structure, because such a ternary tree partitioning is able to capture objects located at the center of a block in one continuous partition, whereas quadtrees and binary trees always partition along the center of a block and therefore partition objects into separate partitions. In some implementations, the width and height of the partitions of the example ternary tree are always powers of 2 to avoid additional transformations.

[0145] The above partitioning schemes can be combined in any way at different partitioning levels. As an example, the above quadtree and binary tree partitioning schemes can be combined to partition a basic block into a quadtree-binary-tree (QTBT) structure. In such a scheme, a basic block or an intermediate block / partition can be a quadtree partition or a binary tree partition, subject to a set of predefined conditions in the specified case. Fig.14 A specific example is shown in FIG. Fig.14In the example of , the basic block is first quadtree-partitioned into four partitions, as shown in 1402, 1404, 1406, and 1408. Thereafter, each of the resulting partitions is quadtree-partitioned into four additional partitions (e.g., 1408), or is binary-partitioned at the next level (horizontally or vertically) into two additional partitions (e.g., 1402 or 1406, e.g., both are symmetric), or is not partitioned (e.g., 1404). For square-shaped partitions, binary or quadtree partitioning can be recursively allowed, as shown in the overall example partitioning pattern of 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree partitioning and dashed lines represent binary tree partitioning. A flag can be used for each binary tree partition node (non-leaf binary tree partitioning) to indicate whether the binary tree partitioning is horizontal or vertical. For example, as shown in 1420, consistent with the partition structure of 1410, the flag "0" may indicate horizontal binary tree partitioning, and the flag "1" may indicate vertical binary tree partitioning. For quadtree partitioning, there is no need to indicate the partition type, because quadtree partitioning always partitions a block or partition horizontally and vertically to produce 4 sub-blocks / partitions of equal size. In some implementations, the flag "1" may indicate horizontal binary tree partitioning, and the flag "0" may indicate vertical binary tree partitioning.

[0146] In some example implementations of QTBT, the quadtree and binary tree partitioning rule sets may be represented by the following predefined parameters and corresponding functions associated therewith:

[0147] –CTU size: the root node size of the quadtree (the size of the basic block)

[0148] –MinQTSize: Minimum allowed quadtree leaf node size

[0149] –MaxBTSize: Maximum allowed binary tree root node size

[0150] –MaxBTDepth: Maximum allowed binary tree depth

[0151] –MinBTSize: The minimum allowed binary tree leaf node size

[0152] In some example implementations of the QTBT partitioning structure, the CTU size can be set to 128×128 luma samples and two corresponding blocks of 64×64 chroma samples (when example chroma subsampling is considered and used), MinQTSize can be set to 16×16, MaxBTSize can be set to 64×64, MinBTSize (for both width and height) can be set to 4×4, and MaxBTDepth can be set to 4. Quadtree partitioning can first be applied to the CTU to generate quadtree leaf nodes. The size of the quadtree leaf node can be from its minimum allowed size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., the CTU size). If the node is 128×128, it will not be split first by the binary tree because the size exceeds MaxBTSize (i.e., 64×64). Otherwise, nodes that do not exceed MaxBTSize can be partitioned by the binary tree. Fig.14 In the example, the basic block is 128×128. According to a predefined set of rules, the basic block can only be a quadtree partition. The partition depth of the basic block is 0. Each of the four resulting partitions is 64×64 - not exceeding MaxBTSize, and can be further quadtree or binary tree partitioned at level 1. The process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), further partitions can be ignored. When the binary tree node has a width equal to MinBTSize (i.e., 4), further horizontal partitions can be ignored. Similarly, when the binary tree node has a height equal to MinBTSize, further vertical partitions are not considered.

[0153] In some example implementations, the above QTBT scheme can be configured to support the flexibility of having the same QTBT structure or separate QTBT structures for luma and chroma. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU can share the same QTBT structure. However, for I slices, the luma CTB can be divided into CBs by the QTBT structure, and the chroma CTB can be divided into chroma CBs by another QTBT structure. This means that a CU can be used to refer to different color channels in an I slice, for example, an I slice can consist of coding blocks of a luma component or coding blocks of two chroma components, and a CU in a P slice or a B slice can consist of coding blocks of all three color components.

[0154] In some other implementations, the QTBT scheme can be supplemented with the above ternary tree scheme. Such an implementation can be called a multi-type-tree (MTT) structure. For example, in addition to the binary tree partitioning of the nodes, Fig.13In some implementations, only square nodes can be subjected to ternary tree partitioning. An additional flag can be used to indicate whether the ternary tree partition is horizontal or vertical.

[0155] The design of two-level or multi-level trees such as the QTBT implementation and the QTBT implementation supplemented by ternary tree partitioning may be motivated primarily by complexity reduction. In theory, the complexity of traversing the tree is T D , where T is the number of partition types and D is the depth of the tree. A trade-off can be made by using multiple types (T) while reducing the depth (D).

[0156] In some implementations, the CB may be further divided. For example, for the purpose of intra-frame prediction or inter-frame prediction during the encoding and decoding process, the CB may also be divided into a plurality of prediction blocks (Prediction Block, PB). In other words, the CB may be further divided into different sub-partitions, in which separate prediction decisions / configurations may be made. In parallel, for the purpose of depicting the level of transformation or inverse transformation of the video data, the CB may also be divided into a plurality of transform blocks (Transform Block, TB). The schemes for dividing the CB into PB and TB may be the same or different. For example, each partitioning scheme may be performed using its own process based on various characteristics of the video data, for example. In some example implementations, the PB partitioning scheme and the TB partitioning scheme may be independent. In some other example implementations, the PB partitioning scheme and the TB partitioning scheme may be related to the boundary. In some implementations, for example, the TB may be divided after the PB is divided, and in particular, after each PB is determined after the coding block is divided, the PB may then be further divided into one or more TBs. For example, in some implementations, the PB may be divided into one, two, four, or other number of TBs.

[0157] In some implementations, in order to divide the basic block into coding blocks and further into prediction blocks and / or transform blocks, the luma channel and the chroma channel may be processed differently. For example, in some implementations, the division of the coding block into prediction blocks and / or transform blocks may be allowed for the luma channel, while such division of the coding block into prediction blocks and / or transform blocks may not be allowed for the chroma channel. In such an implementation, the transformation and / or prediction of the luma block can therefore be performed only at the coding block level. For another example, the minimum transform block size for the luma channel and the chroma channel may be different, for example, the coding block for the luma channel may be allowed to be divided into smaller transform blocks and / or prediction blocks than the chroma channel. For another example, the maximum depth of dividing the coding block into transform blocks and / or prediction blocks may be different between the luma channel and the chroma channel, for example, the coding block for the luma channel may be allowed to be divided into deeper transform blocks and / or prediction blocks than the chroma channel. For a specific example, the luma coding block may be partitioned into transform blocks of multiple sizes that may be represented by recursive partitioning down to up to 2 levels, and transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4 and transform block sizes from 4×4 to 64×64 may be allowed. However, for chroma blocks, only the largest possible transform block specified for the luma block may be allowed.

[0158] In some example implementations for partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partition may depend on whether the PB is intra-coded or inter-coded.

[0159] The partitioning of a coding block (or prediction block) into transform blocks may be implemented in various example schemes, including but not limited to recursively or non-recursively performing quadtree partitioning and predefined pattern partitioning, and additionally taking into account transform blocks at the boundaries of coding blocks or prediction blocks. In general, the generated transform blocks may be at different partitioning levels, may not have the same size, and may not need to be square in shape (e.g., they may be rectangular with some allowed size and aspect ratio). The following is about Fig.15 , Fig.16 and Fig.17 Additional examples are described in further detail.

[0160] However, in some other implementations, the CB obtained via any of the above partitioning schemes can be used as a basic or minimum coding block for prediction and / or transformation. In other words, no further segmentation is performed for inter-frame prediction / intra-frame prediction purposes and / or for transformation purposes. For example, the CB obtained from the above QTBT scheme can be used directly as a unit for performing prediction. Specifically, such a QTBT structure removes the concept of multiple partitioning types, that is, the separation of CU, PU and TU is removed, and supports more flexibility in the CU / CB partition shape as described above. In such a QTBT block structure, the CU / CB can have a square or rectangular shape. The leaf nodes of such a QTBT are used as units for prediction and transformation processing without any further partitioning. This means that CU, PU and TU have the same block size in such an example QTBT coding block structure.

[0161] The above various CB partitioning schemes and CB to PB and / or TB can be combined in any way

[0162] The following specific implementations are provided as non-limiting examples.

[0163] A specific example implementation of coding block and transform block segmentation is described below. In such an example implementation, recursive quadtree segmentation or the above-mentioned predefined segmentation mode (e.g., Fig. 9 and Fig.10 The basic block is divided into coding blocks (such as those in the CU). At each level, whether the further quadtree segmentation of a specific partition should continue can be determined by the local video data characteristics. The resulting CB can be at different quadtree segmentation levels and can have different sizes. A decision can be made at the CB level (or CU level, for all three color channels) about whether to use inter-frame picture (time) prediction or intra-frame picture (spatial) prediction to encode the picture area. Each CB can be further divided into one, two, four or other number of PBs according to a predefined PB segmentation type. Within a PB, the same prediction process can be applied, and relevant information can be sent to the decoder based on the PB. After obtaining the residual block by applying the prediction process based on the PB segmentation type, the CB can be divided into TBs according to another quadtree structure similar to the coding tree for the CB. In this specific implementation, the CB or TB can be, but not necessarily limited to, a square shape. In addition, in this specific example, for inter-frame prediction, the PB can be a square or rectangular shape, while for intra-frame prediction, the PB can be only a square. The coding block can be divided into, for example, four square-shaped TBs. Each TB can be further recursively partitioned (using quadtree partitioning) into smaller TBs, which are called Residual Quadtrees (RQTs).

[0164] Another example implementation of dividing a basic block into CB, PB, and / or TB is further described below. For example, instead of using Fig. 9 or Fig.10 Instead of using the multiple partition unit types shown, a quadtree with nested multi-type trees that use binary and ternary tree partitioning segment structures (e.g., QTBT as described above or QTBT with ternary tree partitioning) may be used. The separation of CB, PB, and TB (i.e., partitioning CB into PB and / or TB, and partitioning PB into TB) may be abandoned except when a CB with a size that is too large for the maximum transform length is required, wherein such a CB may require further partitioning. The example partitioning scheme may be designed to support more flexibility in the shape of the CB partitions, so that both prediction and transformation may be performed at the CB level without further partitioning. In such a coding tree structure, the CB may have a square or rectangular shape. Specifically, the coding tree blocks (CTBs) may be first partitioned by a quadtree structure. Then, the quadtree leaf nodes may be further partitioned by a nested multi-type tree structure. In Fig.11 An example of a nested multi-type tree structure using binary or ternary tree partitioning is shown in FIG. Fig.11 The example multi-type tree structure includes four types of segmentation, which are called vertical binary tree segmentation (SPLIT_BT_VER) (1102), horizontal binary tree segmentation (SPLIT_BT_HOR) (1104), vertical ternary tree segmentation (SPLIT_TT_VER) (1106) and horizontal ternary tree segmentation (SPLIT_TT_HOR) (1108). The CB then corresponds to the leaves of the multi-type tree. In this example implementation, unless the CB is too large for the maximum transform length, the segment is used for both prediction and transform processing without any further partitioning. This means that in most cases, CB, PB and TB have the same block size in a quadtree with a nested multi-type tree coding block structure. The exception occurs when the maximum supported transform length is less than the width or height of the color component of the CB. In some implementations, in addition to binary tree or ternary tree segmentation, Fig.11 The nested pattern can further include quadtree partitioning.

[0165] exist Fig.12 A specific example of a quadtree with a nested multi-type tree coding block structure with block partitioning (including quadtree, binary tree and ternary tree partitioning options) for one basic block is shown in FIG. In more detail, Fig.12 The basic block 1200 is shown to be partitioned by a quadtree into four square partitions 1202, 1204, 1206 and 1208. Further use is made for each of the quadtree partitions Fig.11 The multi-type tree structure and quadtree are used to make further segmentation decisions. Fig.12 In the example of FIG. 1204, partition 1204 is not further partitioned. Partitions 1202 and 1208 are both partitioned by another quadtree. For partition 1202, the upper left partition, upper right partition, lower left partition, and lower right partition of the second level quadtree partition are partitioned by the third level quadtree, Fig.11 Horizontal binary tree partition 1104, no partition and Fig.11 The horizontal ternary tree partition 1108. The partition 1208 adopts another quadtree partition, and the upper left partition, upper right partition, lower left partition and lower right partition of the second level quadtree partition adopt Fig.11 The third level of vertical ternary tree segmentation 1106, no segmentation, no segmentation and Fig.11 The horizontal binary tree partition 1104. The two sub-partitions in the sub-partitions of the third level upper left partition 1208 are respectively based on Fig.11 The horizontal binary tree partition 1104 and the horizontal ternary tree partition 1108 are further partitioned. The partition 1206 is adopted according to Fig.11 The second level partitioning pattern of the vertical binary tree partitioning 1102 is divided into two partitions, and the two partitions are divided into two partitions according to Fig.11 The horizontal ternary tree partition 1108 and the vertical binary tree partition 1102 are further partitioned into a third level. Fig.11 The horizontal binary tree partition 1104 further applies the fourth level partition to one of the partitions in the third level partition.

[0166] For the specific example above, the maximum luma transform size may be 64×64, and the maximum supported chroma transform size may be different from luma, for example, 32×32. Fig.12 The example CB in the example is usually not further divided into smaller PBs and / or TBs. When the width or height of the luminance coding block or the chrominance coding block is greater than the maximum transform width or height, the luminance coding block or the chrominance coding block may also be automatically split in the horizontal and / or vertical directions to meet the transform size limit in that direction.

[0167] In the above specific example for dividing the basic block into CBs, and as described above, the coding tree scheme can support the ability of luma and chroma to have separate block tree structures. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU can share the same coding tree structure. For example, for I slices, luma and chroma can have separate coding block tree structures. When a separate block tree structure is applied, the luma CTB can be divided into luma CBs by one coding tree structure, and the chroma CTB can be divided into chroma CBs by another coding tree structure. This means that a CU in an I slice can include coding blocks for a luma component or coding blocks for two chroma components, while a CU in a P slice or a B slice always includes coding blocks for all three color components unless the video is monochrome.

[0168] When the coding block is further divided into multiple transform blocks, the transform blocks therein can be arranged in the bitstream in various orders or scanning modes. The following is further described in detail an example implementation for dividing a coding block or prediction block into transform blocks and the encoding order of the transform blocks. In some example implementations, as described above, the transform partitioning can support transform blocks of various shapes, such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, where the transform block size ranges from, for example, 4×4 to 64×64. In some implementations, if the coding block is less than or equal to 64×64, the transform block partitioning can be applied only to the luminance component, so that for the chrominance block, the transform block size is the same as the coding block size. In addition, if the coding block width or height is greater than 64, both the luminance coding block and the chrominance coding block can be implicitly split into multiple min(W,64)×min(H,64) and min(W,32)×min(H,32) transform blocks, respectively.

[0169] In some example implementations of transform block partitioning, for both intra-coded blocks and inter-coded blocks, the coding blocks may be further partitioned into multiple transform blocks with a partition depth up to a predefined number of levels (e.g., 2 levels). The transform block partition depth and size may be related. For some example implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown below in Table 1.

[0170] Table 1: Transform partition size settings

[0171]

[0172] Based on the example mapping of Table 1, for a 1:1 square block, the next level transform partitioning can create four 1:1 square sub-transform blocks. The transform partitioning can stop at, for example, 4×4. Therefore, the transform size of 4×4 at the current depth corresponds to the same size of 4×4 at the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform partitioning will create two 1:1 square sub-transform blocks, and for a 1:4 / 4:1 non-square block, the next level transform partitioning will create two 1:2 / 2:1 sub-transform blocks.

[0173] In some example implementations, for the luma component of an intra-coded block, additional restrictions may be applied to the transform block partitioning. For example, for each level of transform partitioning, all sub-transform blocks may be restricted to have equal size. For example, for a 32×16 coding block, a level 1 transform partitioning creates two 16×16 sub-transform blocks and a level 2 transform partitioning creates eight 8×8 sub-transform blocks. In other words, the second level partitioning must be applied to all first level sub-blocks to keep the transform units of equal size. Fig.15 1 shows an example of transform block partitioning of an intra-coded square block according to Table 1, and the coding order indicated by arrows. Specifically, 1502 shows a square coding block. A first level partitioning into 4 equal-sized transform blocks according to Table 1 is shown in 1504, where the coding order is indicated by arrows. A second level partitioning into 16 equal-sized transform blocks according to Table 1 is shown in 1506, where the coding order is indicated by arrows.

[0174] In some example implementations, for the luma component of an inter-coded block, the above restrictions for intra-coded blocks may not apply. For example, after the first level of transform partitioning, any of the sub-transform blocks may be further partitioned independently at a higher level. Thus, the resulting transform blocks may or may not have the same size. Fig.16 An example of dividing an inter-frame coding block into transform blocks and the coding order of the transform blocks is shown in FIG. Fig.16 In the example of , the inter-frame coding block 1602 is divided into transform blocks at two levels according to Table 1. At the first level, the inter-frame coding block is divided into four transform blocks of equal size. Then, only one of the four transform blocks (rather than all transform blocks) is further divided into four sub-transform blocks, resulting in a total of 7 transform blocks with two different sizes, as shown in 1604. The example coding order of these 7 transform blocks is as follows: Fig.16 As shown by the arrow in 1604.

[0175] In some example implementations, for chroma components, some additional restrictions may be applied to transform blocks. For example, for chroma components, the transform block size may be as large as the coding block size, but not smaller than a predefined size, such as 8×8.

[0176] In some other example implementations, for coding blocks whose width (W) or height (H) is greater than 64, both luma coding blocks and chroma coding blocks may be implicitly split into multiple min(W, 64)×min(H, 64) and min(W, 32)×min(H, 32) transform units, respectively. Here, in the present disclosure, "min(a, b)" may return the smaller value between a and b.

[0177] Fig.17 Another alternative example scheme for dividing a coding block or prediction block into transform blocks is also shown. Fig.17 As shown, instead of using recursive transform partitioning, a predefined set of partition types can be applied to the coding block according to the transform type of the coding block. Fig.17 In the specific example shown, one of six example partition types may be applied to partition a coding block into various numbers of transform blocks. Such a generated transform block partitioning scheme may be applied to a coding block or a prediction block.

[0178] In more detail, Fig.17 The partitioning scheme of provides up to 6 example partitioning types for any given transform type (a transform type refers to, for example, the type of a primary transform such as ADST). In this scheme, a transform partitioning type may be assigned to each coding block or prediction block based on, for example, a rate-distortion cost. In an example, the transform partitioning type assigned to a coding block or prediction block may be determined based on the transform type of the coding block or prediction block. A particular transform partitioning type may correspond to a transform block partitioning size and mode, as indicated by Fig.17 The six transform partition types shown are shown. The correspondence between various transform types and various transform partition types can be predefined. The following shows an example in which the uppercase label indicates the transform partition type that can be assigned to the coding block or prediction block based on the rate-distortion cost:

[0179] ●PARTITION_NONE: Allocate a transform size equal to the block size.

[0180] ●PARTITION_SPLIT: Allocate a transform size of 1 / 2 the block size in width and 1 / 2 the block size in height.

[0181] ●PARTITION_HORZ: Allocate a transform size with the same width as the block size and 1 / 2 the height of the block size.

[0182] ●PARTITION_VERT: Allocate a transform size with a width of 1 / 2 the block size and the same height as the block size.

[0183] ●PARTITION_HORZ4: Allocates a transform size with the same width as the block size and 1 / 4 of the height of the block size.

[0184] ●PARTITION_VERT4: Allocates a transform size with a width of 1 / 4 of the block size and the same height as the block size.

[0185] In the above example, if Fig.17 The transform partition type shown contains uniform transform sizes for all the partitioned transform blocks. This is only an example and not a limitation. In some other implementations, a mixed transform block size may be used for the partitioned transform blocks under a specific partition type (or mode).

[0186] Then, the PB (or CB, also referred to as PB when not further divided into prediction blocks) obtained from any of the above partitioning schemes can become a separate block encoded via intra-frame prediction or inter-frame prediction. For inter-frame prediction for the current PB, a residual between the current block and the prediction block can be generated, encoded, and included in the encoded bitstream.

[0187] Inter prediction may be implemented, for example, in a single reference mode or a composite reference mode. In some implementations, a skip flag may first be included in the bitstream of the current block (or at a higher level) to indicate whether the current block is inter-coded and is not skipped. If the current block is inter-coded, another flag may be further included in the bitstream as a signal indicating whether a single reference mode or a composite reference mode is used for prediction of the current block. For a single reference mode, one reference block may be used to generate a prediction block for the current block. For a composite reference mode, two or more reference blocks may be used to generate a prediction block, for example, by weighted averaging. A composite reference mode may be referred to as more than one reference mode, two reference modes, or multiple reference modes. One or more reference blocks may be identified using one or more reference frame indexes and additionally using one or more motion vectors indicating a displacement between the reference block and the current block in position, for example, in horizontal pixels and vertical pixels. For example, an inter-frame prediction block for a current block may be generated from a single reference block identified as a prediction block in a single reference mode by a motion vector in a reference frame, while for a composite reference mode, the prediction block may be generated by a weighted average of two reference blocks in two reference frames indicated by two reference frame indices and two corresponding motion vectors. The motion vector may be encoded and included in a bitstream in various ways.

[0188] In some implementations, the encoding or decoding system may maintain a decoded picture buffer (Decoded PictureBuffer, DPB). Some images / pictures may be kept in the DPB to wait for display (in the decoding system), and some images / pictures in the DPB may be used as reference frames to enable inter-frame prediction (in the decoding system or the encoding system). In some implementations, the reference frames in the DPB may be marked as short-term references or long-term references for the current image being encoded or decoded. For example, a short-term reference frame may include a frame used to perform inter-frame prediction on a block in the current frame or a block in a predefined number (e.g., 2) of subsequent video frames closest to the current frame in decoding order. A long-term reference frame may include a frame in the DPB that may be used to predict image blocks in a frame that is more than a predefined number of frames away from the current frame in decoding order. Information about such tags for short-term reference frames and long-term reference frames may be referred to as a reference picture set (Reference Picture Set, RPS) and may be added to the header of each frame in the encoded bitstream. Each frame in the encoded video stream can be identified by a Picture Order Counter (POC), which is numbered in an absolute manner according to the playback order or relative to a group of pictures, for example starting from an I frame.

[0189] In some example implementations, one or more reference picture lists containing identifications of short-term reference frames and long-term reference frames for inter-frame prediction may be formed based on information in the RPS. For example, a single picture reference list may be formed for unidirectional inter-frame prediction, denoted as L0 reference (or reference list 0), while two picture reference lists may be formed for bidirectional inter-frame prediction, denoted as L0 (or reference list 0) and L1 (or reference list 1) for each of the two prediction directions. The reference frames included in the L0 list and the L1 list may be ordered in various predefined ways. The lengths of the L0 list and the L1 list may be signaled in the video bitstream. Unidirectional inter-frame prediction may be performed in single reference mode, or in composite reference mode when multiple references used to generate a prediction block by weighted averaging in composite prediction mode are on the same side of the block to be predicted. Bidirectional inter-frame prediction may be performed only in composite mode because bidirectional inter-frame prediction involves at least two reference blocks.

[0190] Adaptive Loop Filter

[0191] In Versatile Video Coding (VVC), an adaptive loop filter (ALF) with block-based filter adaptation is applied. For the luminance component, one of many filters is selected for each 4×4 block based on the direction and activity of the local gradient. In one example, there may be 25 filters to choose from.

[0192] Fig.18 An example adaptive loop filter (ALF) shape is shown. Specifically, Fig.18 Two diamond filter shapes are shown: A 7x7 diamond shape is applied to the luma component and a 5x5 diamond shape is applied to the chroma components.

[0193] The block classification can be calculated for different examples as follows. For the luma component, each 4×4 block is classified into one of 25 classes. The classification index C is based on its directionality D and activity The quantitative value of is obtained as follows:

[0194]

[0195] In order to calculate D and First, use the 1-D Laplacian to calculate the gradient in the horizontal, vertical, and two diagonal directions:

[0196]

[0197]

[0198]

[0199]

[0200] Where indices i and j refer to the coordinates of the top left sample within the 4×4 block, and R(i, j) indicates the reconstructed sample at coordinates (i, j).

[0201] To reduce the complexity of block classification, a subsampled 1-D Laplacian calculation can be applied. Figures 19a to 19d As shown in , the same subsampling positions can be used for gradient calculations in all directions. Fig.19a The sub-sampling positions in the Laplacian calculation of the vertical gradient are shown. Fig.19b The sub-sampling positions in the Laplacian calculation of the horizontal gradient are shown. Fig.19c The sub-sampling positions in the Laplacian calculation of the diagonal gradient are shown. Fig.19d The sub-sampling positions in the Laplacian calculation of another diagonal gradient are shown.

[0202] Then set the maximum and minimum values ​​of D for the horizontal and vertical gradients to:

[0203]

[0204] Set the maximum and minimum values ​​of the gradients in the two diagonal directions to:

[0205]

[0206] In order to derive a value for the directivity D, these values ​​are compared with one another and with the aid of two threshold values ​​t1 and t2 .

[0207] Step 1. If and If both are true, D is set to 0.

[0208] Step 2. If Then continue from step 3; otherwise, continue from step 4.

[0209] Step 3. If Then set D to 2; otherwise set D to 1.

[0210] Step 4. If Then set D to 4; otherwise set D to 3.

[0211] The activity value A is calculated as:

[0212]

[0213] A is further quantized to the range of 0 to 4, inclusive, and the quantized value is represented as For the chrominance components in a picture, no classification method is applied. In other words, a single set of ALF coefficients can be applied for each chrominance component.

[0214] There may be geometric transformations of the filter coefficients and the clipping values. Before filtering each 4×4 luma block, geometric transformations such as rotation or diagonal flipping and vertical flipping may be applied to the filter coefficients f(k, l) and the corresponding filter clipping values ​​c(k, l) according to the gradient values ​​calculated for the block. This may be equivalent to applying these transformations to samples in the filter support region. This may make different blocks more uniform by aligning the directionality of different blocks to which the ALF is applied. The three geometric transformations may include diagonal, vertical flipping, and rotation:

[0215] Diagonal: f D (k, l) = f(l, k), c D (k, l) = c(l, k) (9)

[0216] Flip vertically: f V(k, l) = f(k, Kl-1), c V (k, l) = c(k, Kl-1) (10)

[0217] Rotation: f R (k, l) = f(Kl-1, k), c R (k, l) = c(Kl-1, k) (11)

[0218] Where K is the size of the filter, and 0≤k, l≤K-1 are the coefficient coordinates, so that the position (0,0) is at the upper left corner, and the position (K-1,K-1) is at the lower right corner. According to the gradient value calculated for the block, the transform is applied to the filter coefficients f(k,l) and the clipping value c(k,l). The relationship between the transform and the four gradients in the four directions is summarized in Table 2 below.

[0219] Gradient Value Transform <![CDATA[g d2 <g d1 And g h <g v ]]> No change <![CDATA[g d2 <g d1 And g v <g h ]]> diagonal <![CDATA[g d1 <g d2 And g h <g v ]]> Flip Vertically <![CDATA[g d1 <g d2 And g v <g h ]]> Rotation

[0220] Table 2 - Mapping of gradients computed for a block to transformations

[0221] In VVC, the ALF filter parameters are signaled in an adaptation parameter set (APS). In one APS, multiple groups of luminance filter coefficients and clipping value indices can be used. For example, there can be 25 groups of luminance filters. In addition, multiple groups of chrominance filter coefficients and clipping value indices can be signaled. In one example, there can be up to 8 groups of chrominance filter coefficients and clipping value indices that can be signaled. In order to reduce bit overhead, filter coefficients of different classifications of luminance components can be merged. In the slice header, the index of the APS used for the current slice can be signaled. The signaling of ALF can be based on the coding tree unit (CTU).

[0222] The clip value index decoded from the APS allows the use of a table of clip values ​​for luminance and chrominance to determine the clip value. These clip values ​​may depend on the internal bit depth. More precisely, the table of clip values ​​can be obtained by the following formula:

[0223] AlfClip = {round(2 B-α*n )for n∈[0..N-1]} (12)

[0224] In the case where B is equal to the internal bit depth, α is a predefined constant value equal to 2.35, and N is equal to 4, which is the number of allowed clipping values ​​in VVC in one embodiment. Table 3 shows the output of equation (12).

[0225]

[0226] Table 3 - Specification of Alfclip depending on bitDepth and clipIdx

[0227] In one slice header example, up to 7 APS indices can be signaled to specify the luma filter bank for the current slice. The filtering process can be further controlled at the coding tree block (CTB) level. A flag can be signaled to indicate whether the ALF is applied to the luma CTB. In one example, the luma CTB can select a filter bank from 16 fixed filter banks and a filter bank from the APS. The filter bank index is signaled for the luma CTB to indicate which filter bank is applied. The 16 fixed filter banks can be predefined and hard-coded in both the encoder and the decoder. For the chroma component, the APS index is signaled in the slice header to indicate the chroma filter bank for the current slice. At the CTB level, if there is more than one chroma filter bank in the APS, the filter index is signaled for each chroma CTB. The filter coefficients can be quantized with a norm equal to 128. In order to limit the multiplication complexity, bitstream consistency is applied so that the coefficient values ​​at non-central positions can be in the range of -27 to 27-1, including the end values. The center position coefficient is not signaled in the bitstream and is considered equal to 128.

[0228] In the VVC example, the syntax and semantics of the clip index and clip value may be defined as follows: alf_luma_clip_idx[sfIdx][j] specifies the clip index of the clip value to be used before multiplication with the jth coefficient of the signaled luma filter indicated by sfIdx. A requirement for bitstream conformance may be that the value of alf_luma_clip_idx[sfIdx][j] with sfIdx=0..alf_luma_num_filters_signalled_minus1 and j=0..11 shall be in the range of 0 to 3, inclusive. From the bitDepth set equal to BitDepthY and the clipIdx set equal to alf_luma_clip_idx[alf_luma_coeff_delta_idx[filtIdx]][j], the luma filter clipping value AlfclipL[adaptation_parameter_set_id] with the elements AlfClipL[adaptation_parameter_set_id][filtIdx][j] is derived as specified in Table 3, where filtIdx = 0..NumAlfFilters-1, j = 0..11. alf_chroma_clip_idx[altIdx][j] specifies the clip index of the clipping value used before multiplying the j-th coefficient of the alternative chroma filter with index altIdx. The bitstream conformance requirement is that the value of alf_chroma_clip_idx[altIdx][j] with altIdx=0..alf_chroma_num_alt_filters_minus 1, j=0..5 shall be in the range of 0 to 3, inclusive. From the bitDepth set equal to BitDepthC and the clipIdx set equal to alf_chroma_clip_idx[altIdx][j], the chroma filter clipping values ​​AlfClipC[adaptation_parameter_set_id][altIdx][j] with elements AlfClipC[adaptation_parameter_set_id][altIdx][j] are derived as specified in Table 3, where altIdx=0..alf_chroma_num_alt_filters_minus 1, j=0...5.

[0229] The filtering process can be performed in the following example. At the decoder side, when ALF is enabled for CTB, each sample R(i, j) within the CU is filtered to obtain the sample value R′(i, j):

[0230] R′(i, j)=R(i, j)+((∑ k≠0 ∑ l≠0 f(k,l)×K(R(i+k,j+l)-R(i,j),c(k,l))+64)>>7) (13)

[0231] Where f(k, l) represents the decoded filter coefficients, K(x, y) is the clipping function, and c(k, l) represents the decoded clipping parameters. The variables k and l are in and =min(y, max(-y, x)), which corresponds to the function Clip3(-y, y, x). By incorporating this clipping function, this loop filtering method, as first proposed in JVET-N0242, becomes a nonlinear process, known as nonlinear ALF. The selected clipping value is encoded in the "alf_data" syntax element using a Golomb coding scheme corresponding to the index of the clipping value in Table 3. This coding scheme can be the same as the coding scheme for the filter index.

[0232] There may be a virtual boundary filtering process for line buffer reduction. To reduce the line buffer requirements of the ALF, a modified block classification and filtering may be applied to samples near horizontal CTU boundaries. Thus, Fig. 20 As shown, the virtual boundary can be defined as a line by shifting the horizontal CTU boundary by "N" samples. Fig. 20 An example of modified block classification at a virtual boundary is shown. In this example, N is equal to 4 for the luma component and N is equal to 2 for the chroma components.

[0233] Apply the modified block classification to Fig. 20 . For the one-dimensional (1D) Laplacian gradient calculation of the 4×4 block above the virtual boundary, only the samples above the virtual boundary are used. Similarly, for the one-dimensional Laplacian gradient calculation of the 4×4 block below the virtual boundary, only the samples below the virtual boundary are used. By considering the reduced number of samples used in the one-dimensional Laplacian gradient calculation, the quantization of the activity value A is scaled.

[0234] Fig.21 An example of modified adaptive loop filtering of the luma component at a virtual boundary is shown. For the filtering process, a symmetric padding operation at the virtual boundary can be used for both the luma component and the chroma component. Fig.21 As shown, when the sample being filtered is below the virtual boundary, the adjacent samples above the virtual boundary can be filled in. The corresponding samples on the other side can also be filled in symmetrically.

[0235] Fig. 22 An example of a picture quadtree partition aligned with a maximum coding unit (LCU) is shown. In order to enhance coding efficiency, an adaptive loop filter based on a coding unit synchronized picture quadtree can be used. The luminance picture can be partitioned into several multi-level quadtree partitions, and each partition boundary is aligned with the boundary of the maximum coding unit (LCU). Each partition has its own filtering process and can therefore be referred to as a filter unit (FU). The secondary (pass) encoding process may include the following. At the first time, the quadtree partition mode and the optimal filter for each FU are determined. During the decision process, the filtering distortion is estimated by FFDE. The reconstructed picture is filtered according to the determined quadtree partition mode of all FUs and the selected filter. In the second pass, CU synchronized ALF on / off control is performed. According to the ALF on / off result, the first filtered picture is partially restored by reconstructing the picture.

[0236] A top-down partitioning strategy can be adopted to divide the picture into multiple levels of quadtree partitions by using a rate-distortion criterion. Each partition can be called a filter unit. The partitioning process aligns the quadtree partitions with the LCU boundaries. The coding order of FU follows a zigzag scanning order. For example, in Fig. 22 In , the picture is divided into 10 FUs, and the encoding order is FU0, FU1, FU2, FU3, FU4, FU5, FU6, FU7, FU8 and FU9.

[0237] Fig.23 An example of a quadtree partition flag encoded in a z-order is shown. To indicate the picture quadtree partition mode, the partition flag may be encoded and transmitted in a z-order. Fig.23 Shown with Fig. 22Corresponding quadtree segmentation mode. The filter for each FU is selected from two filter groups based on the rate-distortion criterion. The first group has a 1 / 2 symmetrical square shape filter and a diamond shape filter newly derived for the current FU. The second group comes from the delay filter buffer; the delay filter buffer stores the filters previously derived for the FU of the previous picture. For the current FU, the filter with the minimum rate-distortion cost among the two groups is selected. Similarly, if the current FU is not the minimum FU and can be further divided into 4 sub-FUs, the rate-distortion costs of these 4 sub-FUs are calculated. By recursively comparing the rate-distortion costs in the segmentation case and the non-segmentation case, the quadtree segmentation mode of the picture can be determined. In an example, the maximum quadtree segmentation level can be 2, which means that the maximum number of FUs is 16. During the quadtree segmentation decision, the correlation values ​​for deriving the Wiener coefficients of the 16 FUs (minimum FUs) at the bottom quadtree level can be reused. The remaining FUs can derive their Wiener filters based on the correlation of the 16 FUs at the bottom quadtree level. Therefore, there can be only one frame buffer access for deriving the filter coefficients of all FUs. After deciding the quadtree splitting mode, CU synchronous ALF on / off control is performed to further reduce filtering distortion. By comparing filtering distortion and non-filtering distortion, the leaf CU can explicitly switch ALF on / off in its local area. Coding efficiency can be further improved by redesigning the filter coefficients based on the ALF on / off results. However, the redesign process may require additional frame buffer accesses. In the modified encoder design, there may be no redesign process after the CU synchronous ALF on / off decision to minimize the number of frame buffer accesses.

[0238] Cross-Component Adaptive Loop Filter (CC-ALF)

[0239] Fig.24 An example of a cross-component adaptive loop filter (CC-ALF) placement is shown. The CC-ALF can utilize luma sample values ​​to refine each chroma component. Fig.24 The placement of the CC-ALF relative to the other loop filters is shown.

[0240] Fig.25 An example of a diamond shape filter is shown. CC-ALF can be implemented by applying a filter from Fig.25The filter coefficients are sent in the APS, scaled by a factor of 210 in one example, and rounded for fixed point representation. The application of the filter is controlled by the variable block size and is signaled by the context coding flag received for each sample block. For each chroma component, the block size is received at the slice level along with the CC-ALF enable flag. In one example, the following block sizes (in chroma samples) are supported: 16×16, 32×32, 64×64.

[0241] Example syntax for CC-ALF may include:

[0242]

[0243] The semantics of CC-ALF related syntax may include:

[0244] alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] equal to 0 indicates that the cross-component Cb filter is not applied to the block of Cb color component samples at luma position (xCtb, yCtb).

[0245] alf_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] not equal to 0 indicates that the alf_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]th cross-component Cb filter is applied to the block of Cb color component samples at luma position (xCtb, yCtb).

[0246] alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] equal to 0 indicates that the cross-component Cr filter is not applied to the block of Cr color component samples at luma position (xCtb, yCtb).

[0247] alf_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] not equal to 0 indicates that the alf_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]th cross-component Cr filter is applied to the block of Cr color component samples at luma position (xCtb, yCtb).

[0248] Chroma sampling format

[0249] Fig.26 Example positions of chroma samples relative to luma samples are shown. Fig.26 The indicated relative position of the top left chroma sample is shown when chroma_format_idc is equal to 1 (4:2:0 chroma format) and chroma_sample_loc_type_top_field or chroma_sample_loc_type_bottom_field is equal to the value of the variable ChromaLocType. The area represented by the top left 4:2:0 chroma sample (depicted as a large square with a large red dot at its center) is shown relative to the area represented by the top left luma sample (depicted as a small square with a small dot at its center). The area represented by the adjacent luma sample is depicted as a small shaded gray square with a small shaded gray dot at its center.

[0250] Directional Enhancement Features

[0251] One goal of the in-loop constrained directional enhancement filter (CDEF) is to filter out coding artifacts while preserving image details. In HEVC, the Sample Adaptive Offset (SAO) algorithm can achieve similar goals by defining signal offsets for different categories of pixels. Unlike SAO, CDEF is a nonlinear spatial filter. The design of the filter is constrained to be easy to vectorize (i.e., implementable by SIMD operations), which is not the case for other nonlinear filters such as median filters and bilateral filters. The CDEF design stems from the following observations. The amount of ringing artifacts in the encoded image tends to be roughly proportional to the quantization step size. The amount of detail is a property of the input image, but the minimum detail retained in the quantized image also tends to be proportional to the quantization step size. For a given quantization step size, the amplitude of the ringing is typically smaller than the amplitude of the detail.

[0252] CDEF works by identifying the orientation of each block and then adaptively filtering along the identified orientation and, to a lesser extent, along the orientation rotated 45 degrees from the identified orientation. The filter strengths are explicitly signaled, which allows a high degree of control over blur. An efficient encoder search is designed for the filter strengths. CDEF is based on two previously proposed in-loop filters, and the combined filter is used in the emerging AV1 codec.

[0253] Fig. 27An example of a directional search is shown. The directional search operates only on reconstructed pixels after the deblocking filter. Since these pixels are available to the decoder, the direction does not need to be signaled. The search operates on 8×8 blocks, which are small enough to adequately handle non-straight edges, yet large enough to reliably estimate the direction when applied to a quantized image. Having a constant direction over an 8×8 region also makes vectorization of the filter easier. For each block, we determine the direction that best matches the pattern in the block by minimizing the sum of squared differences (SSD) between the quantized block and the nearest perfectly oriented block. A perfectly oriented block is a block where all pixels along a line in one direction have the same value. Fig. 27 is an example of a directional search on an 8×8 block.

[0254] There may be a non-linear low-pass directional filter. One reason to identify the direction is to align the filter taps along that direction to reduce ringing while preserving directional edges or patterns. However, directional filtering alone sometimes does not reduce ringing sufficiently. It may also be desirable to use filter taps for pixels that are not along the main direction. To reduce the risk of blurring, these additional taps are handled more appropriately. For this purpose, CDEF defines primary taps and secondary taps. The complete 2D CDEF filter can be expressed as:

[0255]

[0256] Where D is the damping parameter, S (p) and S (s) are the strengths of the primary and secondary taps, respectively, and round(●) rounds away from zero, w k are the filter weights, and f(d, S, D) is a constraint function that operates on the difference between the filtered pixel and each neighboring pixel. For small differences, f(d, S, D) = d, making the filter behave like a linear filter. When the difference is large, f(d, S, D) = 0, which effectively ignores the filter taps.

[0257] Loop recovery

[0258] A set of in-loop recovery schemes are proposed for post-deblocking of video coding to generally denoise and improve edge quality beyond conventional deblocking operations. The schemes are switchable within a frame in units of appropriately sized tiles. The specific schemes described are based on separable symmetric Wiener filters and dual self-guided filters with subspace projection. Because content statistics can vary significantly within a frame, these tools are integrated within a switchable framework where different tools can be triggered in different regions of the frame.

[0259] There may be a separable symmetric Wiener filter used as a restoration tool. Each pixel in the degraded frame can be reconstructed as a non-causal filtered version of the pixels within the w×w window around it, where w=2r+1 is an odd number for integer r. If the 2D filter taps are represented in a w2×1 element vector F in column vectorized form, direct LMMSE optimization results in the filter parameters being given by F=H-1M, where H=E[XXT] is the autocovariance of x, x is a column vectorized version of the w2 samples in the w×w window around the pixel, and M=E[YXT] is the cross-correlation of x with the scalar source sample y to be estimated. The encoder can estimate H and M based on the implementation in the deblocked frame and source, and send the resulting filter F to the decoder. However, this will not only incur a large bit rate cost when transmitting w2 taps, but non-separable filtering will make decoding extremely complex. Therefore, several additional constraints are imposed on the properties of F. First, F is constrained to be separable so that filtering can be implemented as separable horizontal and vertical w-tap convolutions. Second, each of the horizontal filter and the vertical filter is constrained to be symmetric. Third, it is assumed that the sum of both the horizontal filter coefficient and the vertical filter coefficient is 1.

[0260] There can be a dual self-guided filtering with subspace projection for image filtering, where the local linear model is:

[0261] y=Fx+G (15)

[0262] The local linear model is used to calculate the filtered output y from the unfiltered sample x, where F and G are determined based on the statistical information of the guidance image and the degraded image in the vicinity of the filtered pixel. If the guidance image is the same as the degraded image, the resulting so-called self-guided filtering has the effect of edge-preserving smoothing. The specific form of the proposed self-guided filtering depends on two parameters: the radius r and the noise parameter e, and is listed as follows:

[0263] 1. Get the mean μ and variance σ of the pixels in the (2r+1)×(2r+1) window around each pixel 2 This can be efficiently achieved by box filtering based on integral imaging.

[0264] 2. Calculation for each pixel: f = σ 2 / (σ 2 +e); g = (1-f) μ

[0265] 3. Calculate F and G for each pixel as the average of the f-values ​​and the average of the g-values ​​in a 3×3 window around the pixel for use.

[0266] The filtering can be controlled by r and e, where larger r means larger spatial variance, and larger e means larger range variance.

[0267] Fig.28 An example of subspace projection is shown. Fig.28 The principle of subspace projection is schematically shown in . Even if restorations X1, X2 are not close to source Y, appropriate multipliers {α, β} can bring them closer to source Y as long as they are slightly moved in the right direction.

[0268] Cross Component Sample Offset (CCSO)

[0269] The loop filtering method may include cross-component sample offset (CCSO) to reduce the distortion of the reconstructed samples. In CCSO, a nonlinear mapping is used to derive an output offset considering the processed input reconstructed samples of a first color component, and the output offset is added to the reconstructed samples of another color component during the filtering process of the proposed CCSO.

[0270] Fig.29 An example of a filter support region is shown. The input reconstructed samples are from the first color component located in the filter support region. Fig.29 As shown, the filter support region includes four reconstructed samples: p0, p1, p2, p3. The four input reconstructed samples are in a cross shape in the vertical and horizontal directions. The center sample in the first color component (denoted by c) and the sample to be filtered in the second color component are juxtaposed. When processing the input reconstructed samples, the following steps are applied:

[0271] ■Step 1: First calculate the difference values ​​between p0 to p3 and c, which are represented as m0, m1, m2 and m3.

[0272] ■Step 2: The difference values ​​m0 to m3 are further quantized, and the quantized values ​​are represented as d0, d1, d2, d3. Based on the following quantization process, the quantized values ​​can be -1, 0, 1:

[0273] a. If m<-N, d=-1; (16)

[0274] b. If -N<=m<=N, d=0; (17)

[0275] c. If m>N, d=1. (18)

[0276] Here, N is called the quantization step size, and example values ​​of N are 4, 8, 12, and 16.

[0277] Variables d0 to d3 can be used to identify a combination of nonlinear mappings. In this example, the CCSO has four filter taps d0 to d3, and each filter tap can have one of three quantized values, so there are 3^4=81 combinations in total. Table 4 (below) shows 81 example combinations, and the last column represents the output offset value of each combination. Example offset values ​​are integers, such as 0, 1, -1, 3, -3, 5, -5, -7.

[0278] combination d0 d1 d2 d3 Offset 0 -1 -1 -1 -1 s0 1 -1 -1 -1 0 s1 2 -1 -1 -1 1 s2 3 -1 -1 0 -1 s3 4 -1 -1 0 0 s4 5 -1 -1 0 1 s5 6 -1 -1 1 -1 s6 7 -1 -1 1 0 s7 8 -1 -1 1 1 s8 9 -1 0 -1 -1 s9 10 -1 0 -1 0 s10 11 -1 0 -1 1 s11 12 -1 0 0 -1 s12 13 -1 0 0 0 s13 14 -1 0 0 1 s14 15 -1 0 1 -1 s15 16 -1 0 1 0 s16 17 -1 0 1 1 s17 18 -1 1 -1 -1 s18 19 -1 1 -1 0 s19 20 -1 1 -1 1 s20 21 -1 1 0 -1 s21 22 -1 1 0 0 s22 23 -1 1 0 1 s23 24 -1 1 1 -1 s24 25 -1 1 1 0 s25 26 -1 1 1 1 s26 27 0 -1 -1 -1 s27 28 0 -1 -1 0 s28 29 0 -1 -1 1 s29 30 0 -1 0 -1 s30 31 0 -1 0 0 s31 32 0 -1 0 1 s32 33 0 -1 1 -1 s33 34 0 -1 1 0 s34 35 0 -1 1 1 s35 36 0 0 -1 -1 s36 37 0 0 -1 0 s37 38 0 0 -1 1 s38 39 0 0 0 -1 s39 40 0 0 0 0 s40 41 0 0 0 1 s41 42 0 0 1 -1 s42 43 0 0 1 0 s43 44 0 0 1 1 s44 45 0 1 -1 -1 s45 46 0 1 -1 0 s46 47 0 1 -1 1 s47 48 0 1 0 -1 s48 49 0 1 0 0 s49 50 0 1 0 1 s50 51 0 1 1 -1 s51 52 0 1 1 0 s52 53 0 1 1 1 s53 54 1 -1 -1 -1 s54 55 1 -1 -1 0 s55 56 1 -1 -1 1 s56 57 1 -1 0 -1 s57 58 1 -1 0 0 s58 59 1 -1 0 1 s59 60 1 -1 1 -1 s60 61 1 -1 1 0 s61 62 1 -1 1 1 s62 63 1 0 -1 -1 s63 64 1 0 -1 0 s64 65 1 0 -1 1 s65 66 1 0 0 -1 s66 67 1 0 0 0 s67 68 1 0 0 1 s68 69 1 0 1 -1 s69 70 1 0 1 0 s70 71 1 0 1 1 s71 72 1 1 -1 -1 s72 73 1 1 -1 0 s73 74 1 1 -1 1 s74 75 1 1 0 -1 s75 76 1 1 0 0 s76 77 1 1 0 1 s77 78 1 1 1 -1 s78 79 1 1 1 0 s79 80 1 1 1 1 s80

[0279] Table 4 - Sample combinations identified by d0 to d3

[0280] The final filtering process of CCSO is applied as follows:

[0281] f'=clip(f+s), (19)

[0282] Where f is the reconstructed sample to be filtered and s is the output offset value retrieved from Table 4, the filtered sample value f' is further clipped to a range associated with the bit depth.

[0283] Local Sample Offset (LSO) is another example offset implementation. In LSO, a similar filtering approach as in CCSO is applied, but the output offset is applied to the color components that are the same color components as the reconstructed samples were used as input to the filtering process.

[0284] In an alternative embodiment, the simplified CCSO design may be adopted into the reference software of AV2, ie, the AVM of CWG-B022.

[0285] Fig.30 An example loop filtering circuit is shown. CCSO is a loop filtering process performed in parallel with CDEF in the loop filtering circuit, that is, the input is the same as CDEF, and the output is applied to the samples filtered by CDEF, such as Fig.30 Note that CCSO can be applied only to chrominance color components.

[0286] Fig.31 An example input of a cross component sample offset (CCSO) is shown. The CCSO filter is applied to the chroma reconstructed samples, which are denoted as rc. The concatenated luma reconstructed samples of rc are denoted as rl. Fig.31 An example of a CCSO filter is shown in . In CCSO, a set of 3-tap filters is used. The input luminance reconstruction samples at three filter taps include the center rl and two adjacent samples p0 and p0.

[0287] Given p iand rl, where i = 0, 1, apply the following steps to process the input sample:

[0288] -First calculate p i The difference between rl and m is represented by i

[0289] -Use the following quantization process to convert the difference value m i Quantized to d i :

[0290] -When m is less than -Q CCSO When d i Set to -1

[0291] -When m in -Q CCSO With Q CCSO Included between -Q CCSO and Q CCSO When d i Set to 0

[0292] -When m is greater than Q CCSO When d i is set equal to 1.

[0293] In the above steps, Q CCSO It is called the quantization step size, Q CCSO It can be 8, 16, 32, 64.

[0294] After calculating d0 and d1, the offset value (denoted as s) is derived using the CCSO's look-up table (LUT). The CCSO's LUT is shown in Table 5. Each combination of d0 and d1 is used to identify the row in the LUT to retrieve the offset value. The offset value is an integer including 0, 1, -1, 3, -3, 5, -5, and -7.

[0295]

[0296]

[0297] Table 5: Lookup table (LUT) used in CCSO

[0298] Finally, the resulting offset of the CCSO is applied to the chrominance color components as follows:

[0299] rc'=clip(rc+s), (20)

[0300] Where rc is the reconstructed sample to be filtered by CCSO and s is the derived offset value retrieved from the LUT, the filtered sample value rc' is further clipped into the range specified by the bit depth.

[0301] Fig.32 An example filter shape in the cross component sample offset (CCSO) is shown. Fig.32 As shown, in CCSO, there are six optional filter shapes, which are denoted as f i , i=1, ..., 6. These six filter shapes are switchable at frame level, and the selection is signaled by the syntax ext_filter_support using a 3-bit fixed-length code.

[0302] Signaling of cross-component sample offset (CCSO) may be performed at frame level and block level.

[0303] At the frame level, the signal may include:

[0304] ●1-bit flag indicating whether CCSO is applied

[0305] ● A 3-bit syntax ext_filter_support indicating the selection of CCSO filter shape

[0306] ● A 2-bit index indicating the choice of quantization step size

[0307] ● 9 3-bit offset values ​​used in the LUT

[0308] At the 128x128 chroma block level, a flag is signaled to indicate whether the CCSO filter is enabled.

[0309] Sample Adaptive Offset (SAO)

[0310] In HEVC, sample adaptive offset (SAO) is applied to the reconstructed signal after the deblocking filter by using the offset value given in the slice header. For luma samples, the encoder decides whether the SAO filter is applied to the current slice. If SAO is enabled, the current picture allows recursive partitioning into four sub-regions, and each region can select one of the six SAO types shown in Table 6. SAO classifies reconstructed pixels into categories and reduces distortion by adding offsets to pixels of each category in the current region. Pixel classification in SAO types 1 to 4 is performed using edge attributes, and pixel classification in SAO types 5 to 6 is performed using pixel intensity.

[0311]

[0312] Table 6 - Specifications of SAO types

[0313] Band Offset (BO) classifies all pixels of a region into multiple bands, where each band contains pixels of the same intensity interval. The intensity range is equally divided into 32 intervals from zero to the maximum intensity value (e.g., 255 for 8-bit pixels), and each interval has an offset. Next, the 32 bands are divided into two groups. One group consists of the center 16 bands, and the other group consists of the remaining 16 bands. Only the offset in one group is sent. Regarding the pixel classification operation in BO, the top five bits of each pixel can be used directly as a band index.

[0314] Fig.33 Example pixel patterns are shown. Taking into account edge orientation information, edge offset (EdgeOffset, EO) uses four 1-dimensional (1-D) 3-pixel patterns for pixel classification, such as Fig.33 As shown. Each region of the picture can select a mode to classify pixels into multiple categories by comparing each pixel with its two neighboring pixels. This selection will be sent in the bitstream as side information. Table 7 shows the pixel classification rules of EO.

[0315]

[0316]

[0317] Table 7 - Pixel classification rules for edge offset (EO)

[0318] The SAO on the decoder side can operate independently of the LCU, so that the line buffer can be saved. To achieve this, when the 90-degree mode, 135-degree mode, and 45-degree classification mode are selected, the pixels of the top and bottom rows in each LCU may not be SAO processed. When the 0-degree mode, 135-degree mode, and 45-degree mode are selected, the pixels of the leftmost and rightmost columns in each LCU may not be SAO processed.

[0319] Table 8 below shows an example syntax that may need to be signaled for a CTU if the parameters are not merged from neighboring CTUs:

[0320]

[0321] Table 8 - Sample Adaptive Offset VLC Syntax

[0322] Cross-Component Sample Offset (CCSO) and Local Sample Offset (LSO) can use the value of the pixel to be filtered to select an offset value on a color component. However, further extending those inputs for offset selection may significantly increase the signaling overhead of CCSO and LSO, which may limit / degrade coding performance, especially for smaller resolution sequences.

[0323] As described, CCSO is defined as a filtering process that uses reconstructed samples of a first color component as input (e.g., Y or Cb or Cr), and the output is applied to a second color component that is a different color component from the first color component. Fig.29 An example filter shape for CCSO is shown in . LSO is a filtering process that uses reconstructed samples of a first color component as input (e.g., Y or Cb or Cr) and applies the output to the same first color component. Therefore, the difference between LSO and CCSO is the different input.

[0324] As described below and in Fig.34 As illustrated in , a generalized design of CCSO and LSO is presented by considering not only the difference values ​​between neighboring samples of a collocated (or current) sample as considered in CCSO and LSO but also the level value of the collocated (or current) sample itself.

[0325] Fig.34 A flow chart of a method according to an example implementation of the present disclosure is shown. In box 3402, encoded information of a reconstructed sample of a current component in a current picture of an encoded video bitstream is decoded. The encoded information indicates a sample offset filter to be applied to the reconstructed sample. In one example, the sample offset filter may include two types of offset values, namely a gradient offset (GO) and a band offset (BO). In box 3404, an offset type is selected for use with the sample offset filter. In box 3406, an output value of the sample offset filter is determined based on a first reconstructed sample and the selected offset type. In box 3408, a filtered sample value of the reconstructed sample of the current component is determined based on the output value of the sample offset filter and the reconstructed sample. Further implementations are described below.

[0326] The generalized sample offset (GSO) method may include two types of offset values ​​for CCSO and LSO, including gradient offset (GO) and band offset (BO). The selection of the offset type may be signaled or derived implicitly.

[0327] In one embodiment, the gradient offset may be an offset derived using a difference value between a collocated sample and an adjacent sample of different color components (in the case of CCSO) or a difference value between an adjacent sample and a current sample to be filtered (in the case of CCSO or LSO).

[0328] In one embodiment, the band offset can be an offset derived using the value of the juxtaposed samples of different color components or the current sample to be filtered. The band can be used to determine the offset value. In one example, the value of the juxtaposed samples of different color components or the current sample to be filtered can be represented as a variable v, and the BO value is derived using v>>s, where >> indicates a right shift operation and s is a predefined value specifying the interval of sample values ​​using the same band offset. In one example, the value of s can vary for different color components. In another example, the value of the juxtaposed samples of different color components or the current sample to be filtered is represented as a variable v, and the band index bi is derived using a predefined lookup table, where the input to the lookup table is v, and the output value is the band index bi, and the BO value is derived using the band index bi.

[0329] In one embodiment, when a combination of GO and BO is applied (e.g., used simultaneously), the offset is derived using: 1) the difference value between collocated samples of different color components and adjacent samples (in the case of CCSO) or the difference value between the current sample to be filtered and adjacent samples (in the case of CCSO or LSO); and 2) the value of collocated samples of different color components or the current sample to be filtered.

[0330] In one embodiment, the application of GO or BO is notified by signaling. The signaling can be applied in a high-level syntax. As a few examples, the signaling can include VPS, PPS, SPS, slice header, picture header, frame header, super block header, CTU header, tile header.

[0331] In another embodiment, whether GO or BO is signaled in a block level, the block level includes but is not limited to a coding unit (block) level, a prediction block level, a transform block level, or a filtering unit level. This example includes signaling at the block level to identify GO or BO.

[0332] In another embodiment, a flag is used to signal GO or BO. A flag is first signaled to indicate whether LSO and / or CCSO is applied to one or more color components, and then another flag is signaled to indicate whether GO or BO is applied. For example, a flag is first signaled to indicate whether LSO and / or CCSO is applied to one or more color components, and then another flag is signaled to indicate whether GO is applied in conjunction with BO, wherein BO is always applied regardless of whether GO is applied. In another example, a flag is first signaled to indicate whether LSO and / or CCSO is applied to one or more color components, and then another flag is signaled to indicate whether BO is applied in conjunction with GO, wherein GO is always applied regardless of whether BO is applied.

[0333] In some embodiments, a signal for determining whether to use GO or BO or a combination thereof may be derived. The signal may be derived implicitly using encoded information, including but not limited to reconstructed samples of the current color component and / or different color components, whether the current block is intra-coded or inter-coded, whether the current picture is a key (or intra-frame) picture, whether the current sample (or block) is encoded by a specific prediction mode (e.g., a specific intra-frame or inter-frame prediction mode, a transform selection mode, a quantization parameter).

[0334] As described, LSO may be similar to CCSO, but the output offset is applied to the color component, which is the same color component as the color component to which the reconstructed sample is used as input to the filtering process. GSO is a generalized design that considers two types of offset values, including gradient offset (GO) and band offset (BO). The gradient offset considers the difference value between adjacent samples of the juxtaposed or current sample, and the band offset (BO) considers the level value of the current sample itself.

[0335] In addition, the signaling aspects of CCSO, LSO and GSO can be classified into frame level and block level. At the frame level, the activation of CCSO, LSO and / or GSO can be signaled (for example, by using a 1-bit flag), and / or the offset value of the LUT can be signaled (for example, by using 3 bits each). At the block level, for M×N blocks, a flag is signaled to indicate the activation of CCSO, LSO and / or GSO for the current block. In various embodiments, the values ​​of M and N can each include, but are not limited to, 16, 32, 64, 128 or 256. As used herein, the term block refers to a region of a frame.

[0336] The adaptive application of GSO is described below.

[0337] As described, CCSO is defined as a filtering process that uses reconstructed samples of a first color component as input (e.g., Y or Cb or Cr) and applies the output to a second color component that is a different color component than the first color component. Fig.29 An example filter shape for CCSO is shown in . In addition, LSO is a filtering process that uses a reconstructed sample of a first color component as input (e.g., Y or Cb or Cr) and applies the output to the same first color component. GSO is a filtering process that can be regarded as a generalized design of CCSO and LSO, and utilizes the differential value between adjacent samples of the collocated (or current) sample as in CCSO and LSO, and also takes into account the level value of the current sample. Two types of offset values ​​are used in GSO, namely: a gradient offset (GO), which corresponds to the differential value between adjacent samples of the collocated or current sample; and a band offset (BO), which corresponds to the level value of the current sample itself.

[0338] Fig.35 A flow chart of a method 3500 according to an example implementation of the present disclosure is shown. The example method 3500 may be performed by a device configured to decode a video stream, such as the video decoder (510) or the video decoder (810) described above. Fig.35 At least some of the actions in the method may be performed by Figure 5 The loop filter (556) in the loop filter is performed.

[0339] At block 3502, an encoded video stream (e.g., a bitstream) is received. The encoded video stream includes a syntax element indicating whether a sample offset filtering process is applied to a frame in the encoded video stream at a block level. At block 3504, based on the syntax element being a predefined value, it is determined that at least one of GSO, LSO, or CCSO is enabled for the frame at the block level. The predefined value may be one of a plurality of predefined values. Each of the plurality of predefined values ​​may indicate whether a particular sample offset process (GSO, LSO, or CCSO) or a particular combination of two or more sample offset processes (two or more of GSO, LSO, CCSO) is enabled. Thus, in some embodiments, at block 3504, it is determined that at least GSO is enabled. Additionally or alternatively, it is determined that at least LSO is enabled. Additionally or alternatively, it is determined that at least CCSO is enabled. At block 3506, the encoded video stream may be decoded by applying at least one of GSO, LSO, or CCSO at the block level. For example, at least one of the GSO, LSO, or CCSO applied at block 3506 may correspond to or be based on the GSO, LSO, and / or CCSO determined to be enabled at block 3504. That is, if it is determined at block 3504 that GSO is enabled, then GSO may be applied at block 3506; if it is determined at block 3504 that LSO is enabled, then LSO may be applied at block 3506; and if it is determined at block 3504 that CCSO is enabled, then CCSO may be applied at block 3506. For at least some embodiments of the method 3500, the block sizes of frames in the encoded video stream are different for luma components and chroma components in the corresponding frames.

[0340] In some embodiments, applying at least one of GSO, LSO, or CCSO at block 3506 includes applying GSO where a first block size of the luma component is the same as or smaller than a second block size of the chroma component. For example, the granularity of applying GSO is finer for the luma component than for the chroma component. As an example, when the M×N of chroma is 128×128, the M×N for luma GSO is 128×128 or less. The values ​​of M and N may include, but are not limited to, 16, 32, 64, 128, or 256.

[0341] In some embodiments, the method 3500 may further include receiving a signal indicating a first block level flag for a luma component having a first block size when signaling a second block level flag for a chroma component having a second block size, the first block size being the same as or smaller than the second block size for the chroma component. For example, the granularity of signaling the block level flag is finer for the luma component than for the chroma component. As an example, when signaling a flag for a chroma component for a block size of 128×128, the block level flag is signaled for a block size smaller than 128×128.

[0342] In some embodiments, a block-level flag indicating whether one of CCSO, LSO, or GSO is enabled or disabled, for example, for application to a block may also indicate whether another of CCSO, LSO, or GSO is enabled or disabled for the block. For example, a block-level flag indicating that CCSO is enabled for the chrominance component of a block may also indicate that LSO is enabled for the luminance component of the block. Similarly, a block-level flag indicating that LSO is enabled for the luminance component of a block may also indicate that CCSO is enabled for the chrominance component of the block. As an example, a block-level flag indicating whether CCSO is enabled or applied for a block may depend on a block-level flag indicating whether LSO is enabled or applied for the block, and / or a block-level flag indicating whether LSO is applied for a block may depend on a block-level flag indicating whether CCSO is applied for a block. In addition or alternatively, a block-level flag indicating whether CCSO or LSO is applied for a block may be used as a context for signaling a block-level flag to indicate whether LSO or CCSO is applied to a block, respectively.

[0343] In some embodiments, the method 3500 may further include: receiving a signal indicating a granularity of at least one of a chroma component or a luma component for application of the GSO in a high-level syntax, wherein the high-level syntax includes: a video parameter set (VPS), a picture parameter set (PPS), a sequence parameter set (SPS), an adaptation parameter set (APS), a frame header, a slice header, a picture header, a tile header, a super-block header, or a coding tree unit (CTU) header. For example, the granularity of applying the GSO to the luma component and / or the chroma component may be signaled in a high-level syntax, the high-level syntax including but not limited to a VPS, a PPS, a SPS, an APS, a frame header, a slice header, a picture header, a tile header, a super-block header, and / or a CTU header.

[0344] In some embodiments, the syntax element is or includes a high-level syntax, GSO is enabled by the high-level syntax, and the method 3500 may also include: receiving a signal including additional high-level syntax to indicate the maximum number of bands for band offset. For example, when GSO is enabled by high-level syntax, additional high-level syntax is signaled to indicate the maximum number of bands for BO. For some of these embodiments, the high-level syntax includes but is not limited to VPS, PPS, SPS, APS, frame header, slice header, picture header, tile header, super block header and / or CTU header.

[0345] In some embodiments, a block level flag enables GSO for a block, and the method 3500 may further include: receiving a signal including additional high-level syntax to indicate the number of bands used for band offset (BO) in the block. For example, when a block level flag enables GSO for a block, additional block level syntax is signaled to indicate the number of bands used for BO in the block. In some implementations of these embodiments, the number of bands used for band offset in the block is less than the number of bands signaled at the frame level. For example, the number of bands used for BO at the block level is always less than or equal to the maximum number of bands signaled at the frame level. In addition or alternatively, the syntax element is or includes a high-level syntax, and the method 3500 may further include: receiving a signal including a flag to indicate whether a single band or a maximum number of bands specified in the high-level syntax is applied to the current block. For example, a flag is signaled to indicate whether a single band or a maximum number of bands specified in the high-level syntax is applied to the current block.

[0346] The embodiments of the present disclosure may be used alone or in any order. In addition, each of the method (or embodiment), encoder and decoder may be implemented by a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium. The term block may include a prediction block, a coding block, or a coding unit, i.e., a CU. The embodiments of the present disclosure may be applied to a luminance block or a chrominance block.

[0347] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Fig.36 A computer system (3600) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0348] Computer software may be encoded using any suitable machine code or computer language, which may be assembled, compiled, linked, or the like to create code comprising instructions that may be executed directly by one or more computer central processing units (CPU), graphics processing units (GPU), etc., or through interpretation, microcode execution, or the like.

[0349] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0350] Fig.36 The components shown for the computer system (3600) are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Neither should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components shown in the exemplary embodiment of the computer system (3600).

[0351] The computer system (3600) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), olfactory input (not depicted). The human-machine interface devices may also be used to capture certain media that are not necessarily directly related to a person's conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, including three-dimensional video including stereoscopic video).

[0352] Input human-machine interface devices may include one or more of the following (only one of each is depicted): keyboard (3601), mouse (3602), touchpad (3603), touch screen (3610), data gloves (not shown), joystick (3605), microphone (3606), scanner (3607), camera (3608).

[0353] The computer system (3600) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include: tactile output devices (e.g., tactile feedback through a touch screen (3610), a data glove (not shown), or a joystick (3605), but there may also be tactile feedback devices that are not used as input devices); audio output devices (e.g., speakers (3609), headphones (not depicted)); visual output devices (e.g., screens (3610), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which may be capable of outputting two-dimensional visual output or more than three-dimensional output through means such as stereoscopic image output; virtual reality glasses (not depicted); holographic displays and smoke generators (not depicted)); and printers (not depicted).

[0354] The computer system (3600) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (3620) with CD / DVD etc. media (3621), thumb drives (3622), removable hard drives or solid-state drives (3623), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD based devices such as security dongles (not depicted), etc.

[0355] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0356] The computer system (3600) may also include an interface (3654) to one or more communication networks (3655). The network may be, for example, a wireless network, a wired network, an optical network. The network may also be a local area network, a wide area network, a metropolitan area network, an in-vehicle and industrial network, a real-time network, a delay-tolerant network, etc. Examples of networks include: local area networks such as Ethernet; wireless LAN (Local Area Network, LAN); cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; TV wired or wireless wide area digital networks including cable TV, satellite TV and terrestrial broadcast TV; vehicle and industrial networks including CAN bus, etc. Some networks typically require an external network interface adapter attached to some universal data port or peripheral bus (3649) (for example, a USB (Universal Serial Bus, USB) port of the computer system (3600)); other networks are typically integrated into the core of the computer system (3600) by attaching to the system bus as described below (for example, an Ethernet interface to a PC computer system or a cellular network interface to a smart phone computer system). The computer system (3600) can communicate with other entities using any of these networks. Such communications can be one-way receive only (e.g., broadcast TV), one-way send only (e.g., a CAN bus to certain CAN bus devices), or two-way, for example, to other computer systems using a local area digital network or a wide area digital network. Certain protocols and protocol stacks can be used on each of those networks and network interfaces as described above.

[0357] The above-mentioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (3640) of the computer system (3600).

[0358] The core (3640) may include one or more central processing units (CPUs) (3641), graphics processing units (GPUs) (3642), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (3643), hardware accelerators (3644) for certain tasks, graphics adapters (3650), etc. These devices as well as read-only memory (random-access memory, ROM) (3645), random access memory (3646), internal mass storage devices (e.g., internal non-user accessible hard disk drives, SSDs, etc.)

[0359] (3647) can be connected through a system bus (3648). In some computer systems, the system bus (3648) can be accessed in the form of one or more physical plugs to enable expansion through additional CPUs, GPUs, etc. Peripheral devices can be attached to the core's system bus (3648) directly or through a peripheral bus (3649). In an example, a screen (3610) can be connected to a graphics adapter (3650). The architecture of the peripheral bus includes PCI (Peripheral Component Interconnect / Interface, PCI), USB, etc.

[0360] The CPU (3641), GPU (3642), FPGA (3643) and accelerator (3644) can execute certain instructions, which can be combined to form the computer code mentioned above. The computer code can be stored in ROM (3645) or RAM (Random Access Memory, RAM) (3646). Transition data can also be stored in RAM (3646), while permanent data can be stored in, for example, an internal mass storage device (3647). Fast storage and retrieval of any memory device in the memory device can be achieved by using a high-speed cache memory, which can be closely associated with one or more CPUs (3641), GPUs (3642), mass storage devices (3647), ROMs (3645), RAMs (3646), etc.

[0361] The computer readable medium may have thereon computer codes for performing various computer-implemented operations. These media and computer codes may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0362] As a non-limiting example, a computer system (3600) having an architecture, in particular a core (3640), can provide functionality provided by a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software implemented in one or more tangible computer-readable media. Such a computer-readable medium can be a medium associated with a user-accessible mass storage device as described above, and certain storage devices of the core (3640) having a non-transient nature, such as a core internal mass storage device (3647) or a ROM (3645). Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core (3640). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the core (3640) and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in the RAM (3646) and modifying such a data structure according to a software-defined process. In addition or as an alternative, a computer system may provide functionality (e.g., accelerator (3644)) due to logic hardwired or otherwise embodied in circuitry that may operate in place of or in conjunction with software to perform a particular process or particular portion of a particular process described herein. Where appropriate, reference to software may include logic, and conversely, reference to logic may include software. Where appropriate, reference to a computer-readable medium may include circuitry (e.g., an integrated circuit (IC)) storing software for execution, circuitry implementing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0363] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various substitution equivalents that fall within the scope of the present disclosure. It will therefore be appreciated that those skilled in the art will be able to conceive of many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

[0364] Appendix A: Acronyms

[0365] ALF: Adaptive Loop Filter

[0366] AMVP: Advanced Motion Vector Prediction

[0367] APS: Adaptive Parameter Set

[0368] ASIC: Application-Specific Integrated Circuit

[0369] AV1: Alliance for Open Media Video 1

[0370] AV2: Alliance for Open Media Video 2

[0371] BCW: Bidirectional Prediction with CU-Level Weights

[0372] BM: Bilateral Matching

[0373] BMS: Benchmark Set

[0374] CANBus: Controller Area Network Bus

[0375] CC-ALF: Cross-Component Adaptive Loop Filter

[0376] CCSO: Cross Component Sample Offset

[0377] CD: Compact Disc

[0378] CDEF: Constrained Directional Enhancement Filter

[0379] CDF: Cumulative density function

[0380] CfL: Predict chrominance components based on luma components

[0381] CIIP: Combined Intra-Inter Prediction

[0382] CPU: Central Processing Unit

[0383] CRT: cathode ray tube

[0384] CTB: Coding Tree Block

[0385] CTB: Coding Tree Block

[0386] CTU: Coding Tree Unit

[0387] CTU: Coding Tree Unit

[0388] CU: Coding Unit

[0389] DMVR: Decoder-side Motion Vector Refinement

[0390] DPB: Decoded Picture Buffer

[0391] DPS: Decoding Parameter Set

[0392] DVD: Digital Video Disc

[0393] FPGA: Field Programmable Gate Array

[0394] GBI: Generalized Bidirectional Prediction

[0395] GOP: Group of Pictures

[0396] GPU: Graphics Processing Unit

[0397] GSM: Global System for Mobile Communications

[0398] HDR: High Dynamic Range

[0399] HEVC: High Efficiency Video Coding

[0400] HRD: Hypothesized Reference Decoder

[0401] IBC (or IntraBC): Intra Block Copy

[0402] IC: Integrated Circuit

[0403] ISP: Intra-frame sub-partitioning

[0404] JEM: Joint Exploration Model

[0405] JVET: Joint Video Exploration Team

[0406] LAN: Local Area Network

[0407] LCD: Liquid Crystal Display

[0408] LCU: Largest Coding Unit

[0409] LR: Loop Restoration Filter

[0410] LSO: Local Sample Offset

[0411] LTE: Long Term Evolution

[0412] MMVD: Merge mode with motion vector difference

[0413] MPM: Most Probable Mode

[0414] MV: Motion Vector

[0415] MV: Motion Vector

[0416] MVD: Motion Vector Difference

[0417] MVD: Motion Vector Difference

[0418] MVP: Motion Vector Predictor

[0419] OLED: Organic Light Emitting Diode

[0420] PB: prediction block

[0421] PCI: Peripheral Component Interconnect

[0422] PDPC: Position Dependent Prediction Combination

[0423] PLD: Programmable Logic Device

[0424] POC: Image Sequential Counting

[0425] PPS: Picture Parameter Set

[0426] PU: prediction unit

[0427] PU: prediction unit

[0428] RAM: Random Access Memory

[0429] ROM: Read Only Memory

[0430] RPS: Reference Picture Set

[0431] SAD: Sum of Absolute Differences

[0432] SAO: Sample Adaptive Offset

[0433] SB: Super Block

[0434] SCC: Screen Content Coding

[0435] SDP: Semi-Decoupled Partitioning

[0436] SDR: Standard Dynamic Range

[0437] SDT: Semi-Decoupled Tree

[0438] SEI: Supplementary Enhancement Information

[0439] SNR: Signal to Noise Ratio

[0440] SPS: Sequence parameter settings

[0441] SSD: Solid State Drive

[0442] SST: Semi-Separating Tree

[0443] TM: Template Matching

[0444] TU: Transform Unit

[0445] TU: Transform Unit

[0446] USB: Universal Serial Bus

[0447] VPS: Video Parameter Set

[0448] VUI: Video Availability Information

[0449] VVC: Versatile Video Coding

[0450] WAIP: Wide Angle Intra Prediction

Claims

1. A method for decoding a video stream, characterized in that: The method comprises: receiving an encoded video stream, the encoded video stream comprising a syntax element indicating whether to apply a sample offset filtering process to frames in the encoded video stream at a block level; Based on the syntax element being a predefined value, determining to enable at least one of a generalized sample offset (GSO), a local sample offset (LSO), or a cross-component sample offset (CCSO) for the frame at the block level; and decoding the encoded video stream by applying at least one of the GSO, the LSO, or the CCSO at the block level; When the first block size of the luminance component is the same as or smaller than the second block size of the chrominance component, GSO is applied; receiving a block-level flag indicating that one of the GSO, the LSO, or the CCSO is enabled for a block, wherein the block-level flag further indicates that the other of the GSO, the LSO, or the CCSO is enabled or disabled for the block; the block-level flag indicating whether CCSO or LSO is applied to the block is used as a context for signaling a block-level flag to indicate whether LSO or CCSO is applied to the block, respectively; The filtering process of CCSO uses the reconstructed samples of the first color component as input, and the output is applied to the second color component which is a different color component from the first color component; the filtering process of LSO uses the reconstructed samples of the first color component as input, and the output is applied to the same first color component; two types of offset values ​​are used in GSO, namely the gradient offset GO and the band offset BO, the gradient offset GO corresponds to the difference value between the adjacent samples of the juxtaposition or current sample in CCSO and LSO, and the band offset BO corresponds to the level value of the current sample itself.

2. The method according to claim 1, characterized in that Determining to enable at least one of the GSO, the LSO, or the CCSO includes: determining to enable the GSO based on the syntax element being the predefined value.

3. The method according to claim 1, characterized in that Determining to enable at least one of the GSO, the LSO, or the CCSO includes: determining to enable the LSO based on the syntax element being the predefined value.

4. The method according to claim 1, characterized in that Determining to enable at least one of the GSO, the LSO, or the CCSO includes: determining to enable the CCSO based on the syntax element being the predefined value.

5. The method according to any one of claims 1 or 2, characterized in that: The method further comprises: In a case where the syntax element indicates that the GSO is enabled, a signal indicating a first block level flag of a luminance component having a first block size is received when a second block level flag of a chroma component having a second block size is signaled, and the first block size is the same as or smaller than the second block size of the chroma component.

6. The method according to any one of claims 1 or 2, characterized in that: The method further comprises: In a case where the syntax element indicates that the GSO is enabled, a signal indicating a granularity of at least one of a chroma component or a luma component for application of the GSO is received in a high-level syntax, wherein the high-level syntax includes: a video parameter set (VPS), a picture parameter set (PPS), a sequence parameter set (SPS), an adaptation parameter set (APS), a frame header, a slice header, a picture header, a tile header, a super block header, or a coding tree unit (CTU) header.

7. The method according to any one of claims 1 or 2, characterized in that: The syntax element includes a high-level syntax, and the GSO is enabled by the high-level syntax. The method further includes: A signal is received including additional high level syntax to indicate a maximum number of bands for band offsets.

8. The method according to claim 7, characterized in that The high-level syntax includes: video parameter set (VPS), picture parameter set (PPS), sequence parameter set (SPS), adaptation parameter set (APS), frame header, slice header, picture header, tile header, super block header or coding tree unit (CTU) header.

9. The method according to any one of claims 1 or 2, characterized in that: A block level flag enables the GSO for a block, the method further comprising: A signal is received including additional high level syntax to indicate the number of bands used for the band offsets in the block.

10. The method according to claim 9, characterized in that The number of bands used for the band offset in the block is smaller than the number of bands signaled at frame level.

11. The method according to claim 9, characterized in that The grammar element includes a high-level grammar, and the method further includes: A signal including a flag is received to indicate whether a single band or a maximum number of bands specified in the high-level syntax is applied for a current block.

12. The method according to any one of claims 1 to 4, characterized in that The block sizes of the frames in the encoded video stream are different for luma and chroma components corresponding to the frames.

13. A device for decoding a video stream, characterized in that: The device comprises: a memory storing a plurality of instructions; and a processor configured to execute the plurality of instructions, and when executing the plurality of instructions, the processor is configured to: receiving an encoded video stream, the encoded video stream comprising a syntax element indicating whether to apply a sample offset filtering process to frames in the encoded video stream at a block level; Based on the syntax element being a predefined value, determining to enable at least one of a generalized sample offset (GSO), a local sample offset (LSO), or a cross-component sample offset (CCSO) for the frame at the block level; and decoding the encoded video stream by applying at least one of the GSO, the LSO, or the CCSO at the block level; The processor is further configured to: When the first block size of the luminance component is the same as or smaller than the second block size of the chrominance component, GSO is applied; The processor is further configured, when executing the plurality of instructions, to: receive a block-level flag indicating that one of the GSO, the LSO, or the CCSO is enabled for a block, wherein the block-level flag further indicates that another of the GSO, the LSO, or the CCSO is enabled or disabled for the block; the block-level flag indicating whether CCSO or LSO is applied to a block is used as a context for signaling a block-level flag to indicate whether LSO or CCSO is applied to a block, respectively; The filtering process of CCSO uses the reconstructed samples of the first color component as input, and the output is applied to the second color component which is a different color component from the first color component; the filtering process of LSO uses the reconstructed samples of the first color component as input, and the output is applied to the same first color component; two types of offset values ​​are used in GSO, namely the gradient offset GO and the band offset BO, the gradient offset GO corresponds to the difference value between the adjacent samples of the juxtaposition or current sample in CCSO and LSO, and the band offset BO corresponds to the level value of the current sample itself.

14. The device according to claim 13, characterized in that The processor is further configured to: In a case where the syntax element indicates that the GSO is enabled, a signal indicating a first block level flag of a luminance component having a first block size is received when a second block level flag of a chroma component having a second block size is signaled, and the first block size is the same as or smaller than the second block size of the chroma component.

15. The device according to claim 13, characterized in that The processor is further configured to: A signal is received in a high-level syntax indicating a granularity of at least one of a chroma component or a luma component for an application of the GSO, wherein the high-level syntax comprises: a video parameter set (VPS), a picture parameter set (PPS), a sequence parameter set (SPS), an adaptation parameter set (APS), a frame header, a slice header, a picture header, a tile header, a super-block header, or a coding tree unit (CTU) header.

16. The device according to claim 13, characterized in that The syntax elements include a high level syntax by which the GSO is enabled, and the processor, when executing the plurality of instructions, is further configured to receive a signal including additional high level syntax to indicate a maximum number of bands for band offset; or A block level flag enables the GSO for a block, and the processor, when executing the plurality of instructions, is further configured to: receive a signal including additional high level syntax to indicate a number of bands for band offsets in the block.

17. A non-transitory computer-readable storage medium storing a plurality of instructions executable by a processor, characterized in that: When executed by the processor, the plurality of instructions The method is configured to cause the processor to perform the method according to any one of claims 1 to 12.

18. A method for processing a video bitstream, characterized in that: The video bit stream is decoded according to the decoding method according to any one of claims 1 to 12.

Images