Method and apparatus for video filtering

Abstract

To provide a method and apparatus for filtering in video encoding / decoding.SOLUTION: An apparatus for video filtering includes a processing circuit. The processing circuit determines a first offset value for applying a non-linear mapping based filter based on a first reconstructed sample at a first node in a loop filter chain. The processing circuit then adds the first offset value to an intermediate reconstructed sample at a second node along the loop filter chain to generate a second reconstructed sample.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Patent Application No. 17 / 449,126, entitled "METHOD AND APPARATUS FOR VIDEO FILTERING," filed September 28, 2021, which claims the benefit of priority to U.S. Provisional Application No. 63 / 164,478, entitled "FLEXIBLE FILTER LOCATION FOR SAMPLE OFFSET," filed March 22, 2021. The entire disclosures of the prior applications are hereby incorporated by reference in their entireties.

[0002] This disclosure generally describes embodiments related to video coding. [Background technology]

[0003] The discussion of the background art provided herein is intended to provide a general context for the present disclosure. The work of the inventors named herein is not admitted, expressly or impliedly, as prior art to the present disclosure to the extent that their work is described in this Background section, and aspects of the description that may not otherwise qualify as prior art at the time of filing.

[0004] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each with spatial dimensions of, for example, 1920x1080 luma samples and associated chroma samples. The series of pictures can have a fixed or variable (also informally known as frame rate) picture rate of, for example, 60 pictures per second or 60 Hz. Uncompressed video has specific bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luma sample resolution at 60 Hz frame rate) at 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0005] One goal of video coding and decoding may be to reduce redundancy in the input video signal through compression. Compression may help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than one order of magnitude. Both lossless and lossy compression may be employed, as well as combinations thereof. Lossless compression refers to techniques that can restore an exact copy of the original signal from the compressed original signal. With lossy compression, the restored signal may not be identical to the original signal, but the distortion between the original and restored signals is small enough to make the restored signal useful for the intended application. For video, lossy compression is widely adopted. The amount of acceptable distortion depends on the application, e.g., users of a particular consumer streaming application may tolerate higher distortion than users of a television distribution application. The achievable compression ratio may reflect that the greater the acceptable / tolerable distortion, the higher the compression ratio may be.

[0006] Video encoders and decoders can utilize techniques from a number of broad categories, including, for example, motion compensation, transform, quantization, and entropy coding.

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture may be an intra picture. Intra pictures, and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state, and therefore can be used as the first picture in a coded video bitstream and video session, or as a still image. Samples of an intra block may be subjected to a transform, and the transform coefficients may be quantized before entropy coding. Intra prediction may be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficients, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0008] Conventional intra-coding, for example as known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to predict from surrounding sample data and / or metadata obtained during the encoding / decoding of a block of data that is spatially adjacent and preceding in decoding order. Such techniques are hereafter referred to as "intra-prediction" techniques. It should be noted that in at least some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from a reference picture.

[0009] Intra prediction may take many different forms. When two or more of such techniques may be used in a given video coding technique, the technique in use may be coded in intra prediction mode. In some examples, a mode may have sub-modes and / or parameters, which may be coded separately or may be included in a mode codeword. Which codeword is used for a given mode / sub-mode / parameter combination may affect the coding efficiency improvement by intra prediction, and therefore may affect the entropy coding technique used to convert the codeword into a bitstream.

[0010] A specific mode of intra prediction was introduced in H.264, improved in H.265, and further improved in newer coding techniques such as the Joint Search Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). A predictor block can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of the neighboring samples are copied to the predictor block according to a direction. The reference to the direction in use can be coded in the bitstream or it can be predicted itself.

[0011] Referring to FIG. 1A, depicted at the bottom right is a subset of 9 predictor directions known from the 33 possible predictor directions (corresponding to the 33 angle modes of the 35 intra modes) of H.265. The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction from which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] 1A, at the top left is depicted a square block (104) of 4×4 samples (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions in the block (104). Since the block is 4×4 samples in size, S44 is at the bottom right. Also shown are reference samples that follow a similar numbering scheme. The reference samples are labeled with R, their Y position (e.g., row index), and X position (column index) relative to the block (104). In both H.264 and H.265, the predicted samples are adjacent to the block being reconstructed, so negative values ​​do not need to be used.

[0013] Intra-picture prediction can work by copying reference sample values ​​from adjacent samples as assigned by the signaled prediction direction. For example, assume that the coded video bitstream includes signaling for this block indicating a prediction direction that coincides with the arrow (102), i.e., the sample is predicted from one or more prediction samples located at an angle of 45 degrees from the horizontal and to the upper right. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the orientation is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example by interpolation, to calculate the reference sample.

[0015] The number of possible directions has increased as video coding techniques have developed. In H.264 (2003), nine different directions could be represented. In H.265 (2013), this increased to 33, and at the time of this disclosure, JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and certain techniques of entropy coding are used to represent those likely directions with a small number of bits, accepting a certain penalty for the less likely directions. Furthermore, the direction itself can sometimes be predicted from neighboring directions used in adjacent already decoded blocks.

[0016] FIG. 1B shows a schematic diagram (180) depicting 65 intra prediction directions with JEM to illustrate the increasing number of prediction directions over time.

[0017] The mapping of intra-prediction direction bits in a coded video bitstream that represent the direction may vary from one video coding technique to another, and can range, for example, from a simple direct mapping of prediction directions to intra-prediction modes, to complex adaptation schemes involving codewords, most likely modes, and similar techniques. In all cases, however, there may be certain directions that are statistically less likely to occur in the video content than certain other directions. Because the goal of video compression is to reduce redundancy, these less likely directions are represented by a greater number of bits than more likely directions in well-performing video coding techniques.

[0018] Motion compensation may be a lossy compression technique, and may refer to a technique in which blocks of sample data from a previously reconstructed picture or part thereof (reference picture) are used to predict a newly reconstructed picture or part of a picture after being spatially shifted in a direction indicated by a motion vector (hereafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or three dimensions, with the third dimension being an indication of the reference picture in use (the latter may indirectly be a temporal dimension).

[0019] In some video compression techniques, the MV applicable to a particular region of sample data can be predicted from other MVs, e.g., from an MV associated with another region of sample data that is spatially adjacent to the region being restored and precedes that MV in decoding order. Doing so can significantly reduce the amount of data required to code the MV, thereby eliminating redundancy and increasing the compression ratio. For example, when coding an input video signal derived from a camera (known as natural video), MV prediction can work effectively because there exists a statistical possibility that a region larger than the region to which a single MV is applicable moves in a similar direction, and thus, in some cases, can be predicted using similar motion vectors derived from MVs of neighboring regions. As a result, the detected MV for a given region is similar or the same as the MV predicted from the surrounding MVs, and after entropy coding, can be represented with fewer bits than would be used when coding the MV directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself can be lossy, e.g., due to rounding errors when computing a predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms that H.265 offers, the one described herein is a technique hereafter referred to as "spatial merging".

[0021] Referring to Figure 2, a current block (201) contains samples that the encoder found during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of coding its MV directly, the MV can be derived from metadata associated with one or more reference pictures, e.g., the most recent reference picture (in decoding order), using MVs associated with any one of five surrounding samples, denoted A0, A1, and B0, B1, B2 (202-206, respectively). In H.265, MV prediction can use predictors from the same reference picture that neighboring blocks use. Summary of the Invention [Means for solving the problem]

[0022] Aspects of the present disclosure provide a method and apparatus for filtering in video encoding / decoding. In some examples, the apparatus for video filtering includes a processing circuit. The processing circuit determines a first offset value for applying a nonlinear mapping-based filter based on a first reconstructed sample at a first node in a loop filter chain. The processing circuit then adds the first offset value to an intermediate reconstructed sample at a second node along the loop filter chain to generate a second reconstructed sample.

[0023] In some examples, the nonlinear mapping-based filter is a inter-component sample offset (CCSO) filter, and the intermediate reconstruction samples and the first reconstruction samples are samples of different color components.

[0024] In some examples, the nonlinear mapping-based filter is a local sample offset (LSO) filter, and the intermediate reconstruction samples and the first reconstruction samples are samples of the same color component.

[0025] In some embodiments, the first node and the second node are of the same node, such as an input node of a loop filter chain, an output node of a loop filter chain, or an intermediate node of a loop filter chain. In one example, the first restored sample is generated by a processing module prior to a deblocking filter. In another example, the first restored sample is generated by a deblocking filter. In another example, the first restored sample is generated by a constrained directional enhancement filter. In another example, the first restored sample is generated by a loop restoration filter.

[0026] In some examples, the first node and the second node are of different nodes. In one example, the first restored sample is generated by a processing module prior to the deblocking filter, and the intermediate restored sample is generated by at least one of the deblocking filter, the constrained directional enhancement filter, or the loop restoration filter.

[0027] In some examples, the first reconstructed sample is generated by a deblocking filter, and the intermediate reconstructed sample is generated by at least one of a constrained directional enhancement filter or a loop reconstruction filter.

[0028] In some examples, the first reconstruction sample is generated by a constrained directional extension filter, and the intermediate reconstruction sample is generated by a loop reconstruction filter.

[0029] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any of the methods for video encoding / decoding.

[0030] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0031] [Figure 1A] FIG. 2 is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B] FIG. 2 is a diagram of an example intra-prediction direction. [Diagram 2] FIG. 2 is a schematic diagram of a current block and its surrounding spatial merge candidates in one example. [Diagram 3] 1 is a schematic diagram of a simplified block diagram of a communication system (300), according to one embodiment. [Figure 4] FIG. 4 is a schematic diagram of a simplified block diagram of a communication system (400) according to one embodiment. [Diagram 5] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 6] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 7] FIG. 4 is a block diagram of an encoder according to another embodiment. [Figure 8] FIG. 4 is a block diagram of a decoder according to another embodiment. [Figure 9] 4A-4C are diagrams illustrating example filter shapes according to embodiments of the present disclosure. [Figure 10A] FIG. 13 illustrates an example of sub-sampled positions used to calculate gradients, according to an embodiment of the present disclosure. [Figure 10B] FIG. 13 illustrates an example of sub-sampled positions used to calculate gradients, according to an embodiment of the present disclosure. [Figure 10C] FIG. 13 illustrates an example of sub-sampled positions used to calculate gradients, according to an embodiment of the present disclosure. [Figure 10D] FIG. 13 illustrates an example of sub-sampled positions used to calculate gradients, according to an embodiment of the present disclosure. [Figure 11A] FIG. 13 illustrates an example of a virtual boundary filtering process, according to an embodiment of the present disclosure. [Figure 11B] FIG. 13 illustrates an example of a virtual boundary filtering process, according to an embodiment of the present disclosure. [Figure 12A] 1A-1C are diagrams illustrating examples of symmetric padding operations at virtual boundaries according to embodiments of the present disclosure. [Figure 12B] 1A-1C are diagrams illustrating examples of symmetric padding operations at virtual boundaries according to embodiments of the present disclosure. [Figure 12C] 1A-1C are diagrams illustrating examples of symmetric padding operations at virtual boundaries according to embodiments of the present disclosure. [Figure 12D] 1A-1C are diagrams illustrating examples of symmetric padding operations at virtual boundaries according to embodiments of the present disclosure. [Figure 12E] 1A-1C are diagrams illustrating examples of symmetric padding operations at virtual boundaries according to embodiments of the present disclosure. [Figure 12F] 1A-1C are diagrams illustrating examples of symmetric padding operations at virtual boundaries according to embodiments of the present disclosure. [Figure 13] FIG. 2 illustrates example partitions of a picture, in accordance with some embodiments of the present disclosure. [Figure 14] A diagram showing quadtree division patterns for pictures in some examples. [Figure 15] FIG. 2 illustrates an inter-component filter according to one embodiment of the present disclosure. [Figure 16] FIG. 2 illustrates an example of a filter shape, according to one embodiment of the present disclosure. [Figure 17] FIG. 2 illustrates an example syntax for an inter-component filter, in accordance with some embodiments of the present disclosure. [Figure 18A] 4A-4C are diagrams illustrating example positions of chroma samples relative to luma samples, according to embodiments of the present disclosure. [Figure 18B] 4A-4C are diagrams illustrating example positions of chroma samples relative to luma samples, according to embodiments of the present disclosure. [Figure 19] FIG. 1 illustrates an example of a direction search according to one embodiment of the present disclosure. [Figure 20] FIG. 13 illustrates an example showing subspace projection in some examples. [Figure 21] 1 is a table of multiple sample adaptive offset (SAO) types, according to one embodiment of the present disclosure. [Figure 22] FIG. 13 illustrates example patterns for pixel classification at edge offset in some examples. [Figure 23] 13 is a table for pixel classification rules for edge offsets in some examples. [Figure 24] FIG. 1 illustrates an example of a syntax that may be signaled. [Diagram 25] FIG. 2 illustrates an example of a filter support area, according to some embodiments of the present disclosure. [Figure 26] FIG. 13 illustrates an example of another filter support area according to some embodiments of the present disclosure. [Figure 27A] 1 is a table having 81 combinations according to one embodiment of the present disclosure. [Figure 27B] 1 is a table having 81 combinations according to one embodiment of the present disclosure. [Figure 27C] 1 is a table having 81 combinations according to one embodiment of the present disclosure. [Figure 28] FIG. 1 illustrates seven filter shape configurations of three filter taps in one example. [Figure 29] 1 is a block diagram of a loop filter chain in some examples. [Figure 30A] FIG. 2 illustrates an example of a loop filter chain including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 30B] FIG. 2 illustrates an example of a loop filter chain including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 30C] FIG. 2 illustrates an example of a loop filter chain including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 30D]FIG. 2 illustrates an example of a loop filter chain including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 31A] FIG. 2 illustrates an example of a loop filter chain including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 31B] FIG. 2 illustrates an example of a loop filter chain including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 32A] FIG. 2 illustrates an example of a loop filter chain including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 32B] FIG. 2 illustrates an example of a loop filter chain including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 32C] FIG. 2 illustrates an example of a loop filter chain including nonlinear mapping-based filters at different positions in the loop filter chain. [Diagram 33] FIG. 2 illustrates another example of a loop filter chain including a nonlinear mapping-based filter in some examples. [Diagram 34] FIG. 2 illustrates an example of a loop filter chain including multiple nonlinear mapping-based filters in some examples. [Diagram 35] FIG. 2 illustrates another example of a loop filter chain including multiple nonlinear mapping-based filters in some examples. [Diagram 36] 1 is a flowchart outlining a process according to one embodiment of the present disclosure. [Figure 37] 1 is a schematic diagram of a computer system, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] FIG. 3 shows a simplified block diagram of a communication system (300) according to one embodiment of the present disclosure. The communication system (300) includes, for example, a plurality of terminal devices that can communicate with each other via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) perform unidirectional transmission of data. For example, the terminal device (310) can encode video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data can be transmitted in the form of one or more coded video bitstreams. The terminal device (320) can receive the coded video data from the network (350), decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. Unidirectional data transmission may be common, such as in media serving applications.

[0033] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) performing a bidirectional transmission of coded video data, such as may occur during a video conference. In the case of a bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) can code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) can also receive coded video data transmitted by the other of the terminal devices (330) and (340), can decode the coded video data to recover the video pictures, and can display the video pictures on an accessible display device according to the recovered video data.

[0034] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) may be depicted as a server, a personal computer, and a smartphone, although the principles of the present disclosure may not be so limited. Embodiments of the present disclosure find application with laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (350) represents any number of networks that convey coded video data between the terminal devices (310), (320), (330), and (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of the network (350) may not be important to the operation of the present disclosure unless otherwise described herein below.

[0035] 4 shows an arrangement of a video encoder and a video decoder in a streaming environment as one example for application of the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital television, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0036] The streaming system may include a video source (401), e.g., a capture subsystem (413) that may include, e.g., a digital camera, that creates a stream of uncompressed video pictures (402). In one example, the stream of video pictures (402) includes samples taken by a digital camera. The stream of video pictures (402), depicted as thick lines to emphasize a larger amount of data when compared to the encoded video data (404) (or coded video bitstream), may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or coded video bitstream (404)), depicted as thin lines to emphasize a smaller amount of data when compared to the stream of video pictures (402), may be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes an input copy (407) of the encoded video data and creates an output stream (411) of video pictures that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not depicted). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) can be encoded according to a particular video coding / compression standard. Examples of such standards include ITU-T Recommendation H.265.In one example, the developing video coding standard is informally known as Versatile Video Coding (VVC), and the disclosed subject matter may be used in the context of VVC.

[0037] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0038] 5 shows a block diagram of a video decoder (510) according to one embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) of the example of FIG. 4.

[0039] The receiver (531) may receive one or more coded video sequences, in the same or another embodiment, one coded video sequence at a time, to be decoded by the video decoder (510), with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (531) may receive the coded video data along with other data, e.g., coded audio data and / or auxiliary data streams, that may be forwarded to their respective using entities (not depicted). The receiver (531) may separate the coded video sequences from the other data. To combat network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In other applications, it may be outside the video decoder (510) (not depicted). In still other applications, there may be a buffer memory (not depicted) outside the video decoder (510), e.g., to combat network jitter, plus another buffer memory (515) inside the video decoder (510), e.g., to handle playout timing. When the receiver (531) is receiving data from a store / forward device of sufficient bandwidth and controllability, or from an equally synchronous network, the buffer memory (515) may not be needed or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (515) may be needed and may be relatively large, may be advantageously adaptively sized, and may be implemented at least in part in an operating system or similar element (not depicted) outside the video decoder (510).

[0040] The video decoder (510) may include a parser (520) to recover symbols (521) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (510) and potentially information for controlling a rendering device such as a rendering device (512) (e.g., a display screen) that is not an integral part of the electronic device (530) but may be coupled to the electronic device (530) as shown in FIG. 5. The control information for the rendering device may be in the form of a supplemental enhancement information (SEI message) or a video usability information (VUI) parameter set fragment (not depicted). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding, etc., with or without context sensitivity. The parser (520) can extract, from the coded video sequence, a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups can include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (520) can also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0041] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0042] The recovery of the symbols (521) can involve a number of different units, depending on the type of coded video picture or part thereof (inter-picture and intra-picture, inter-block and intra-block, etc.), as well as other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not depicted for the sake of clarity.

[0043] Beyond the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0044] The first unit is a scalar / inverse transform unit (551). The scalar / inverse transform unit (551) receives quantized transform coefficients as well as control information including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. as symbols (521) from the parser (520). The scalar / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0045] In some cases, the output samples of the scalar / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current picture buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (555) adds, possibly on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scalar / inverse transform unit (551).

[0046] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (521) related to the block, these samples may be added to the output of the scalar / inverse transform unit (551) by the aggregator (555) to generate output sample information (in this case referred to as residual samples or residual signals). The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (553) in the form of symbols (521), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0047] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filter techniques that are controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but may also be responsive to previously reconstructed and loop filtered sample values ​​as well as to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence.

[0048] The output of the loop filter unit (556) may be a sample stream that can be stored in a reference picture memory (557) for use in future inter-picture prediction as well as output to a rendering device (512).

[0049] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified (e.g., by the parser (520)) as a reference picture, the current picture buffer (558) can become part of the reference picture memory (557), and any unused current picture buffer can be reallocated before beginning reconstruction of the next coded picture.

[0050] The video decoder (510) may perform decoding operations according to a given video compression technique in a standard such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence adheres to both the syntax of the video compression technique or standard and the profile documented in the video compression technique. Specifically, the profile may select some tools from all tools available in the video compression technique or standard as the only tools available for use under that profile. Also, what is required for conformance may be that the complexity of the coded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled within the coded video sequence.

[0051] In one embodiment, the receiver (531) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately recover the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0052] 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the video encoder (403) of the example of FIG. 4.

[0053] The video encoder (603) can receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of FIG. 6) that can capture video images that are encoded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0054] The video source (601) may provide a source video sequence that is coded by the video encoder (603) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0055] According to one embodiment, the video encoder (603) can code and compress pictures of a source video sequence into a coded video sequence (643) in real-time or under any other time constraint required by an application. Enforcing an appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units described below. For clarity, coupling is not depicted. Parameters set by the controller (650) can include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other appropriate functions associated with the video encoder (603) optimized for a particular system design.

[0056] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture), as well as a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to that which the (remote) decoder also creates (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques contemplated in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream leads to a bit-exact result regardless of the location of the decoder (local or remote), the content in the reference picture memory (634) is also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder "sees" when using the prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is also used in several related technologies.

[0057] The operation of the "local" decoder (633) may be the same as the operation of a "remote" decoder, such as the video decoder (510), already described in detail above in conjunction with Figure 5. However, and with brief reference to Figure 5, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515), and the parser (520), may not be fully implemented in the local decoder (633).

[0058] An observation that can be made at this point is that any decoder techniques other than parsing / entropy decoding present in a decoder must necessarily be present in the corresponding encoder in substantially identical functional form. For this reason, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder techniques can be omitted, since they are the inverse of the decoder techniques described generically. Only in certain areas is more detailed description necessary and is provided below.

[0059] In operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0060] The local video decoder (633) can decode coded video data of pictures that may be designated as reference pictures based on symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a replica of the source video sequence, usually with some errors. The local video decoder (633) can replicate the decoding process that may be performed by the video decoder on the reference pictures, such that the reconstructed reference pictures are stored in a reference picture cache (634). In this way, the video encoder (603) can locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures obtained by a far-end video decoder (without transmission errors).

[0061] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (635) may operate on sample blocks on a pixel block by pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0062] The controller (650) can manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0063] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0064] The transmitter (640) can buffer the coded video sequence created by the entropy coder (645) and prepare it for transmission over a communication channel (660), which may be a hardware / software link to a storage device that stores the coded video data. The transmitter (640) can merge the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0065] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a particular coded picture type to each coded picture, which may affect the coding techniques that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:

[0066] An intra picture (I-picture) may be a picture that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow various types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures, as well as their respective uses and characteristics.

[0067] A predictive picture (P picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0068] A bidirectionally predicted picture (B-picture) may be a picture that can be coded and decoded using intra- or inter-prediction that uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-prediction picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0069] A source picture is usually spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective picture of the block. For example, blocks of an I picture may be non-predictively coded or they may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0070] The video encoder (603) may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec. H.265. In its operations, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0071] In one embodiment, the transmitter (640) can transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0072] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that was previously coded and is still buffered in the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0073] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, both of which are earlier in decoding order than the current picture in a video (but may be past and future in display order, respectively), are used. A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.

[0074] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0075] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64×64 pixel CTU can be partitioned into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter prediction type or an intra prediction type. The CU is partitioned into one or more prediction units (PUs) depending on the temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values) of 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0076] 7 shows a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive a processed block (e.g., a predictive block) of sample values ​​in a current video picture in a sequence of video pictures and to encode the processed block into a coded picture that is part of a coded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) of the example of FIG. 4.

[0077] In an HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8×8 samples. The video encoder (703) determines whether the processing block is best coded using intra-mode, inter-mode, or bi-predictive mode, e.g., using rate-distortion optimization. When the processing block is coded in intra-mode, the video encoder (703) can code the processing block into a coded picture using intra-prediction techniques, and when the processing block is coded in inter-mode or bi-predictive mode, the video encoder (703) can code the processing block into a coded picture using inter-prediction or bi-prediction techniques, respectively. In certain video coding techniques, the merge mode may be an inter-picture prediction sub-mode in which motion vectors are derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown), to determine the mode of the processing block.

[0078] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in FIG.

[0079] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-prediction information (e.g., a description of redundancy information, motion vectors, merge mode information due to inter-coding techniques), and calculate an inter-prediction result (e.g., a prediction block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that is decoded based on the coded video information.

[0080] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), possibly compare the block with already coded blocks in the same picture, generate quantized coefficients after transformation, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). In one example, the intra encoder (722) also calculates intra prediction results (e.g., prediction blocks) based on the intra prediction information and reference blocks in the same picture.

[0081] The generic controller (721) is configured to determine generic control data and control other components of the video encoder (703) based on the generic control data. In one example, the generic controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is an intra mode, the generic controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream, and when the mode is an inter mode, the generic controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0082] The residual calculator (723) is configured to calculate a difference (residual data) between the received block and a prediction result selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) is configured to operate on the residual data to encode the residual data and generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intra-encoder (722) and the inter-encoder (730) as appropriate. For example, the inter-encoder (730) can generate decoded blocks based on the decoded residual data and the inter-prediction information, and the intra-encoder (722) can generate decoded blocks based on the decoded residual data and the intra-prediction information. The decoded blocks are appropriately processed to generate decoded pictures, which may be buffered in a memory circuit (not shown) and used as reference pictures in some examples.

[0083] The entropy encoder (725) is configured to format the bitstream to include the encoded block. The entropy encoder (725) is configured to include various information according to an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. It should be noted that, according to the disclosed subject matter, no residual information is present when coding a block in a merged sub-mode of either the inter-mode or the bi-prediction mode.

[0084] 8 shows a diagram of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) is used in place of the video decoder (410) of the example of FIG. 4.

[0085] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) coupled together as shown in FIG.

[0086] The entropy decoder (871) may be configured to recover from the coded picture certain symbols that represent syntax elements of which the coded picture is composed. Such symbols may include, for example, prediction information (e.g., intra prediction information or inter prediction information) that may identify the mode in which the block is coded (e.g., intra mode, inter mode, bi-predictive mode, merged sub-mode or the latter two of another sub-mode), certain samples or metadata used for prediction by the intra decoder (872) or the inter decoder (880), respectively, such as residual information in the form of quantized transform coefficients. In one example, when the prediction mode is an inter mode or a bi-predictive mode, the inter prediction information is provided to the inter decoder (880), and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information may undergo inverse quantization and is provided to the residual decoder (873).

[0087] The inter decoder (880) is configured to receive the inter prediction information and to generate inter prediction results based on the inter prediction information.

[0088] The intra decoder (872) is configured to receive the intra prediction information and to generate a prediction result based on the intra prediction information.

[0089] The residual decoder (873) is configured to perform inverse quantization to extract inverse quantized transform coefficients, and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantizer parameters (QP)), which may be provided by the entropy decoder (871) (this may be only a small amount of control information, so the data path is not depicted).

[0090] The reconstruction module (874) combines, in the spatial domain, the residual output by the residual decoder (873) with the prediction result (possibly output by an inter-prediction module or an intra-prediction module) to form a reconstructed block that may be part of a reconstructed picture, which in turn may be part of a reconstructed video. It should be noted that other suitable operations, such as a deblocking operation, may be performed to improve appearance.

[0091] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technique. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0092] Aspects of this disclosure provide filtering techniques for video coding / decoding.

[0093] An adaptive loop filter (ALF) with block-based filter adaptation can be applied by the encoder / decoder to reduce artifacts. For the luma component, for example, one of multiple filters (e.g., 25 filters) can be selected for a 4x4 luma block based on local gradient direction and activity.

[0094] The ALF may have any suitable shape and size. Referring to FIG. 9, the ALFs (910)-(911) have diamond shapes, such as a 5×5 diamond shape for the ALF (910) and a 7×7 diamond shape for the ALF (911). In the ALF (910), the elements (920)-(932) form a diamond shape and may be used in the filtering process. Seven values ​​(e.g., C0-C6) may be used for the elements (920)-(932). In the ALF (911), the elements (940)-(964) form a diamond shape and may be used in the filtering process. Thirteen values ​​(e.g., C0-C12) may be used for the elements (940)-(964).

[0095] Referring to FIG. 9, in some examples, two ALFs (910)-(911) with diamond filter shapes are used. A 5×5 diamond shaped filter (910) may be applied to the chroma components (e.g., chroma blocks, chroma CB) and a 7×7 diamond shaped filter (911) may be applied to the luma components (e.g., luma blocks, luma CB). Other suitable shapes and sizes may be used in the ALFs. For example, a 9×9 diamond shaped filter may be used.

[0096] The filter coefficients at the positions indicated by the values ​​(e.g., C0-C6 in (910) or C0-C12 in (920)) may be non-zero. Furthermore, if the ALF includes a clipping function, the clipping values ​​at those positions may be non-zero.

[0097] For block classification of the luminance component, a 4×4 block (or luminance block, luminance CB) can be categorized or classified as one of multiple (e.g., 25) classes. The classification index C is calculated using Equation (1) by quantizing the directional parameter D and the activity value A.

number

number

number

number

[0098] To reduce the complexity of the block classification described above, a subsampled 1-D Laplacian calculation can be applied. Figures 10A-10D show the gradients g in the vertical direction (Figure 10A), horizontal direction (Figure 10B), and two diagonal directions d1 (Figure 10C) and d2 (Figure 10D), respectively. v , g h , g d1 , and g d2 10A shows an example of the subsampled positions used to compute the vertical gradient g. The same subsampled positions can be used for gradient computations in different directions. In FIG. 10A, the label “V” denotes the vertical gradient g.v In FIG. 10B, the label “H” indicates the subsampled positions for computing the horizontal gradient g h In FIG. 10C, the label “D1” indicates the subsampled positions for computing the diagonal gradient g d1 In FIG. 10D, the label “D2” indicates the subsampled positions for computing the diagonal gradient g d2 indicates the subsampled positions for computing

[0099] Horizontal gradient g h and the vertical gradient g v Maximum of

number

number

number

number

number

number

number

number

number

number

number

[0100] The activity value A is

number

number

[0101] For chroma components in a picture, no block classification is applied and therefore a single set of ALF coefficients can be applied per chroma component.

[0102] A geometric transformation may be applied to the filter coefficients and the corresponding filter clipping values ​​(also called clipping values). Before filtering a block (e.g., a 4×4 luminance block), for example, the gradient values ​​(e.g., g v , g h , g d1 , and / or g d2Depending on the filter coefficients f(k,l) and corresponding filter clipping values ​​c(k,l), a geometric transformation such as a rotation or a diagonal and vertical flip may be applied to the filter coefficients f(k,l) and corresponding filter clipping values ​​c(k,l). The geometric transformation applied to the filter coefficients f(k,l) and corresponding filter clipping values ​​c(k,l) may be equivalent to applying a geometric transformation to the samples in the area supported by the filter. The geometric transformation may make the different blocks to which the ALF is applied more similar by aligning their respective directionality.

[0103] Three geometric transformations can be performed, including diagonal flip, vertical flip, and rotation, as described by equations (9)-(11), respectively. f D (k,l) = f(l,k), c D (k,l)=c(l,k) Equation (9) f V (k,l)=f(k,Kl-1), c V (k,l)=c(k,Kl-1) Equation (10) f R (k,l)=f(Kl-1,k), c R (k,l)=c(Kl-1,k) Equation (11) where K is the size of the ALF or filter, and 0≦k, l≦K-1 are the coordinates of the coefficients. For example, a filter f or a clipping value matrix (or clipping matrix) c has position (0,0) at the top left corner and position (K-1,K-1) at the bottom right corner. Transforms can be applied to the filter coefficients f(k,l) and clipping values ​​c(k,l) depending on the gradient values ​​calculated for the block. An example of the relationship between the transforms and the four gradients is summarized in Table 1.

[0104] [Table 1]

[0105] In some embodiments, the ALF filter parameters are signaled within an adaptive parameter set (APS) for a picture. In the APS, one or more sets (e.g., up to 25 sets) of luma filter coefficients and clipping value indexes can be signaled. In one example, a set of the one or more sets can include luma filter coefficients and one or more clipping value indexes. One or more sets (e.g., up to 8 sets) of chroma filter coefficients and clipping value indexes can be signaled. To reduce signaling overhead, filter coefficients of different classifications (e.g., having different classification indexes) for the luma component can be merged. In the slice header, an index of the APS used for the current slice can be signaled.

[0106] In one embodiment, a clipping value index (also called a clipping index) can be decoded from the APS. The clipping value index can be used to determine a corresponding clipping value, for example, based on a relationship between the clipping value index and the corresponding clipping value. The relationship can be predefined and stored in the decoder. In one example, the relationship is described by a table, such as a luma table (e.g., used for luma CB) of clipping value index and corresponding clipping value, a saturation table (e.g., used for saturation CB) of clipping value index and corresponding clipping value. The clipping value may depend on the bit depth B. The bit depth B may refer to the internal bit depth, the bit depth of the reconstructed sample in the CB to be filtered, etc. In some examples, the table (e.g., luma table, saturation table) is obtained using Equation (12).

number

[0107] [Table 2]

[0108] In the slice header for the current slice, one or more APS indices (e.g., up to 7 APS indices) may be signaled to specify the luma filter sets that may be used for the current slice. The filtering process may be controlled at one or more appropriate levels, such as the picture level, slice level, CTB level, etc. In one embodiment, the filtering process may be further controlled at the CTB level. A flag may be signaled to indicate whether the ALF is applied to the luma CTB. The luma CTB may select a filter set from among multiple fixed filter sets (e.g., 16 fixed filter sets) and a filter set (also referred to as a signaled filter set) signaled in the APS. A filter set index may be signaled to the luma CTB to indicate the filter set to be applied (e.g., a filter set among the multiple fixed filter sets and the signaled filter set). The multiple fixed filter sets may be predefined and hard-coded in the encoder and decoder, and may be referred to as a predefined filter set.

[0109] For the chroma component, an APS index can be signaled in the slice header to indicate the chroma filter set currently used for the slice. At the CTB level, if more than one chroma filter set exists in an APS, a filter set index can be signaled for each chroma CTB.

[0110] The filter coefficients may be quantized with a norm equal to 128. To reduce the multiplication complexity, bitstream conformance may be applied such that the coefficient values ​​of non-center locations may be within the range of -27 to 27-1, inclusive. In one example, center location coefficients are not signaled in the bitstream and may be assumed to be equal to 128.

[0111] In some embodiments, the syntax and semantics of clipping indexes and clipping values ​​are defined as follows: alf_luma_clip_idx[sfIdx][j] may be used to specify a clipping index of the clipping value to use before multiplying the jth coefficient of the signaled luma filter indicated by sfIdx. Bitstream conformance requirements may include that the value of alf_luma_clip_idx[sfIdx][j], where sfIdx=0 to alf_luma_num_filters_signalled_minus1 and j=0 to 11, should be in the range of 0 to 3, inclusive. The luma filter clipping value AlfClipL[adaptation_parameter_set_id] with element AlfClipL[adaptation_parameter_set_id][filtIdx][j], where filtIdx = 0 to NumAlfFilters-1 and j = 0 to 11, may be derived as specified in Table 2 depending on bitDepth set equal to BitDepthY and clipIdx set equal to alf_luma_clip_idx[alf_luma_coeff_delta_idx[filtIdx]][j]. alf_chroma_clip_idx[altIdx][j] may be used to specify a clipping index for the clipping value to use before multiplying the jth coefficient of the alternative chroma filter with index altIdx. Bitstream conformance requirements may include that the value of alf_chroma_clip_idx[altIdx][j], where altIdx=0 to alf_chroma_num_alt_filters_minus1, j=0 to 5, should be in the range of 0 to 3, inclusive. The chroma filter clipping value AlfClipC[adaptation_parameter_set_id][altIdx] with element AlfClipC[adaptation_parameter_set_id][altIdx][j], where altIdx=0 to alf_chroma_num_alt_filters_minus1, j=0 to 5, may be derived as specified in Table 2 depending on bitDepth set equal to BitDepthC and clipIdx set equal to alf_chroma_clip_idx[altIdx][j].

[0112] In one embodiment, the filtering process can be described as follows: At the decoder side, when ALF is enabled for CTB, the sample R(i,j) in the CU (or CB) can be filtered, and the filtered sample value R'(i,j) is obtained as shown below using equation (13). In one example, each sample in the CU is filtered.

number

[0113] In a non-linear ALF, multiple sets of clipping values ​​can be provided in Table 3. In one example, the luma set includes four clipping values ​​{1024, 181, 32, 6} and the chroma set includes four clipping values ​​{1024, 161, 25, 4}. The four clipping values ​​in the luma set can be selected by approximately equally dividing the full range of sample values ​​(encoded with 10 bits) for the luma block (e.g., 1024) in the logarithmic domain. The range can be from 4 to 1024 for the chroma set.

[0114] [Table 3]

[0115] The selected clipping value may be coded within the "alf_data" syntax element as follows: An appropriate coding scheme (e.g., Golomb coding scheme) may be used to code the clipping index corresponding to the selected clipping value as shown in Table 3. The coding scheme may be the same coding scheme used to code the filter set index.

[0116] In one embodiment, a virtual boundary filtering process can be used to reduce the line buffer requirements of the ALF. Thus, modified block classification and filtering can be employed for samples near a CTU boundary (e.g., a horizontal CTU boundary). The virtual boundary (1130) is defined as the horizontal CTU boundary (1120) as "N samples " can be defined as a line by shifting the sample by N samples can be a positive integer. In one example, N samples is equal to 4 for the luminance component, and N samples is equal to 2 for the chroma component.

[0117] Referring to Figure 11A, the modified block classification can be applied to the luma component. In one example, in a 1D Laplacian gradient calculation of a 4x4 block (1110) above a virtual boundary (1130), only samples above the virtual boundary (1130) are used. Similarly, referring to Figure 11B, in a 1D Laplacian gradient calculation of a 4x4 block (1111) below a virtual boundary (1131) shifted from the CTU boundary (1121), only samples below the virtual boundary (1131) are used. The quantization of the activity value A can be scaled accordingly by taking into account the reduction in the number of samples used in the 1D Laplacian gradient calculation.

[0118] For the filtering process, symmetric padding operation at the virtual boundary can be used for both luma and chroma components. Figures 12A-12F show an example of such modified ALF filtering for luma components at the virtual boundary. When the sample being filtered is located below the virtual boundary, the adjacent samples located above the virtual boundary can be padded. When the sample being filtered is located above the virtual boundary, the adjacent samples located below the virtual boundary can be padded. With reference to Figure 12A, the adjacent sample C0 can be padded with the sample C2 located below the virtual boundary (1210). With reference to Figure 12B, the adjacent sample C0 can be padded with the sample C2 located above the virtual boundary (1220). With reference to Figure 12C, the adjacent samples C1-C3 can be padded with the samples C5-C7 located below the virtual boundary (1230), respectively. With reference to Figure 12D, the adjacent samples C1-C3 can be padded with the samples C5-C7 located above the virtual boundary (1240), respectively. Referring to Figure 12E, adjacent samples C4-C8 can be padded with samples C10, C11, C12, C11, and C10, respectively, located below the virtual boundary (1250). Referring to Figure 12F, adjacent samples C4-C8 can be padded with samples C10, C11, C12, C11, and C10, respectively, located above the virtual boundary (1260).

[0119] In some instances, the above description may be appropriately adapted when a sample and an adjacent sample are located to the left (or right) and right (or left) of a virtual boundary.

[0120] According to aspects of the present disclosure, a picture can be partitioned based on a filtering process to improve coding efficiency. In some examples, a CTU is also referred to as a largest coding unit (LCU). In one example, a CTU or an LCU can have a size of 64×64 pixels. In some embodiments, an LCU-aligned picture quadtree partition can be used for filtering-based partitioning. In some examples, a coding unit-synchronized picture quadtree-based adaptive loop filter can be used. For example, a luma picture can be divided into several multi-level quadtree partitions, and the boundaries of each partition are aligned to the boundaries of an LCU. Each partition has its own filtering process and may therefore be referred to as a filter unit (FU).

[0121] In some examples, a two-pass encoding flow may be used. In the first pass of the two-pass encoding flow, a quadtree partitioning pattern of the picture and a best filter for each FU may be determined. In some embodiments, the determination of the quadtree partitioning pattern of the picture and the determination of the best filter for the FU are based on the filtering distortion. The filtering distortion may be estimated by a fast filtering distortion estimation (FFDE) technique during the determination process. The picture is partitioned using a quadtree partition. The reconstructed picture may be filtered according to the determined quadtree partitioning pattern and the selected filters of all the FUs.

[0122] In the second pass of the two-pass encoding flow, a CU-synchronous ALF on / off control is performed. According to the ALF on / off result, the first filtered picture is partially restored by the reconstructed picture.

[0123] Specifically, in some examples, a top-down partitioning procedure is adopted to partition an image into multi-level quadtree partitions using a rate-distortion criterion. Each partition is called a filter unit (FU). The partitioning process aligns the quadtree partitions to the boundaries of LCUs. The coding order of the FUs follows the z-scan order.

[0124] Figure 13 illustrates an example partition according to some embodiments of the present disclosure. In the example of Figure 13, a picture (1300) is partitioned into 10 FUs, with the coding order being FU0, FU1, FU2, FU3, FU4, FU5, FU6, FU7, FU8, and FU9.

[0125] Figure 14 shows a quadtree partitioning pattern (1400) for a picture (1300). In the example of Figure 14, a partition flag is used to indicate the partition pattern of the picture. For example, a "1" indicates that quadtree partitioning is performed on the block, and a "0" indicates that no further partitioning is performed. In some examples, the smallest size FU has an LCU size, and no partition flag is required for the smallest size FU. The partition flag is coded and transmitted in z-order as shown in Figure 14.

[0126] In some examples, the filters for each FU are selected from two filter sets based on a rate-distortion criterion. The first set has 1 / 2 symmetric square and diamond filters derived for the current FU. The second set comes from a time-delay filter buffer, which stores filters previously derived for the FUs of the previous picture. The filter with the smallest rate-distortion cost of these two sets can be selected for the current FU. Similarly, if the current FU is not the smallest FU and can be further divided into four child FUs, the rate-distortion costs of the four child FUs are calculated. By recursively comparing the rate-distortion costs for the division and no division cases, a picture quadtree division pattern can be determined.

[0127] In some examples, a maximum quadtree division level may be used to limit the maximum number of FUs. In one example, when the maximum quadtree division level is 2, the maximum number of FUs is 16. Furthermore, during quadtree division decision, correlation values ​​for deriving Wiener coefficients of the 16 FUs at the lowest quadtree level (the smallest FUs) can be reused. The remaining FUs can derive their Wiener filters from the correlations of the 16 FUs at the lowest quadtree level. Thus, in that example, only one frame buffer access is performed to derive the filter coefficients of all FUs.

[0128] After the quadtree division pattern is determined, CU-synchronous ALF on / off control can be performed to further reduce the filtering distortion. By comparing the filtering distortion and the non-filtering distortion in each leaf CU, the leaf CU can explicitly switch ALF on / off in its local region. In some examples, the coding efficiency can be further improved by redesigning the filter coefficients according to the ALF on / off results.

[0129] The inter-component filtering process can apply an inter-component filter, such as an inter-component adaptive loop filter (CC-ALF). The inter-component filter can use the luma sample value of the luma component (e.g., luma CB) to refine the chroma component (e.g., chroma CB corresponding to luma CB). In one example, the luma CB and chroma CB are included in a CU.

[0130] FIG. 15 illustrates an inter-component filter (e.g., CC-ALF) used to generate a chroma component according to one embodiment of the present disclosure. In some examples, FIG. 15 illustrates a filtering process for a first chroma component (e.g., first chroma CB), a second chroma component (e.g., second chroma CB), and a luma component (e.g., luma CB). The luma component can be filtered by a sample adaptive offset (SAO) filter (1510) to generate an SAO filtered luma component (1541). The SAO filtered luma component (1541) can be further filtered by an ALF luma filter (1516) to become a filtered luma CB (1561) (e.g., “Y”).

[0131] The first chroma component may be filtered by the SAO filter (1512) and the ALF chroma filter (1518) to generate a first intermediate component (1552). Furthermore, the SAO filtered luma component (1541) may be filtered by an inter-component filter (e.g., CC-ALF) (1521) for the first chroma component to generate a second intermediate component (1542). A filtered first chroma component (1562) (e.g., “Cb”) may then be generated based on at least one of the second intermediate component (1542) and the first intermediate component (1552). In one example, the filtered first chroma component (1562) (e.g., “Cb”) may be generated by combining the second intermediate component (1542) and the first intermediate component (1552) with an adder (1522). The inter-component adaptive loop filtering process for the first chroma component may include steps performed by a CC-ALF (1521) and steps performed, for example, by an adder (1522).

[0132] The above description can be adapted to the second chroma component. The second chroma component can be filtered by the SAO filter (1514) and the ALF chroma filter (1518) to generate a third intermediate component (1553). Furthermore, the SAO filtered luma component (1541) can be filtered by an inter-component filter (e.g., CC-ALF) (1531) for the second chroma component to generate a fourth intermediate component (1543). Then, a filtered second chroma component (1563) (e.g., “Cr”) can be generated based on at least one of the fourth intermediate component (1543) and the third intermediate component (1553). In one example, the filtered second chroma component (1563) (e.g., “Cr”) can be generated by combining the fourth intermediate component (1543) and the third intermediate component (1553) with an adder (1532). In one example, the inter-component adaptive loop filtering process for the second chroma component may include steps performed by a CC-ALF (1531) and steps performed, for example, by an adder (1532).

[0133] The inter-component filters (e.g., CC-ALF(1521), CC-ALF(1531)) may operate by applying a linear filter having any suitable filter shape to the luma component (or luma channel) in order to refine each chroma component (e.g., first chroma component, second chroma component).

[0134] FIG. 16 illustrates an example of a filter (1600) according to an embodiment of the present disclosure. The filter (1600) may include non-zero filter coefficients and zero filter coefficients. The filter (1600) has a diamond shape (1620) formed by the filter coefficients (1610) (shown as solid circles). In one example, the non-zero filter coefficients in the filter (1600) are included in the filter coefficients (1610), and the filter coefficients that are not included in the filter coefficients (1610) are zero. Thus, the non-zero filter coefficients in the filter (1600) are included in the diamond shape (1620), and the filter coefficients that are not included in the diamond shape (1620) are zero. In one example, the number of filter coefficients of the filter (1600) is equal to the number of filter coefficients (1610), which is 18 in the example shown in FIG. 16.

[0135] The CC-ALF may include any suitable filter coefficients (also referred to as CC-ALF filter coefficients). Referring back to Figure 15, the CC-ALF (1521) and the CC-ALF (1531) may have the same filter shape, such as the diamond shape (1620) shown in Figure 16, and the same number of filter coefficients. In one example, the values ​​of the filter coefficients in the CC-ALF (1521) are different from the values ​​of the filter coefficients in the CC-ALF (1531).

[0136] In general, filter coefficients in CC-ALF (e.g., non-zero filter coefficients) can be transmitted, for example, in APS. In one example, the filter coefficients are multiplied by a factor (e.g., 2 10) and may be rounded for fixed-point representation. Application of CC-ALF is controlled for variable block sizes and may be signaled by a context coding flag (e.g., a CC-ALF enable flag) received for each block of samples. Context coding flags such as the CC-ALF enable flag may be signaled at any appropriate level, such as at the block level. Block sizes along with the CC-ALF enable flag may be received at the slice level for each chroma component. In some examples, block sizes (in chroma samples) of 16×16, 32×32, and 64×64 may be supported.

[0137] Figure 17 shows an example syntax for CC-ALF, according to some embodiments of the present disclosure. In the example of Figure 17, alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is an index indicating whether an inter-component Cb filter is used, and if so, the index of the inter-component Cb filter. For example, when alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is equal to 0, then no cross-component Cb filter is applied to the block of Cb color component samples at luma location (xCtb, yCtb), and when alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is not equal to 0, then alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is the index for the filter to be applied. For example, the alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]th cross-component Cb filter is applied to the block of Cb color component samples at luma position (xCtb, yCtb).

[0138] Further, in the example of FIG. 17, alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is used to indicate whether an inter-component Cr filter is used, and if so, is the index of the inter-component Cr filter. For example, when alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is equal to 0, then no inter-component Cr filter is applied to the block of Cr color component samples at luma position (xCtb, yCtb), and when alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is not equal to 0, then alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is the index of the inter-component Cr filter. For example, the alf_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]th inter-component Cr filter may be applied to a block of Cr color component samples at luma location (xCtb, yCtb).

[0139] In some examples, a chroma subsampling technique is used, so that the number of samples in each of the chroma blocks can be less than the number of samples in the luma block. The chroma subsampling format (also referred to as a chroma subsampling format and specified by, for example, chroma_format_idc) can indicate a chroma horizontal subsampling factor (e.g., SubWidthC) and a chroma vertical subsampling factor (e.g., SubHeightC) between each of the chroma blocks and the corresponding luma block. In one example, the chroma subsampling format is 4:2:0, so that the chroma horizontal subsampling factor (e.g., SubWidthC) and the chroma vertical subsampling factor (e.g., SubHeightC) are 2, as shown in Figures 18A-18B. In one example, the chroma subsampling format is 4:2:2, so that the chroma horizontal subsampling factor (e.g., SubWidthC) is 2 and the chroma vertical subsampling factor (e.g., SubHeightC) is 1. In one example, the chroma subsampling format is 4:4:4, and therefore the chroma horizontal subsampling factor (e.g., SubWidthC) and the chroma vertical subsampling factor (e.g., SubHeightC) are 1. The chroma sample type (also called the chroma sample position) may indicate a relative position of a chroma sample in a chroma block with respect to at least one corresponding luma sample in a luma block.

[0140] 18A-18B illustrate example locations of chroma samples relative to luma samples, according to an embodiment of the present disclosure. With reference to FIG. 18A, luma samples (1801) are located in rows (1811)-(1818). The luma samples (1801) illustrated in FIG. 18A may represent a portion of a picture. In one example, a luma block (e.g., Luma CB) includes a luma sample (1801). The luma block may correspond to two chroma blocks having a chroma subsampling format of 4:2:0. In one example, each chroma block includes a chroma sample (1803). Each chroma sample (e.g., chroma sample (1803(1))) corresponds to four luma samples (e.g., luma samples (1801(1)) through (1801(4)). In one example, the four luma samples are the top left sample (1801(1)), the top right sample (1801(2)), the bottom left sample (1801(3)), and the bottom right sample (1801(4)). A chroma sample (e.g., (1803(1))) corresponds to the top left sample (1801(1)) and the bottom left sample (1801(4)). The chroma sample type of a chroma block having chroma sample (1803) located at a left center position between the top left sample (1801(1)) and the bottom left sample (1801(3)) may be referred to as chroma sample type 0. Chroma sample type 0 indicates a relative position 0, which corresponds to a left center position halfway between the top left sample (1801(1)) and the bottom left sample (1801(3)). The four luma samples (e.g., (1801(1)) through (1801(4))) may be referred to as neighboring luma samples of chroma sample (1803)(1).

[0141] In one example, each chroma block includes a chroma sample (1804). The above description referring to the chroma sample (1803) may be adapted to the chroma sample (1804), and thus detailed description may be omitted for brevity. Each of the chroma samples (1804) may be located at a central position of the four corresponding luma samples, and the chroma sample type of the chroma block having the chroma sample (1804) may be referred to as chroma sample type 1. The chroma sample type 1 indicates a relative position 1, which corresponds to the central position of the four luma samples (e.g., (1801(1)) to (1801(4))). For example, one of the chroma samples (1804) may be located at the central portion of the luma samples (1801(1)) to (1801(4)).

[0142] In one example, each chroma block includes a chroma sample (1805). Each of the chroma samples (1805) may be located at a top left position that is the same as the top left sample of the four corresponding luma samples (1801), and the chroma sample type of the chroma block having the chroma sample (1805) may be referred to as a chroma sample type 2. Thus, each of the chroma samples (1805) is located at the same position as the top left sample of the four luma samples (1801) that correspond to the respective chroma sample. The chroma sample type 2 indicates a relative position 2 that corresponds to the top left position of the four luma samples (1801). For example, one of the chroma samples (1805) may be located at the top left position of the luma samples (1801(1))-(1801(4)).

[0143] In one example, each chroma block includes a chroma sample (1806). Each of the chroma samples (1806) may be located at an upper center position between a corresponding upper left sample and a corresponding upper right sample, and a chroma sample type of a chroma block having the chroma sample (1806) may be referred to as a chroma sample type 3. The chroma sample type 3 indicates a relative position 3, which corresponds to an upper center position between the upper left sample and the upper right sample. For example, one of the chroma samples (1806) may be located at an upper center position of the luma samples (1801(1))-(1801(4)).

[0144] In one example, each chroma block includes a chroma sample (1807). Each of the chroma samples (1807) may be located at a bottom left position that is co-located with the bottom left sample of the four corresponding luma samples (1801), and the chroma sample type of the chroma block having the chroma sample (1807) may be referred to as chroma sample type 4. Thus, each of the chroma samples (1807) is located at the bottom left sample of the four luma samples (1801) that correspond to the respective chroma sample. The chroma sample type 4 indicates a relative position 4, which corresponds to the bottom left position of the four luma samples (1801). For example, one of the chroma samples (1807) may be located at the bottom left position of the luma samples (1801(1))-(1801(4)).

[0145] In one example, each chroma block includes a chroma sample (1808). Each of the chroma samples (1808) is located at a lower center position between a lower left sample and a lower right sample, and the chroma sample type of the chroma block having the chroma sample (1808) can be referred to as a chroma sample type 5. The chroma sample type 5 indicates a relative position 5, which corresponds to a lower center position between a lower left sample and a lower right sample of the four luma samples (1801). For example, one of the chroma samples (1808) can be located between the lower left sample and the lower right sample of the luma samples (1801(1))-(1801(4)).

[0146] In general, any suitable chroma sample type may be used for the chroma subsampling format. Chroma sample types 0-5 are exemplary chroma sample types described for chroma subsampling format 4:2:0. Additional chroma sample types may be used for chroma subsampling format 4:2:0. Additionally, other chroma sample types and / or variations of chroma sample types 0-5 may be used for other chroma subsampling formats, such as 4:2:2, 4:4:4, etc. In one example, a chroma sample type that combines chroma samples (1805) and (1807) is used for chroma subsampling format 4:2:2.

[0147] In one example, a luma block is considered to have alternating rows such as rows (1811)-(1812) each including the top two samples (e.g., (1801(1))-(1801(2))) of the four luma samples (e.g., (1801(1))-(1801(4))) and the bottom two samples (e.g., (1801(3))-(1801(4))) of the four luma samples (e.g., (1801(1))-(1801(4))). Thus, rows (1811), (1813), (1815), and (181 Rows (1801(1)) through (1801(4)) are located in the current row (e.g., (1811)) and the next row (e.g., (1812)). Relative positions 2 through 3 are located in the current row, relative positions 0 through 1 are located between each current row and the respective next row, and relative positions 4 through 5 are located in the next row.

[0148] Chroma samples (1803), (1804), (1805), (1806), (1807), or (1808) are located in rows (1851)-(1854) within each chroma block. The specific location of rows (1851)-(1854) may depend on the chroma sample's chroma sample type. For example, for chroma samples (1803)-(1804) with respective chroma sample types 0-1, row (1851) is located between rows (1811)-(1812). For chroma samples (1805)-(1806) with respective chroma sample types 2-3, row (1851) is in the same position as the current row (1811). For chroma samples (1807)-(1808) having respective chroma sample types 4-5, row (1851) is in the same position as the next row (1812). The above description can be appropriately adapted for rows (1852)-(1854), and a detailed description will be omitted for the sake of brevity.

[0149] Any suitable scanning method may be used to display, store, and / or transmit the luma blocks and corresponding chroma blocks described above in Figure 18A. In one example, progressive scanning is used.

[0150] Interlace scanning can be used, as shown in Figure 18B. As described above, the chroma subsampling format is 4:2:0 (e.g., chroma_format_idc equals 1). In one example, the variable chroma location type (e.g., ChromaLocType) indicates either the current row (e.g., ChromaLocType is chroma_sample_loc_type_top_field) or the next row (e.g., ChromaLocType is chroma_sample_loc_type_bottom_field). The current rows (1811), (1813), (1815), and (1817) and the next rows (1812), (1814), (1816), and (1818) may be scanned separately, for example, the current rows (1811), (1813), (1815), and (1817) may be scanned first, followed by the next rows (1812), (1814), (1816), and (1818). The current row may include luminance samples (1801) and the next row may include luminance samples (1802).

[0151] Similarly, corresponding chroma blocks may be interlaced. Rows (1851) and (1853) containing chroma samples (1803), (1804), (1805), (1806), (1807), or (1808) with no fill may be referred to as the current row (or current chroma row), and rows (1852) and (1854) containing chroma samples (1803), (1804), (1805), (1806), (1807), or (1808) with gray fill may be referred to as the next row (or next chroma row). In one example, during interlacing, rows (1851) and (1853) are scanned first, followed by rows (1852) and (1854).

[0152] In some examples, a constrained directional enhancement filtering technique can be used. The use of an in-loop constrained directional enhancement filter (CDEF) can remove coding artifacts while preserving image details. In one example (e.g., HEVC), a sample adaptive offset (SAO) algorithm can achieve a similar objective by defining signal offsets for different classes of pixels. Unlike SAO, CDEF is a nonlinear spatial filter. In some examples, CDEF can be constrained to be easily vectorizable (i.e., implementable with single instruction multiple data (SIMD) operations). Note that other nonlinear filters such as median filters, bilateral filters, etc. cannot be treated similarly.

[0153] In some cases, the amount of ringing artifacts in a coded image tends to be roughly proportional to the quantization step size. Although the amount of detail is a property of the input image, the smallest detail retained in the quantized image also tends to be proportional to the quantization step size. For a given quantization step size, the amplitude of the ringing is generally smaller than the amplitude of the detail.

[0154] CDEF can be used to identify the orientation of each block and then adaptively filter along the identified orientation and to a lesser extent along orientations rotated 45 degrees from the identified orientation. In some examples, the encoder can look up the filter strength, or the filter strength can be signaled explicitly, allowing a high degree of control over the blurring.

[0155] Specifically, in some examples, the direction search is performed immediately after the deblocking filter on the restored pixels. The direction can be searched by the decoder since those pixels are available to the decoder, and thus, in one example, the direction does not require signaling. In some examples, the direction search can operate on a specific block size, such as an 8×8 block, which is small enough to properly handle non-linear edges, but large enough to reliably estimate the direction when applied to the quantized image. Also, having a constant directionality on the 8×8 region makes it easier to vectorize the filter. In some examples, each block (e.g., 8×8) can be compared to a fully directional block to determine the difference. A fully directional block is a block in which all pixels along a line of a direction have the same value. In one example, a difference measure of each of the block and the fully directional block, such as sum of squared differences (SSD), root mean square (RMS) error, etc., can be calculated. The fully directional block with the smallest difference (e.g., smallest SSD, smallest RMS, etc.) can then be determined, and the direction of the determined fully directional block can be the direction that best matches the pattern in the block.

[0156] FIG. 19 illustrates an example of a direction search according to an embodiment of the present disclosure. In one example, block (1910) is an 8×8 block that has been restored and output from a deblocking filter. In the example of FIG. 19, the direction search can determine a direction from eight directions indicated by (1920) for block (1910). Eight fully directional blocks (1930) are formed corresponding to the eight directions (1920), respectively. The fully directional blocks corresponding to a direction are blocks whose pixels along the line of that direction have the same value. Furthermore, difference measures such as SSD, RMS error, etc., of each of block (1910) and fully directional block (1930) can be calculated. In the example of FIG. 19, the RMS error is indicated by (1940). As indicated by (1943), the RMS error of block (1910) and fully directional block (1933) is the smallest, and thus direction (1923) is the direction that best matches the pattern in block (1910).

[0157] After the direction of the block is identified, a nonlinear low-pass directional filter can be determined. For example, the filter taps of the nonlinear low-pass directional filter can be aligned along the identified direction to reduce ringing while preserving the directional edge or directional pattern. However, in some examples, directional filtering alone may not be able to sufficiently reduce ringing. In one example, additional filter taps are also used for pixels that are not along the identified direction. The extra filter taps are treated more conservatively to reduce the risk of blurring. For this reason, the CDEF includes first and second order filter taps. In one example, a complete 2-D CDEF filter is given by Equation (14):

number

[0158] In some examples, in addition to the deblocking operation, an in-loop restoration scheme is used in the video coding after deblocking, generally to remove noise and mark and improve edge quality. In one example, the in-loop restoration scheme is switchable within a frame for tiles of appropriate size. The in-loop restoration scheme is based on a separable symmetric Wiener filter, a dual self-guided filter with subspace projection, and a domain transform recursive filter. Since content statistics can change significantly within a frame, the in-loop restoration scheme is integrated within a switchable framework that can trigger different schemes within different regions of a frame.

[0159] A separable symmetric Wiener filter can be one of the in-loop restoration schemes. In some examples, every pixel in the degraded frame can be restored as a non-causal filtered version of the pixels in a w × w window around it, where w = 2r + 1 is odd with respect to the integer r. The 2D filter taps are expressed as w in column vectorized form. 2 If the vector F is denoted by a 1 × 1 element vector, then the straightforward LMMSE optimization is F = H -1 M, where H=E[XX T ] is the autocovariance of x and w in a w × w window around the pixel 2 is a column vectorized version of the samples, where M=E[YX T] is the cross-correlation of x and the scalar source sample y to be estimated. In one example, the encoder can estimate H and M from realizations in the deblocked frame and the source, and send the resulting filter F to the decoder. However, doing so would result in w 2 Not only does it incur a significant bit rate cost in transmitting the taps, but non-separable filtering makes decoding prohibitively complex. In some embodiments, some additional constraints are placed on the nature of F. The first constraint constrains F to be separable, so that the filtering can be implemented as a separable horizontal and vertical w-tap convolution. The second constraint constrains each of the horizontal and vertical filters to be symmetric. The third constraint assumes that both the horizontal and vertical filter coefficients sum to one.

[0160] Dual self-guided filtering with subspace projection can be one of the in-loop restoration schemes. Guided filtering is an image filtering technique in which a locally linear model, shown by equation (15), is used to compute the filtered output y from the unfiltered samples x, where y=Fx+G Equation (15) Here, F and G are determined based on the statistics of the degraded image and the guidance image in the neighborhood of the filtered pixel. If the guide image is the same as the degraded image, the obtained so-called self-guided filtering has the effect of edge-preserving smoothing. In one example, a specific form of self-guided filtering can be used. The specific form of self-guided filtering depends on two parameters: radius r and noise parameter e, and is listed as the following steps: 1. The mean μ and variance σ of pixels in a (2r+1)×(2r+1) window around every pixel 2 This step can be efficiently implemented with box filtering based on integral imaging. 2. f = σ for every pixel2 / (σ 2 +e), and calculate g = (1 - f)μ. 3. Calculate F and G for every pixel as the average of the f and g values ​​in a 3x3 window around the pixel being used.

[0161] The specific form of self-guided filtering is controlled by r and e, with larger r resulting in larger spatial variance and larger e resulting in larger range variance.

[0162] Figure 20 shows an example showing subspace projection in some cases. As shown in Figure 20, even if neither of the reconstructions X1, X2 are close to the source Y, a suitable multiplier {α, β} can make them quite close to the source as long as they are somewhat moved in the right direction.

[0163] In some examples (e.g., HEVC), a filtering technique called sample adaptive offset (SAO) can be used. In some examples, SAO is applied to the reconstructed signal after the deblocking filter. SAO can use an offset value given in the slice header. In some examples, for luma samples, the encoder can determine whether to apply (enable) SAO to the slice. When SAO is enabled, the current picture allows a recursive division of the coding unit into four sub-regions, and each sub-region can select an SAO type from multiple SAO types based on features within the sub-region.

[0164] FIG. 21 illustrates a table (2100) of multiple SAO types according to one embodiment of the present disclosure. In the table (2100), SAO types 0 to 6 are illustrated. Note that SAO type 0 is used to indicate that no SAO is applied. Furthermore, each SAO type, SAO type 1 to SAO type 6, includes multiple categories. SAO can reduce distortion by classifying the restored pixels of a sub-region into categories and adding an offset to the pixels of each category in the sub-region. In some examples, edge characteristics can be used for pixel classification in SAO types 1 to 4, and pixel intensity can be used for pixel classification in SAO types 5 to 6.

[0165] Specifically, in one embodiment, such as SAO types 5-6, a band offset (BO) may be used to classify all pixels of a sub-region into multiple bands. Each band of the multiple bands includes pixels in the same intensity interval. In some examples, the intensity range is divided equally into multiple intervals, such as 32 intervals from 0 to a maximum intensity value (e.g., 255 for 8-bit pixels), and each interval is associated with an offset. Further, in one example, the 32 bands are divided into two groups, such as a first group and a second group. The first group includes the middle 16 bands (e.g., 16 intervals in the middle of the intensity range), and the second group includes the remaining 16 bands (e.g., 8 intervals on the low side of the intensity range and 8 intervals on the high side of the intensity range). In one example, only one offset of the two groups is transmitted. In some embodiments, when pixel classification operations in BO are used, the 5 most significant bits of each pixel may be directly used as a band index.

[0166] Additionally, in one embodiment, such as SAO types 1-4, edge offset (EO) can be used in determining pixel classification and offset. For example, pixel classification can be determined based on a one-dimensional three-pixel pattern taking into account edge directionality information.

[0167] FIG. 22 shows an example of a three-pixel pattern for pixel classification at edge offset in some examples. In the example of FIG. 22, a first pattern (2210) (shown by three gray pixels) is called a 0 degree pattern (the 0 degree pattern is associated with a horizontal direction), a second pattern (2220) (shown by three gray pixels) is called a 90 degree pattern (the 90 degree pattern is associated with a vertical direction), a third pattern (2230) (shown by three gray pixels) is called a 135 degree pattern (the 135 degree pattern is associated with a 135 degree diagonal direction), and a fourth pattern (2240) (shown by three gray pixels) is called a 45 degree pattern (the 45 degree pattern is associated with a 45 degree diagonal direction). In one example, one of the four directivity patterns shown in FIG. 22 can be selected taking into account edge directivity information of the sub-region. The selection can be transmitted in one example as side information in the coded video bitstream. The pixels within the sub-region can then be classified into multiple categories by comparing each pixel with its two neighboring pixels in the direction associated with the directional pattern.

[0168] 23 is a table (2300) for pixel classification rules for edge offsets in some examples. Specifically, pixel c (also shown in each pattern of FIG. 22) is compared with two adjacent pixels (also shown in gray in each pattern of FIG. 22), and pixel c can be classified into one of categories 0 to 4 based on the comparison according to the pixel classification rules shown in FIG. 23.

[0169] In some embodiments, the decoder-side SAO can operate independently of the largest coding unit (LCU) (e.g., CTU) so that line buffers can be saved. In some examples, the pixels in the top and bottom rows in each LCU are not SAO processed when the 90 degree, 135 degree, and 45 degree classification patterns are selected, and the pixels in the left and right most columns in each LCU are not SAO processed when the 0 degree, 135 degree, and 45 degree patterns are selected.

[0170] FIG. 24 shows an example of syntax (2400) that may need to be signaled for a CTU when parameters are not merged from neighboring CTUs. For example, syntax element sao_type_idx[cldx][rx][ry] may be signaled to indicate the SAO type of the sub-region. The SAO type may be BO (band offset) or EO (edge ​​offset). When sao_type_idx[cldx][rx][ry] has a value of 0, it indicates that SAO is OFF, values ​​of 1 to 4 indicate that one of the four EO categories corresponding to 0°, 90°, 135°, and 45° is used, and a value of 5 indicates that BO is used. In the example of FIG. 24, each of the BO and EO types has four SAO offset values ​​to be signaled (sao_offset[cIdx][rx][ry][0] to sao_offset[cIdx][rx][ry][3]).

[0171] In general, the filtering process may use a reconstructed sample of a first color component as an input (e.g., Y or Cb or Cr, or R or G or B) to generate an output, and the output of the filtering process is applied to a second color component, which may be the same color component as the first color component or another color component different from the first color component.

[0172] In a related example of inter-component filtering (CCF), filter coefficients are derived based on some mathematical equations. The derived filter coefficients are signaled from the encoder side to the decoder side, and the derived filter coefficients are used to generate an offset using a linear combination. The generated offset is then added to the reconstructed sample as a filtering process. For example, an offset is generated based on a linear combination of the filtering coefficients with the luma sample, and the generated offset is added to the reconstructed chroma sample. A related example of CCF is based on the assumption of a linear mapping relationship between the reconstructed luma sample value and the delta value between the original chroma sample and the reconstructed chroma sample. However, the mapping between the reconstructed luma sample value and the delta value between the original chroma sample and the reconstructed chroma sample does not necessarily follow a linear mapping process, and therefore the coding performance of CCF may be limited under the assumption of a linear mapping relationship.

[0173] In some examples, the nonlinear mapping technique can be used in inter-component filtering and / or same-color component filtering without significant signaling overhead. In one example, the nonlinear mapping technique can be used in inter-component filtering to generate inter-component sample offsets. In another example, the nonlinear mapping technique can be used in same-color component filtering to generate local sample offsets.

[0174] For convenience, a filtering process using a nonlinear mapping technique may be referred to as sample offset with nonlinear mapping (SO-NLM). SO-NLM in inter-component filtering process may be referred to as inter-component sample offset (CCSO). SO-NLM in same-color component filtering may be referred to as local sample offset (LSO). A filter using a nonlinear mapping technique may be referred to as a nonlinear mapping-based filter. The nonlinear mapping-based filter may include a CCSO filter, an LSO filter, etc.

[0175] In one example, CCSO and LSO can be used as loop filtering to reduce distortion of the reconstructed samples. CCSO and LSO do not rely on the assumption of linear mapping used in the related exemplary CCF. For example, CCSO does not rely on the assumption of a linear mapping relationship between luma reconstructed sample values ​​and delta values ​​between original chroma samples and chroma reconstructed samples. Similarly, LSO does not rely on the assumption of a linear mapping relationship between color component reconstructed sample values ​​and delta values ​​between original samples of color components and reconstructed samples of color components.

[0176] In the following description, a SO-NLM filtering process is described that uses a reconstructed sample of a first color component as an input (e.g., Y or Cb or Cr, or R or G or B) to generate an output, and the output of the filtering process is applied to a second color component. When the second color component is the same color component as the first color component, the description is applicable to LSO, and when the second color component is different from the first color component, the description is applicable to CCSO.

[0177] In SO-NLM, a nonlinear mapping is derived on the encoder side. The nonlinear mapping is between the reconstructed sample of the first color component in the filter support region and the offset added to the second color component in the filter support region. When the second color component is the same as the first color component, the nonlinear mapping is used in LSO, and when the second color component is different from the first color component, the nonlinear mapping is used in CCSO. The region of the nonlinear mapping is determined by the different combinations of the processed input reconstructed samples (also called possible reconstructed sample value combinations).

[0178] The SO-NLM technique can be illustrated using a concrete example. In the concrete example, a reconstruction sample from a first color component located in a filter support area (also called a "filter support region") is determined. The filter support area is an area to which a filter can be applied, and the filter support area can have any suitable shape.

[0179] FIG. 25 illustrates an example of a filter support area (2500) according to some embodiments of the present disclosure. The filter support area (2500) includes four reconstruction samples of a first color component: P0, P1, P2, and P3. In the example of FIG. 25, the four reconstruction samples may form a cross in the vertical and horizontal directions, and the center position of the cross is the position for the sample to be filtered. The sample in the center position and of the same color component as P0-P3 is denoted by C. The sample in the center position and of the second color component is denoted by F. The second color component may be the same as the first color component of P0-P3 or may be different from the first color component of P0-P3.

[0180] FIG. 26 illustrates an example of another filter support area (2600) according to some embodiments of the present disclosure. The filter support area (2600) includes four restored samples P0, P1, P2, and P3 of a first color component forming a square. In the example of FIG. 26, the center position of the square is the position of the sample to be filtered. The sample at the center position and of the same color component as P0-P3 is denoted by C. The sample at the center position and of the second color component is denoted by F. The second color component may be the same as the first color component of P0-P3 or may be different from the first color component of P0-P3.

[0181] The reconstructed samples are input to the SO-NLM filter and processed appropriately to form the filter taps. In one example, the positions of the reconstructed samples that are input to the SO-NLM filter are called filter tap positions. In a specific example, the reconstructed samples are processed in two steps:

[0182] In the first step, delta values ​​between P0-P3 and C are calculated, respectively. For example, m0 represents the delta value between P0 and C, m1 represents the delta value between P1 and C, m2 represents the delta value between P2 and C, and m3 represents the delta value between P3 and C.

[0183] In a second step, the delta values ​​m0-m3 are further quantized, and the quantized values ​​are denoted as d0, d1, d2, d3. In one example, the quantized value may be one of -1, 0, 1 based on the quantization process. For example, when m is less than -N (N is a positive value and is called the quantization step size), the value m may be quantized to -1, when m is in the range of [-N,N], the value m may be quantized to 0, and when m is greater than N, the value m may be quantized to 1. In some examples, the quantization step size N may be one of 4, 8, 12, 16, etc.

[0184] In some embodiments, the quantized values ​​d0-d3 are filter taps and can be used to identify one combination in a filter domain. For example, the filter taps d0-d3 can form a combination in a filter domain. Each filter tap can have three quantized values, so when four filter taps are used, the filter domain contains 81 (3×3×3×3) combinations.

[0185] 27A-27C are table (2700) having 81 combinations according to one embodiment of the present disclosure. Table (2700) includes 81 rows corresponding to the 81 combinations. In each row corresponding to a combination, the first column includes an index of the combination, the second column includes a value of filter tap d0 for the combination, the third column includes a value of filter tap d1 for the combination, the fourth column includes a value of filter tap d2 for the combination, the fifth column includes a value of filter tap d3 for the combination, and the sixth column includes an offset value associated with the combination for the nonlinear mapping. In one example, once the filter taps d0-d3 are determined, an offset value (denoted by s) associated with the combination of d0-d3 can be determined according to table (2700). In one example, the offset values ​​s0-s80 are integers such as 0, 1, -1, 3, -3, 5, -5, -7, etc.

[0186] In some embodiments, the final filtering process of the SO-NLM can be applied as shown in equation (16): f'=clip(f+s) Equation (16) where f is the reconstructed sample of the second color component to be filtered, and s is an offset value determined according to the filter taps resulting from processing the reconstructed sample of the first color component, such as using table (2700). The sum of the reconstructed sample F and the offset value s is further clipped to a range associated with the bit depth to determine the final filtered sample f' of the second color component.

[0187] Please note that in the case of LSO, the second color component in the above description is the same as the first color component, and in the case of CCSO, the second color component in the above description may be different from the first color component.

[0188] It should be noted that the above description may be adjusted for other embodiments of the present disclosure.

[0189] In some examples, at the encoder side, the encoding device may derive a mapping between the reconstructed samples of the first color component in the filter support region and the offset added to the reconstructed samples of the second color component. The mapping may be any suitable linear or nonlinear mapping. Then, the filtering process may be applied at the encoder side and / or the decoder side based on the mapping. For example, the mapping may be appropriately notified to the decoder (e.g., the mapping is included in the coded video bitstream transmitted from the encoder side to the decoder side), and then the decoder may perform the filtering process based on the mapping.

[0190] The performance of nonlinear mapping-based filters, such as CCSO filters, LSO filters, etc., depends on the filter shape configuration. The filter shape configuration (also called filter shape) of a filter can refer to the characteristics of the pattern formed by the filter tap positions. The pattern can be defined by various parameters such as the number of filter taps, the geometric shape of the filter tap positions, and the distance of the filter tap positions to the center of the pattern. Using a fixed filter shape configuration can limit the performance of a nonlinear mapping-based filter.

[0191] As illustrated by FIG. 24 and FIG. 25 and FIG. 27A-FIG. 27C, some examples use a 5-tap filter design for filter shape configuration of a nonlinear mapping-based filter. The 5-tap filter design can use tap positions of P0, P1, P2, P3, and C. The 5-tap filter design for filter shape configuration can result in a look-up table (LUT) with 81 entries, as illustrated in FIG. 27A-FIG. 27C. The LUT of sample offsets needs to be signaled from the encoder side to the decoder side, and the signaling of the LUT may contribute to most of the signaling overhead and affect the coding efficiency using the nonlinear mapping-based filter. According to some aspects of the present disclosure, the number of filter taps may be different from 5. In some examples, the number of filter taps can be reduced and more information in the filter support area can be captured, improving the coding efficiency.

[0192] In some examples, the filter shape configurations in a group for nonlinear mapping-based filters each have three filter taps.

[0193] 28 illustrates seven filter shape configurations of three filter taps in one example. Specifically, a first filter shape configuration includes three filter taps at positions labeled "1" and "c", where position "c" is the center position of position "1", a second filter shape configuration includes three filter taps at positions labeled "2" and position "c", where position "c" is the center position of position "2", a third filter shape configuration includes three filter taps at positions labeled "3" and position "c", where position "c" is the center position of position "3", and a fourth filter shape configuration includes three filter taps at positions labeled "4" and position "c". a fifth filter shape configuration includes three filter taps at positions labeled "5" and "c", with position "c" being a central position of position "5", a sixth filter shape configuration includes three filter taps at positions labeled "6" and position "c", with position "c" being a central position of position "6", and a seventh filter shape configuration includes three filter taps at positions labeled "7" and position "c", with position "c" being a central position of position "7".

[0194] In accordance with some aspects of the present disclosure, the nonlinear mapping-based filter may be used with other in-loop filters in a loop filter chain. The location of the nonlinear mapping-based filter may affect the coding efficiency of the nonlinear mapping-based filter.

[0195] FIG. 29 shows a block diagram of a loop filter chain (2900) in some examples. The loop filter chain (2900) includes multiple filters connected in series in a filter chain. The loop filter chain (2900) can be used as the loop filter unit (556) in one example. The loop filter chain (2900) can be used in the encoding loop or the decoding loop before storing the reconstructed picture in a decoded picture buffer, such as the reference picture memory (557). The loop filter chain (2900) receives input reconstructed samples from a previous processing module and applies filters to the reconstructed samples to generate output reconstructed samples.

[0196] The loop filter chain (2900) may include any suitable filter. In the example of FIG. 29, the loop filter chain (2900) includes a deblocking filter (labeled DEBLOCKING), a constrained directional extension filter (labeled CDEF), and an in-loop reconstruction filter (labeled LR) connected in the chain. The loop filter chain (2900) has an input node (2901), an output node (2909), and multiple intermediate nodes (2902)-(2903). The input node (2901) of the loop filter chain (2900) receives an input reconstruction sample from a previous processing module, which is provided to the deblocking filter. The intermediate node (2902) receives the reconstruction sample from the deblocking filter (after being processed by the deblocking filter) and provides the reconstruction sample to the CDEF for further filtering. The intermediate node (2903) receives the reconstructed samples from the CDEF (after being processed by the CDEF) and provides the reconstructed samples to the LR filter for further filtering. The output node (2909) receives the output reconstructed samples from the LR filter (after being processed by the LR filter). The output reconstructed samples can be provided to other processing modules, such as a post-processing module for further processing.

[0197] It should be noted that the following description illustrates a technique for using a nonlinear mapping-based filter based on a loop filter chain (2900). The technique for using a nonlinear mapping-based filter can be used in other suitable loop filter chains.

[0198] According to one aspect of the present disclosure, the nonlinear mapping-based filter may be coupled in series with other filters in a loop filter chain, and the input and output of the nonlinear mapping-based filter may be located at the same node of the loop filter chain, and no other filters are present between the input and output of the nonlinear mapping-based filter. For example, the nonlinear mapping-based filter receives a reconstructed sample at a node of the loop filter chain, determines a sample offset based on the reconstructed sample at the node of the loop filter chain, and then applies the sample offset at the reconstructed sample at the node of the loop filter chain.

[0199] 30A-30D show examples of loop filter chains that include a nonlinear mapping-based filter coupled in series with other filters in the loop filter chain.

[0200] FIG. 30A shows an example of a loop filter chain (3000A) in one example. The loop filter chain (3000A) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3000A), a nonlinear mapping-based filter (labeled SO-NLM) is applied at an input node. Specifically, the input (also called the first reconstructed sample) of the nonlinear mapping-based filter is the input reconstructed sample (to the loop filter chain (3000A)) at the first node (3011A). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample at the second node (3012A) to generate an output (also called the second reconstructed sample) at the third node (3013A), and the output is provided to a deblocking filter for further filtering. In the example of FIG. 30A, the first node (3011A) and the second node (3012A) are the same node, and the intermediate reconstructed sample is the first reconstructed sample.

[0201] FIG. 30B shows an example of a loop filter chain (3000B) in one example. The loop filter chain (3000B) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3000B), a nonlinear mapping-based filter (labeled SO-NLM) is applied at an intermediate node of the loop filter chain (3000B). Specifically, the input (also called the first reconstructed sample) of the nonlinear mapping-based filter is the reconstructed sample (generated by the deblocking filter) at the first node (3011B). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample at the second node (3012B) to generate an output (also called the second reconstructed sample) at the third node (3013B), and the output is provided to the CDEF for further filtering. In the example of FIG. 30B, the first node (3011B) and the second node (3012B) are the same node, and the intermediate reconstructed sample is the first reconstructed sample.

[0202] FIG. 30C shows an example of a loop filter chain (3000C) in one example. The loop filter chain (3000C) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3000C), a nonlinear mapping-based filter (labeled SO-NLM) is applied at an intermediate node of the loop filter chain (3000C). Specifically, the input (also called the first reconstructed sample) of the nonlinear mapping-based filter is the reconstructed sample (generated by CDEF) at the first node (3011C). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample at the second node (3012C) to generate an output (also called the second reconstructed sample) at the third node (3013C), and the output is provided to the LR filter for further filtering. In the example of FIG. 30C, the first node (3011C) and the second node (3012C) are the same node, and the intermediate reconstructed sample is the first reconstructed sample.

[0203] FIG. 30D shows an example of a loop filter chain (3000D) in one example. The loop filter chain (3000D) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3000D), a nonlinear mapping-based filter (labeled SO-NLM) is applied at an output node. Specifically, the input (also called the first reconstructed sample) of the nonlinear mapping-based filter is the reconstructed sample (generated by the LR filter) at the first node (3011D). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample at the second node (3012D) to generate an output (also called the second reconstructed sample) at the third node (3013D), and the output is the output of the loop filter chain (3000D). In the example of FIG. 30D, the first node (3011D) and the second node (3012D) are the same node, and the intermediate reconstructed sample is the first reconstructed sample.

[0204] According to one aspect of the present disclosure, the nonlinear mapping-based filter may be coupled with one or more filters in parallel in a loop filter chain, with at least one filter between an input and an output of the nonlinear mapping-based filter. For example, the nonlinear mapping-based filter receives a reconstructed sample at a first node of the loop filter chain, determines a sample offset based on the reconstructed sample at the first node of the loop filter chain, and then applies a sample offset at the reconstructed sample at a second node of the loop filter chain. The reconstructed sample at the second node of the loop filter chain may be obtained by applying one or more filters to the reconstructed sample at the first node of the loop filter chain.

[0205] In some examples, the input of the nonlinear mapping-based filter is the reconstructed samples located after the deblocking filter and before the CDEF, and the output of the nonlinear mapping-based filter is applied to the reconstructed samples after the CDEF and before the LR filter or after the LR filter.

[0206] FIG. 31A shows an example of a loop filter chain (3100A) including a nonlinear mapping-based filter coupled in parallel with a CDEF. The loop filter chain (3100A) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3100A), a nonlinear mapping-based filter (labeled SO-NLM) is applied between two intermediate nodes. Specifically, the input (also called the first reconstructed sample) of the nonlinear mapping-based filter is the reconstructed sample (generated by the deblocking filter) at the first node (3111A). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample (generated by the CDEF) at the second node (3112A) to generate an output (also called the second reconstructed sample) at the third node (3113A), which is provided to the LR filter for further filtering. In the example of Figure 31A, the CDEF is between a first node (3111A) and a second node (3112A). The intermediate reconstructed samples are the output of the CDEF.

[0207] FIG. 31B shows an example of a loop filter chain (3100B) including a nonlinear mapping-based filter coupled in parallel with a CDEF and an LR filter. The loop filter chain (3100B) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3100B), a nonlinear mapping-based filter (labeled SO-NLM) is applied between an intermediate node and an output node. Specifically, the input (also called the first reconstructed sample) of the nonlinear mapping-based filter is the reconstructed sample (generated by the deblocking filter) at the first node (3111B). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample (generated by the LR filter) at the second node (3112B) to generate an output (also called the second reconstructed sample) at the third node (3113B), and the output is the output of the loop filter chain (3100B). In the example of Figure 31B, the CDEF and LR filter are between a first node (3111B) and a second node (3112B). The intermediate reconstructed samples are the output of the LR filter.

[0208] In some examples, the input of the nonlinear mapping-based filter is the reconstructed samples located before the deblocking filter, and the output of the nonlinear mapping-based filter is applied to the reconstructed samples after the deblocking filter and before the CDEF, after the CDEF and before the LR filter, or after the LR filter.

[0209] FIG. 32A shows an example of a loop filter chain (3200A) including a nonlinear mapping-based filter coupled in parallel with a deblocking filter. The loop filter chain (3200A) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3200A), a nonlinear mapping-based filter (labeled SO-NLM) is applied between an input node and an intermediate node of the loop filter chain. Specifically, the input (also called the first reconstructed sample) of the nonlinear mapping-based filter is the input reconstructed sample (the input of the loop filter chain (3200A)) at the first node (3211A). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample (generated by the deblocking filter) at the second node (3212A) to generate an output (also called the second reconstructed sample) at the third node (3213A), and the output is provided to the CDEF for further filtering. In the example of Figure 32A, the deblocking filter is between the first node (3211A) and the second node (3212A). The intermediate reconstructed samples are the output of the deblocking filter.

[0210] FIG. 32B shows an example of a loop filter chain (3200B) including a nonlinear mapping-based filter coupled in parallel with a deblocking filter and a CDEF. The loop filter chain (3200B) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3200B), a nonlinear mapping-based filter (labeled SO-NLM) is applied between an input node and an intermediate node of the loop filter chain. Specifically, the input (also called the first reconstructed sample) of the nonlinear mapping-based filter is the input reconstructed sample (the input of the loop filter chain (3200B)) at the first node (3211B). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample (generated by the CDEF) at the second node (3212B) to generate an output (also called the second reconstructed sample) at the third node (3213B), and the output is provided to the LR for further filtering. In the example of Figure 32B, the deblocking filter and the CDEF are between the first node (3211B) and the second node (3212B). The intermediate reconstructed samples are the output of the CDEF.

[0211] FIG. 32C shows an example of a loop filter chain (3200C) including a nonlinear mapping-based filter coupled in parallel with a deblocking filter, a CDEF, and an LR filter. The loop filter chain (3200C) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3200C), a nonlinear mapping-based filter (labeled SO-NLM) is applied between the input node and the output node of the loop filter chain. Specifically, the input (also called the first reconstructed sample) of the nonlinear mapping-based filter is the input reconstructed sample at the first node (3211C) (the input of the loop filter chain (3200B)). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample (generated by the LR filter) at the second node (3212C) to generate an output (also called the second reconstructed sample) at the third node (3213C), which is the output of the loop filter chain (3200C). In the example of FIG. 32C, the deblocking filter, CDEF, and LR filter are between the first node (3211C) and the second node (3212C). The intermediate reconstructed sample is the output of the LR filter.

[0212] In some examples, the input of the nonlinear mapping-based filter is the reconstructed samples located after the CDEF and before the LR filter, and the output of the nonlinear mapping-based filter is applied to the reconstructed samples after the LR filter.

[0213] FIG. 33 shows an example of a loop filter chain (3300) including a nonlinear mapping-based filter coupled in parallel with an LR filter. The loop filter chain (3300) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3300), a nonlinear mapping-based filter (labeled SO-NLM) is applied between an intermediate node and an output node of the loop filter chain. Specifically, the input (also called the first reconstructed sample) of the nonlinear mapping-based filter is the reconstructed sample (generated by the CDEF) at the first node (3311). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample (generated by the LR filter) at the second node (3312) to generate an output (also called the second reconstructed sample) at the third node (3313), which is the output of the loop filter chain (3300). In the example of Figure 33, the LR filter is between the first node (3311) and the second node (3312). The intermediate reconstructed samples are the output of the LR filter.

[0214] According to another aspect of the present disclosure, multiple nonlinear mapping-based filters can be applied simultaneously at multiple positions in the loop filter chain, each of which can be configured as any of the examples shown in Figures 30A-30D, 31A-31B, 32A-32C, and 33.

[0215] FIG. 34 shows an example of a loop filter chain (3400) including a first nonlinear mapping-based filter denoted by SO-NLM1 and a second nonlinear mapping-based filter denoted by SO-NLM2. The loop filter chain (3400) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. The first nonlinear mapping-based filter is coupled in parallel with the CDEF in a configuration similar to the example shown in FIG. 31A. The second nonlinear mapping-based filter is coupled in series with other filters in the loop filter chain (3400) in a configuration similar to the example shown in FIG. 30D. In one example, the first nonlinear mapping-based filter and the second nonlinear mapping-based filter are the same type of filter, such as CCSO or LSO. In another example, the first nonlinear mapping-based filter and the second nonlinear mapping-based filter are different types of filters, such as one being CCSO and the other being LSO.

[0216] FIG. 35 shows an example of a loop filter chain (3500) including a first nonlinear mapping-based filter denoted by SO-NLM1 and a second nonlinear mapping-based filter denoted by SO-NLM2. The loop filter chain (3500) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. The first nonlinear mapping-based filter is coupled in parallel with the LR filter in a configuration similar to the example shown in FIG. 33. The second nonlinear mapping-based filter is coupled in series with other filters in the loop filter chain (3500) in a configuration similar to the example shown in FIG. 30B. In one example, the first nonlinear mapping-based filter and the second nonlinear mapping-based filter are the same type of filter, such as CCSO or LSO. In another example, the first nonlinear mapping-based filter and the second nonlinear mapping-based filter are different types of filters, such as one being CCSO and the other being LSO.

[0217] FIG. 36 shows a flow chart outlining a process (3600) according to one embodiment of the present disclosure. The process (3600) can be used for video filtering. When the term block is used, the block may be interpreted as a prediction block, a coding unit, a luma block, a chroma block, etc. In various embodiments, the process (3600) is performed by processing circuits in the terminal devices (310), (320), (330), and (340), processing circuits performing the functions of a video encoder (403), processing circuits performing the functions of a video decoder (410), processing circuits performing the functions of a video decoder (510), processing circuits performing the functions of a video encoder (603), etc. In some embodiments, the process (3600) is implemented in software instructions, and thus, the processing circuits perform the process (3600) when the processing circuits execute the software instructions. The process begins at (S3601) and proceeds to (S3610).

[0218] At (S3610), a first offset value for applying the nonlinear mapping-based filter is based on a first reconstructed sample at a first node along the loop filter chain.

[0219] At (S3620), the first offset value is applied to the intermediate reconstructed sample at a second node along the loop filter chain to generate a second reconstructed sample at a third node along the loop filter chain.

[0220] In one example, the nonlinear mapping-based filter is a inter-component sample offset (CCSO) filter, and the intermediate reconstruction samples and the first reconstruction samples are samples of different color components.

[0221] In another example, the nonlinear mapping-based filter is a local sample offset (LSO) filter, and the intermediate reconstruction samples and the first reconstruction samples are samples of the same color component.

[0222] In some embodiments, the first node and the second node are of the same node, which may be an input node of the loop filter chain, or an output node of the loop filter chain, or an intermediate node of the loop filter chain.

[0223] In one example, the first reconstruction sample is generated by a processing module prior to the deblocking filter. In another example, the first reconstruction sample is generated by the deblocking filter. In another example, the first reconstruction sample is generated by a constrained directional extension filter. In another example, the first reconstruction sample is generated by a loop reconstruction filter.

[0224] In some embodiments, the first node and the second node are of different nodes. In some examples, the first restored sample is generated by a processing module prior to the deblocking filter, and the intermediate restored sample is generated by at least one of the deblocking filter, the constrained directional enhancement filter, or the loop restoration filter. In some examples, the first restored sample is generated by the deblocking filter, and the intermediate restored sample is generated by at least one of the constrained directional enhancement filter or the loop restoration filter. In some examples, the first restored sample is generated by the constrained directional enhancement filter, and the intermediate restored sample is generated by the loop restoration filter.

[0225] The process (3600) proceeds to (S3699) and ends.

[0226] It should be noted that in some examples, the nonlinear mapping-based filter is a inter-component sample offset (CCSO) filter, and in some other examples, the nonlinear mapping-based filter is a local sample offset (LSO) filter.

[0227] The process (3600) may be adapted as appropriate. Steps of the process (3600) may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0228] The embodiments of the present disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0229] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 37 illustrates a computer system (3700) suitable for implementing some embodiments of the disclosed subject matter.

[0230] The computer software may be coded using any suitable machine or computer language that can be assembled, compiled, linked, or similar mechanisms to produce code including instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., directly, or via interpretation, microcode execution, etc.

[0231] The instructions may be executed on various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0232] The components illustrated in Figure 37 for the computer system (3700) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of the computer system (3700).

[0233] The computer system (3700) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, clapping), visual input (such as gestures), or olfactory input (not depicted). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (such as voice, music, ambient sounds), images (such as scanned images, photographic images obtained from a still camera), and video (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0234] The input human interface devices may include one or more of a keyboard (3701), a mouse (3702), a trackpad (3703), a touch screen (3710), a data glove (not shown), a joystick (3705), a microphone (3706), a scanner (3707), and a camera (3708) (only one of each is depicted).

[0235] The computer system (3700) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, via haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (3710), data gloves (not shown), or joystick (3705), although there may be haptic feedback devices that do not function as input devices), audio output devices (such as speakers (3709), headphones (not depicted)), visual output devices (such as screens (3710), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or three- or more-dimensional output via means such as stereographic output, virtual reality glasses (not depicted), holographic displays, and smoke tanks (not depicted)), and printers (not depicted).

[0236] The computer system (3700) may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (3720) with CD / DVD or similar media (3721), thumb drives (3722), removable hard drives or solid state drives (3723), legacy magnetic media such as tapes and floppy disks (not depicted), and specialized ROM / ASIC / PLD based devices (not depicted) such as security dongles.

[0237] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0238] The computer system (3700) may also include an interface (3754) to one or more communication networks (3755). The network may be, for example, wireless, wired, optical. The network may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial including CANBus, etc. Certain networks typically require an external network interface adapter attached to a specific general-purpose data port (e.g., a USB port of the computer system (3700)) or peripheral bus (3749), while other networks are typically integrated into the core of the computer system (3700) by attaching to a system bus described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (3700) can communicate with other entities. Such communications may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to a specific CANbus device), or bidirectional, for example, with other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces described above.

[0239] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core (3740) of the computer system (3700).

[0240] The cores (3740) may include specialized programmable processing devices in the form of one or more central processing units (CPUs) (3741), graphics processing units (GPUs) (3742), field programmable gate areas (FPGAs) (3743), hardware accelerators for specific tasks (3744), graphics adapters (3750), and the like. These devices may be connected via a system bus (3748), along with read only memory (ROM) (3745), random access memory (3746), and internal mass storage (3747), such as internal non-user accessible hard drives, SSDs, and the like. In some computer systems, the system bus (3748) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripherals may be attached directly to the core's system bus (3748) or via a peripheral bus (3749). In one example, a display (3710) may be connected to the graphics adapter (3750). Architectures for peripheral buses include PCI, USB, etc.

[0241] The CPU (3741), GPU (3742), FPGA (3743), and accelerator (3744) can combine to execute certain instructions that may constitute the aforementioned computer code. That computer code can be stored in ROM (3745) or RAM (3746). Persistent data can be stored, for example, in internal mass storage (3747), while transitory data can also be stored in RAM (3746). Rapid storage and retrieval from any of the memory devices can be enabled using cache memory, which can be closely associated with one or more of the CPU (3741), GPU (3742), mass storage (3747), ROM (3745), RAM (3746), etc.

[0242] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0243] By way of example, and not by way of limitation, a computer system (3700) having an architecture, and specifically a core (3740), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be the user-accessible mass storage introduced above, as well as media associated with a particular storage of the core (3740) of a non-transitory nature, such as the core internal mass storage (3747) or ROM (3745). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (3740). The computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause the core (3740), and specifically the processor therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (3746) and modifying such data structures according to processes defined by the software. Additionally, or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (3744)) that may operate in place of or together with software to perform certain processes or certain portions of certain processes described herein. Where appropriate, references to software may encompass logic, and vice versa. Where appropriate, references to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software. Appendix A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding MPM: Most Probable Mode WAIP: Wide-angle Intra Prediction SEI: Supplemental Extended Information VUI: Video Usability Information GOP: Group of Pictures TU: conversion unit PU: Prediction unit CTU: Coding Tree Unit CTB: coding tree block PB: Predicted block HRD: Hypothetical Reference Decoder SDR: Standard Dynamic Range SNR: Signal to Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD:Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Area SSD: Solid State Drive IC: Integrated Circuit CU: coding unit PDPC: Position-dependent predictive coupling ISP: Intra Subpartition SPS: Sequence Parameter Settings

[0244] While this disclosure describes several exemplary embodiments, there are alterations, substitutions, and various alternative equivalents that are within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0245] 101 Samples 102 Arrow 103 Arrow 104 Square Block 180 Schematic diagram 201 Current Block 300 Communication Systems 310 Terminal Devices 320 Terminal Devices 330 Terminal Devices 340 Terminal Devices 350 Communication Network 400 Communication Systems 401 Video Source 402 Video Picture Stream 403 Video Encoder 404 Encoded Video Data, Encoded Video Bitstream 405 Streaming Server 406 Client Subsystem 407 Encoded video data, input copy 408 Client Subsystem 409 Encoded video data, copy 410 Video Decoder 411 Video Picture Output Stream 412 Display 413 Capture Subsystem 420 Electronic Devices 430 Electronic Devices 501 Channel 510 Video Decoder 512 Rendering Device 515 Buffer Memory 520 Parser 521 Symbols 530 Electronic Devices 531 Receiver 551 Scaler / Descaler Unit 552 Intra-picture prediction unit 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter Unit 557 Reference Picture Memory 558 Current Picture Buffer 601 Video Sources 603 Video Encoder, Video Coder 620 Electronic Devices 630 Source Coder 632 Coding Engine 633 Local Video Decoder 634 Reference Picture Cache, Reference Picture Memory 635 Predictor 640 Transmitter 643 coded video sequence 645 Entropy Coder 650 Controller 660 Communication Channels 703 Video Encoder 721 General-purpose controller 722 Intra Encoder 723 Residual Calculator 724 Residual Encoder 725 Entropy Encoder 726 Switch 728 Residual Decoder 730 InterEncoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Recovery Module 880 Interdecoder 910 Diamond Shape Filter 911 Diamond Shape Filter 920~932 elements 940~964 elements 1110 4x4 Block 1111 4x4 Block 1120 Horizontal CTU Boundary 1121 CTU boundary 1130 Virtual Boundary 1131 Virtual Boundary 1210 Virtual Boundary 1220 Virtual Boundary 1230 Virtual Boundary 1240 Virtual Boundary 1250 Virtual Boundary 1260 Virtual Boundary 1300 Pictures 1400 Quadtree division pattern 1510 Sample Adaptive Offset (SAO) Filter 1512 SAO Filter 1514 SAO Filter 1516 ALF Luminance Filter 1518 ALF Chroma Filter 1521 CC-ALF 1522 Adder 1531 CC-ALF 1532 Adder 1541 SAO filtered luminance component 1542 Second intermediate component 1543 4th intermediate component 1552 First intermediate component 1553 Third intermediate component 1561 Filtered Luminance CB 1562 filtered first chroma component 1563 Filtered second chroma component 1600 Filter 1610 Filter Coefficients 1620 Diamond shape 1801 Luminance Samples 1801(1) top left sample, luminance sample 1801(2) Top right sample, luminance sample 1801(3) Bottom left sample, luminance sample 1801(4) bottom right sample, luminance sample 1803 Saturation Samples 1803(1) Saturation Samples 1804 saturation samples 1805 saturation samples 1806 Saturation Samples 1807 Saturation Samples 1808 Saturation Samples 1910 Block 1920 direction 1923 direction 1930 Fully directional block 1933 Fully directional block 2100 SAO type table 2210 First Pattern 2220 Second Pattern 2230 Third Pattern 2240 Fourth Pattern 2300 Table for pixel classification rules 2400 Syntax Examples 2500 Filter Support Area 2600 Filter Support Area 2700 table 2900 Loop Filter Chain 2901 Input Node 2902 Intermediate Node 2903 Intermediate Node 2909 Output Node 3000A Loop Filter Chain 3000B Loop Filter Chain 3000C Loop Filter Chain 3000D Loop Filter Chain 3011A First Node 3012A Second Node 3013A 3rd Node 3011B First Node 3012B Second Node 3013B 3rd Node 3011C First Node 3012C Second Node 3013C Third Node 3011D First Node 3012D Second Node 3013D 3rd Node 3100A Loop Filter Chain 3100B Loop Filter Chain 3111A First Node 3112A Second Node 3113A Third Node 3111B First Node 3112B Second Node 3113B Third Node 3200A Loop Filter Chain 3200B Loop Filter Chain 3200C Loop Filter Chain 3211A First Node 3212A Second Node 3213A Third Node 3211B First Node 3212B Second Node 3213B Third Node 3211C First Node 3212C Second Node 3213C Third Node 3300 Loop Filter Chain 3311 First Node 3312 Second Node 3313 Third Node 3400 Loop Filter Chain 3500 Loop Filter Chain 3600 processes 3700 Computer Systems 3701 Keyboard 3702 Mouse 3703 Trackpad 3705 Joystick 3706 Microphone 3707 Scanner 3708 Camera 3709 Speaker 3710 Touch Screen 3720 CD / DVD ROM / RW 3721 CD / DVD or similar media 3722 Thumb Drive 3723 Removable Hard Drive or Solid State Drive 3740 cores 3741 Central Processing Unit (CPU) 3742 Graphics Processing Unit (GPU) 3743 Field Programmable Gate Area (FPGA) 3744 Hardware Accelerator 3745 Read-Only Memory (ROM) 3746 Random Access Memory 3747 Internal Mass Storage 3748 System Bus 3749 Surrounding bus 3750 Graphics Adapter 3754 Interface 3755 Communication Networks

Claims

1. 1. A method for filtering in video coding, comprising the steps of: determining, by a processor, a first offset value by applying a Inter-Component Sample Offset (CCSO) filter to a first reconstructed luma sample at a first node along a loop filter chain including a plurality of video filters, the applying of the CCSO filter using a lookup table to determine the first offset value corresponding to a combination of quantized differences between a current chroma sample and each of the first reconstructed luma samples; adding, by the processor, the first offset value to an intermediate reconstructed sample at a second node along the loop filter chain to generate a second reconstructed sample, the second node being a node downstream of a Constrained Directivity Enhancement Filter (CDEF) in the loop filter chain; A method comprising:

2. The method described in claim 1, wherein the intermediate restored sample and the first restored luminance sample are samples of different color components.

3. The method of claim 1, wherein the first node is located before a deblocking filter in the loop filter chain.

4. The method of claim 1 , wherein the first node and the second node are the same node.

5. The first reconstructed luma sample is Processing modules before the deblocking filter, the deblocking filter, or Loop Recovery Filter The method of claim 1 , wherein the signal is generated by at least one of the following:

6. The method of claim 1 , wherein the first node and the second node are different nodes.

7. The first reconstructed luma sample is generated by a processing module prior to a deblocking filter, and the intermediate reconstructed sample is The method of claim 1 , wherein the signal is generated by a loop reconstruction filter.

8. The method of claim 1 , wherein the first reconstructed luma sample is generated by a deblocking filter and the intermediate reconstructed sample is generated by a loop reconstruction filter.

9. The method of claim 1 , wherein the first reconstructed luma sample is generated by the CDEF and the intermediate reconstructed sample is generated by a loop reconstruction filter.

10. A processor; a memory coupled to the processor, The memory stores instructions configured to, when executed by the processor, cause the processor to perform a method according to any one of claims 1 to 9. Device.

11. A program for carrying out the method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium storing instructions configured to cause execution of a method according to any one of claims 1 to 9.

Images