Method and apparatus for video filtering

Abstract

To provide a method and apparatus for filtering in video encoding / decoding.SOLUTION: An apparatus for video filtering includes a processing circuit. The processing circuit determines a first offset value for applying a non-linear mapping based filter based on a first reconstructed sample at a first node in a loop filter chain. The processing circuit then adds the first offset value to an intermediate reconstructed sample at a second node along the loop filter chain to generate a second reconstructed sample.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Patent Application No. 17 / 449,126, entitled "METHOD AND APPARATUS FOR VIDEO FILTERING," filed September 28, 2021, which in turn claims the benefit of priority to U.S. Provisional Application No. 63 / 164,478, entitled "FLEXIBLE FILTER LOCATION FOR SAMPLE OFFSET," filed March 22, 2021. The entire disclosures of the prior applications are incorporated herein by reference in their entirety.

[0002] This disclosure generally describes embodiments related to video coding. [Background technology]

[0003] The discussion of the background art provided herein is intended to provide a general context for the present disclosure. The work of the inventors named herein, to the extent that their work is described in this background art section, and aspects of the description that may not otherwise qualify as prior art at the time of filing, are not admitted, expressly or impliedly, as prior art to the present disclosure.

[0004] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each with spatial dimensions of, for example, 1920 x 1080 luma samples and associated chroma samples. The series of pictures can have a fixed or variable picture rate (informally known as frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed video has specific bitrate requirements. For example, 1080p60 4:2:0 video (1920 x 1080 luma sample resolution at a 60 Hz frame rate) at 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0005] One goal of video coding and decoding can be to reduce redundancy in an input video signal through compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, sometimes by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to techniques that can restore an exact copy of the original signal from a compressed version. With lossy compression, the restored signal may not be identical to the original, but the distortion between the original and restored signal is small enough to make the restored signal useful for the intended application. For video, lossy compression is widely adopted. The amount of acceptable distortion depends on the application; for example, users of certain consumer streaming applications can tolerate higher distortion than users of television distribution applications. Achievable compression ratios can reflect that the greater the acceptable / tolerable distortion, the higher the compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding.

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture may be an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and therefore can be used as the first picture in a coded video bitstream and video session or as a still image. Samples of intra-blocks may be subjected to a transform, and the transform coefficients may be quantized before entropy coding. Intra-prediction can be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and the smaller the AC coefficients after the transform, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0008] Conventional intra-coding, such as that known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that rely on surrounding sample data and / or metadata obtained during the encoding / decoding of spatially adjacent, preceding blocks of data in decoding order. Such techniques are hereinafter referred to as "intra-prediction" techniques. Note that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from reference pictures.

[0009] Intra-prediction can take many different forms. When two or more of such techniques can be used in a given video coding technique, the technique in use can be coded as an intra-prediction mode. In some examples, a mode can have sub-modes and / or parameters, which can be coded separately or included in a mode codeword. The codeword used for a given mode / sub-mode / parameter combination can affect the coding efficiency of intra-prediction, and therefore can affect the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain modes of intra prediction were introduced in H.264, improved in H.265, and further refined in newer coding techniques such as Joint Search Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Predictor blocks can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of the neighboring samples are copied into the predictor block according to a direction. A reference to the direction in use can be coded in the bitstream or can itself be predicted.

[0011] Referring to FIG. 1A, depicted at the bottom right is a subset of nine known predictor directions from the 33 possible predictor directions (corresponding to the 33 angular modes of the 35 intra modes) in H.265. The point where the arrows converge (101) represents the sample being predicted. The arrows represent the direction from which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right and at an angle of 45 degrees from horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left of sample (101) at an angle of 22.5 degrees from horizontal.

[0012] 1A, a square block (104) of 4x4 samples (indicated by a thick dashed line) is depicted in the upper left. The square block (104) contains 16 samples, each labeled with "S," its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions within the block (104). Because the block is 4x4 samples in size, S44 is located in the lower right. Reference samples, which follow a similar numbering scheme, are also shown. The reference samples are labeled R, their Y position (e.g., row index), and their X position (column index) relative to the block (104). In both H.264 and H.265, since the predicted samples are adjacent to the block being reconstructed, negative values ​​need not be used.

[0013] Intra-picture prediction can work by copying reference sample values ​​from adjacent samples as assigned by the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating a prediction direction consistent with the arrow (102) for this block, i.e., the sample is predicted from one or more prediction samples located at an angle of 45 degrees from the horizontal and to the upper right. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, the values ​​of multiple reference samples may be combined, for example by interpolation, to calculate a reference sample, especially when the direction is not evenly divisible by 45 degrees.

[0015] The number of possible directions has increased as video coding technology has evolved. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), and as of the time of this disclosure, JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and specific entropy coding techniques are used to represent those likely directions with a small number of bits, accepting a certain penalty for less likely directions. Furthermore, the direction itself can sometimes be predicted from neighboring directions used in adjacent, already decoded blocks.

[0016] Figure 1B shows a schematic diagram (180) depicting 65 intra-prediction directions with JEM to illustrate the increasing number of prediction directions over time.

[0017] The mapping of intra-prediction direction bits in a coded video bitstream to represent directions may vary from one video coding technique to another, ranging, for example, from a simple direct mapping of prediction directions to intra-prediction modes to complex adaptation schemes involving codewords, most likely modes, and similar techniques. However, in all cases, there may be certain directions that are statistically less likely to occur in the video content than certain other directions. Because the goal of video compression is to reduce redundancy, these less likely directions are represented by more bits than more likely directions in well-performing video coding techniques.

[0018] Motion compensation may be a lossy compression technique, and may refer to a technique in which blocks of sample data from a previously reconstructed picture or portion thereof (reference picture) are used to predict a newly reconstructed picture or portion of a picture after being spatially shifted in a direction indicated by a motion vector (hereinafter, MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, with the third dimension being an indication of the reference picture in use (the latter may indirectly be a temporal dimension).

[0019] In some video compression techniques, the MV applicable to a particular region of sample data can be predicted from other MVs, e.g., from an MV associated with another region of sample data that is spatially adjacent to the region being restored and precedes that MV in decoding order. Doing so can significantly reduce the amount of data required to code the MV, thereby eliminating redundancy and increasing compression ratios. For example, when coding an input video signal derived from a camera (known as natural video), MV prediction can work effectively because there is a statistical possibility that regions larger than the region to which a single MV is applicable will move in a similar direction, and therefore, in some cases, can be predicted using similar motion vectors derived from MVs of neighboring regions. As a result, the MV detected for a given region is similar or identical to the MV predicted from surrounding MVs and, after entropy coding, can be represented with fewer bits than would be used to code the MV directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., a sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating a predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms provided by H.265, the one described herein is a technique hereafter referred to as "spatial merging".

[0021] Referring to Figure 2, a current block (201) contains samples that the encoder finds during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of coding its MV directly, the MV can be derived from metadata associated with one or more reference pictures, e.g., the most recent reference picture (in decoding order), using MVs associated with any one of five surrounding samples, denoted A0, A1, and B0, B1, B2 (202-206, respectively). In H.265, MV prediction can use predictors from the same reference picture used by neighboring blocks. Summary of the Invention [Means for solving the problem]

[0022] Aspects of the present disclosure provide methods and apparatuses for filtering in video encoding / decoding. In some examples, the apparatus for video filtering includes a processing circuit. The processing circuit determines a first offset value for applying a nonlinear mapping-based filter based on a first reconstructed sample at a first node in a loop filter chain. The processing circuit then adds the first offset value to an intermediate reconstructed sample at a second node along the loop filter chain to generate a second reconstructed sample.

[0023] In some examples, the nonlinear mapping-based filter is a cross-component sample offset (CCSO) filter, and the intermediate reconstructed samples and the first reconstructed samples are samples of different color components.

[0024] In some examples, the nonlinear mapping-based filter is a local sample offset (LSO) filter, and the intermediate reconstruction samples and the first reconstruction samples are samples of the same color component.

[0025] In some embodiments, the first node and the second node are of the same node, such as an input node of a loop filter chain, an output node of a loop filter chain, or an intermediate node of a loop filter chain. In one example, the first restored sample is generated by a processing module prior to the deblocking filter. In another example, the first restored sample is generated by the deblocking filter. In another example, the first restored sample is generated by a constrained directional enhancement filter. In another example, the first restored sample is generated by the loop restoration filter.

[0026] In some examples, the first node and the second node are of different nodes. In one example, the first restored sample is generated by a processing module prior to the deblocking filter, and the intermediate restored sample is generated by at least one of the deblocking filter, the constrained directional enhancement filter, or the loop restoration filter.

[0027] In some examples, the first reconstructed sample is generated by a deblocking filter, and the intermediate reconstructed sample is generated by at least one of a constrained directional extension filter or a loop reconstruction filter.

[0028] In some examples, the first reconstruction sample is generated by a constrained directional extension filter, and the intermediate reconstruction sample is generated by a loop reconstruction filter.

[0029] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any of the methods for video encoding / decoding.

[0030] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0031] [Figure 1A] FIG. 2 is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B] FIG. 1 is a diagram of an exemplary intra-prediction direction. [Figure 2] FIG. 1 is a schematic diagram of a current block and its surrounding spatial merge candidates in one example. [Figure 3] 1 is a schematic diagram of a simplified block diagram of a communication system (300), according to one embodiment. [Figure 4] FIG. 4 is a schematic diagram of a simplified block diagram of a communication system (400), according to one embodiment. [Figure 5] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 6] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 7] FIG. 10 is a block diagram of an encoder according to another embodiment. [Figure 8] FIG. 10 is a block diagram of a decoder according to another embodiment. [Figure 9] 1A-1C illustrate examples of filter shapes according to embodiments of the present disclosure. [Figure 10A] FIG. 10 illustrates an example of sub-sampled positions used to calculate gradients, according to an embodiment of the present disclosure. [Figure 10B] FIG. 10 illustrates an example of sub-sampled positions used to calculate gradients, according to an embodiment of the present disclosure. [Figure 10C] FIG. 10 illustrates an example of sub-sampled positions used to calculate gradients, according to an embodiment of the present disclosure. [Figure 10D] FIG. 10 illustrates an example of sub-sampled positions used to calculate gradients, according to an embodiment of the present disclosure. [Figure 11A] FIG. 10 illustrates an example of a virtual boundary filtering process, according to an embodiment of the present disclosure. [Figure 11B] FIG. 10 illustrates an example of a virtual boundary filtering process, according to an embodiment of the present disclosure. [Figure 12A] 10A-10C illustrate examples of symmetric padding operations on virtual boundaries according to embodiments of the present disclosure. [Figure 12B] 10A-10C illustrate examples of symmetric padding operations on virtual boundaries according to embodiments of the present disclosure. [Figure 12C] 10A-10C illustrate examples of symmetric padding operations on virtual boundaries according to embodiments of the present disclosure. [Figure 12D] 10A-10C illustrate examples of symmetric padding operations on virtual boundaries according to embodiments of the present disclosure. [Figure 12E] 10A-10C illustrate examples of symmetric padding operations on virtual boundaries according to embodiments of the present disclosure. [Figure 12F] 10A-10C illustrate examples of symmetric padding operations on virtual boundaries according to embodiments of the present disclosure. [Figure 13] FIG. 2 illustrates example partitions of a picture, according to some embodiments of the present disclosure. [Figure 14] FIG. 1 illustrates a quadtree division pattern for a picture in some examples. [Figure 15] FIG. 1 illustrates an inter-component filter according to one embodiment of the present disclosure. [Figure 16] FIG. 1 illustrates an example of a filter shape, according to an embodiment of the present disclosure. [Figure 17] FIG. 10 illustrates an example syntax for an inter-component filter, according to some embodiments of the present disclosure. [Figure 18A] 3A-3C are diagrams illustrating exemplary locations of chroma samples relative to luma samples, according to embodiments of the present disclosure. [Figure 18B] 3A-3C are diagrams illustrating exemplary locations of chroma samples relative to luma samples, according to embodiments of the present disclosure. [Figure 19] FIG. 10 illustrates an example of a direction search according to one embodiment of the present disclosure. [Figure 20] FIG. 10 illustrates an example showing subspace projection in some examples. [Figure 21] 1 is a table of multiple sample adaptive offset (SAO) types, according to one embodiment of the present disclosure. [Figure 22] 10A-10C illustrate example patterns for pixel classification at edge offset in some examples. [Figure 23] 10 is a table for pixel classification rules for edge offsets in some examples. [Figure 24] FIG. 10 illustrates an example of a syntax that may be signaled. [Figure 25] FIG. 10 illustrates an example of a filter support area, according to some embodiments of the present disclosure. [Figure 26] FIG. 10 illustrates an example of another filter support area, according to some embodiments of the present disclosure. [Figure 27A] 1 is a table having 81 combinations according to one embodiment of the present disclosure. [Figure 27B] 1 is a table having 81 combinations according to one embodiment of the present disclosure. [Figure 27C] 1 is a table having 81 combinations according to one embodiment of the present disclosure. [Figure 28] FIG. 10 illustrates seven filter shape configurations for three filter taps in one example. [Figure 29] FIG. 1 is a block diagram of a loop filter chain in some examples. [Figure 30A] 1A-1C illustrate examples of loop filter chains including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 30B] 1A-1C illustrate examples of loop filter chains including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 30C] 1A-1C illustrate examples of loop filter chains including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 30D]1A-1C illustrate examples of loop filter chains including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 31A] 1A-1C illustrate examples of loop filter chains including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 31B] 1A-1C illustrate examples of loop filter chains including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 32A] 1A-1C illustrate examples of loop filter chains including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 32B] 1A-1C illustrate examples of loop filter chains including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 32C] 1A-1C illustrate examples of loop filter chains including nonlinear mapping-based filters at different positions in the loop filter chain. [Figure 33] FIG. 10 illustrates another example of a loop filter chain including a nonlinear mapping-based filter in some examples. [Figure 34] FIG. 1 illustrates an example of a loop filter chain including multiple nonlinear mapping-based filters in some examples. [Figure 35] FIG. 10 illustrates another example of a loop filter chain including multiple nonlinear mapping-based filters in some examples. [Figure 36] 1 is a flowchart outlining a process according to one embodiment of the present disclosure. [Figure 37] FIG. 1 is a schematic diagram of a computer system, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0032] Figure 3 shows a simplified block diagram of a communication system (300) according to one embodiment of the present disclosure. The communication system (300) includes, for example, multiple terminal devices that can communicate with each other via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) perform unidirectional transmission of data. For example, the terminal device (310) can encode video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to other terminal devices (320) via the network (350). The encoded video data can be transmitted in the form of one or more coded video bitstreams. The terminal device (320) can receive the coded video data from the network (350), decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. Unidirectional data transmission may be common, such as in media serving applications.

[0033] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of coded video data, such as may occur during a video conference. In the case of bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) can encode video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) can also receive coded video data transmitted by the other of the terminal devices (330) and (340), decode the coded video data to recover the video pictures, and display the video pictures on an accessible display device according to the recovered video data.

[0034] In the example of FIG. 3 , terminal devices 310, 320, 330, and 340 may be depicted as a server, a personal computer, and a smartphone, although the principles of the present disclosure need not be so limited. Embodiments of the present disclosure find application with laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. Network 350 represents any number of networks that convey coded video data between terminal devices 310, 320, 330, and 340, including, for example, wired (wired) and / or wireless communication networks. Communication network 350 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network 350 may not be important to the operation of the present disclosure, unless otherwise described herein below.

[0035] 4 shows an arrangement of a video encoder and a video decoder in a streaming environment as an example for application of the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital television, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0036] The streaming system may include a video source (401), e.g., a capture subsystem (413), which may include a digital camera, that creates a stream of uncompressed video pictures (402). In one example, the stream of video pictures (402) includes samples captured by the digital camera. The stream of video pictures (402), depicted as thick lines to emphasize the large amount of data when compared to the encoded video data (404) (or coded video bitstream), can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404), depicted as thin lines to emphasize the small amount of data when compared to the stream of video pictures (402), (or coded video bitstream (404)), can be stored on a streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) within the electronic device (430). The video decoder (410) decodes the input copy (407) of the encoded video data and creates an output stream (411) of video pictures that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) can be encoded according to a particular video coding / compression standard. Examples of such standards include ITU-T Recommendation H.265.In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC), and the disclosed subject matter may be used in the context of VVC.

[0037] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0038] 5 shows a block diagram of a video decoder (510) according to one embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) in the example of FIG. 4.

[0039] The receiver (531) can receive one or more coded video sequences, one coded video sequence at a time, to be decoded by the video decoder (510), in the same or another embodiment, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (531) can receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, that can be forwarded to their respective using entities (not depicted). The receiver (531) can separate the coded video sequences from other data. To combat network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter, "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In other applications, it may be external to the video decoder (510) (not depicted). In still other applications, there may be a buffer memory (not depicted) external to the video decoder (510), for example, to combat network jitter, plus another buffer memory (515) internal to the video decoder (510), for example, to handle playout timing. When the receiver (531) is receiving data from a store-and-forward device of sufficient bandwidth and controllability or from an equally synchronous network, the buffer memory (515) may not be required or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (515) may be required, may be relatively large, may advantageously be adaptively sized, and may be implemented at least in part in an operating system or similar element (not depicted) external to the video decoder (510).

[0040] The video decoder (510) may include a parser (520) to recover symbols (521) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (510) and, potentially, information for controlling a rendering device, such as a rendering device (512) (e.g., a display screen) that is not an integral part of the electronic device (530) but may be coupled to the electronic device (530) as shown in FIG. 5. Control information for the rendering device may be in the form of a supplemental enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not depicted). The parser (520) can parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding, etc., with or without context sensitivity. The parser (520) can extract, from the coded video sequence, a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups can include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (520) can also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0041] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0042] The recovery of the symbols (521) can involve several different units, depending on the type of coded video picture or portion thereof (e.g., inter- and intra-picture, inter- and intra-block, etc.), as well as other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not depicted for clarity.

[0043] In addition to the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0044] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients as well as control information from the parser (520) as symbols (521), including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. The scalar / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0045] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates blocks of the same size and shape as the block being reconstructed using surrounding, already reconstructed information fetched from a current picture buffer (558). The current picture buffer (558), for example, buffers partially reconstructed and / or fully reconstructed current pictures. The aggregator (555) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0046] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion-compensated prediction unit (553) may access a reference picture memory (557) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (521) associated with the block, these samples may be added to the output of the scalar / inverse transform unit (551) by an aggregator (555) to generate output sample information (in this case, referred to as residual samples or residual signals). The addresses in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches prediction samples may be controlled by motion vectors available to the motion-compensated prediction unit (553), for example, in the form of symbols (521) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0047] The output samples of the aggregator (555) can be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques can include in-loop filter techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but can also respond to previously reconstructed and loop-filtered sample values ​​as well as to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence.

[0048] The output of the loop filter unit (556) may be a sample stream that can be output to the rendering device (512) as well as stored in a reference picture memory (557) for use in future inter-picture prediction.

[0049] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and any unused current picture buffer can be reallocated before beginning reconstruction of the next coded picture.

[0050] The video decoder (510) can perform decoding operations according to a predetermined video compression technique in a standard, such as ITU-T Rec. H.265. A coded video sequence can conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence adheres to both the syntax of the video compression technique or standard and the profile documented in the video compression technique. Specifically, a profile can select some tools from all tools available in the video compression technique or standard as the only tools available for use under that profile. Compliance can also require that the complexity of the coded video sequence be within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level can, in some cases, be further limited by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled within the coded video sequence.

[0051] In one embodiment, the receiver (531) can receive additional (redundant) data along with the coded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately recover the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0052] 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0053] The video encoder (603) can receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of FIG. 6) that can capture video images to be coded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0054] The video source (601) can provide a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream, which can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) can be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual pictures that convey motion when viewed in sequence. The pictures themselves can be organized as a spatial array of pixels, each of which can contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art can readily understand the relationship between pixels and samples. The following description focuses on samples.

[0055] According to one embodiment, the video encoder (603) can code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other time constraints required by the application. Enforcing an appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units described below. For clarity, coupling is not depicted. Parameters set by the controller (650) can include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured with other appropriate functions associated with the video encoder (603) optimized for a particular system design.

[0056] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to that of a (remote) decoder (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques contemplated by the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Because decoding of the symbol stream leads to bit-exact results regardless of the location (local or remote) of the decoder, the contents of the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder "sees" when using the prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is also used in several related technologies.

[0057] The operation of the "local" decoder (633) may be the same as the operation of a "remote" decoder, such as the video decoder (510), already described in detail above in conjunction with Figure 5. However, and with brief reference again to Figure 5, because symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515), and the parser (520), may not be fully implemented in the local decoder (633).

[0058] An observation that can be made at this point is that any decoder technology other than parsing / entropy decoding that is present in a decoder must necessarily be present in the corresponding encoder in substantially identical functional form. For this reason, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder technology can be omitted, as it is the inverse of the decoder technology described generically. Only in certain areas is more detailed description necessary, and is provided below.

[0059] In operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this way, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0060] The local video decoder (633) can decode coded video data of pictures that may be designated as reference pictures based on symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the coded video data is decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a replica of the source video sequence, with some errors. The local video decoder (633) can replicate the decoding process that may be performed by the video decoder on the reference pictures, causing the reconstructed reference pictures to be stored in a reference picture cache (634). In this way, the video encoder (603) can locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures obtained by a far-end video decoder (without transmission errors).

[0061] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new picture. The predictor (635) may operate on sample blocks on a pixel block-by-pixel block basis to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0062] The controller (650) can manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0063] The output of all the aforementioned functional units can undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0064] The transmitter (640) can buffer the coded video sequence created by the entropy coder (645) and prepare it for transmission over a communication channel (660), which may be a hardware / software link to a storage device that stores the coded video data. The transmitter (640) can merge the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0065] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a particular coded picture type to each coded picture, which may affect the coding technique that can be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:

[0066] An intra-picture (I-picture) may be a picture that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow various types of intra-pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures, as well as their respective uses and characteristics.

[0067] A predictive picture (P picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0068] A bidirectionally predicted picture (B picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0069] A source picture is typically spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the block's respective picture. For example, blocks of an I-picture may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0070] The video encoder (603) may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In doing so, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0071] In one embodiment, the transmitter (640) can transmit additional data along with the coded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0072] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that was previously coded and is still buffered in the video, the block in the current picture can be coded by a vector called a motion vector. A motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0073] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which are earlier in decoding order than the current picture in the video (but may be past and future in display order, respectively). A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.

[0074] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0075] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree-decomposed into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In one example, each CU is analyzed to determine the CU's prediction type, such as inter-prediction type or intra-prediction type. The CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values) of 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0076] 7 shows a diagram of a video encoder (703) according to another embodiment of this disclosure. The video encoder (703) is configured to receive a processed block (e.g., a predictive block) of sample values ​​in a current video picture in a sequence of video pictures and encode the processed block into a coded picture that is part of a coded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) of the example of FIG. 4.

[0077] In an HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8x8 samples. The video encoder (703) determines whether the processing block is best coded using intra-mode, inter-mode, or bi-predictive mode, e.g., using rate-distortion optimization. When the processing block is coded in intra-mode, the video encoder (703) can encode the processing block into a coded picture using intra-prediction techniques, and when the processing block is coded in inter-mode or bi-predictive mode, the video encoder (703) can encode the processing block into a coded picture using inter-prediction or bi-prediction techniques, respectively. In certain video coding techniques, merge mode may be an inter-picture prediction sub-mode in which motion vectors are derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown), to determine the mode of the processing block.

[0078] In the example of Figure 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in Figure 7.

[0079] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-prediction information (e.g., a description of redundancy information, motion vectors, merge mode information according to the inter-coding technique), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the coded video information.

[0080] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), possibly compare the block with already coded blocks in the same picture, generate quantized coefficients after transformation, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). In one example, the intra encoder (722) also calculates intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same picture.

[0081] The general-purpose controller (721) is configured to determine general-purpose control data and control other components of the video encoder (703) based on the general-purpose control data. In one example, the general-purpose controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general-purpose controller (721) controls the switch (726) to select intra-mode results for use by the residual calculator (723) and the entropy encoder (725) to select intra-prediction information and include the intra-prediction information in the bitstream. When the mode is inter mode, the general-purpose controller (721) controls the switch (726) to select inter-prediction results for use by the residual calculator (723) and the entropy encoder (725) to select inter-prediction information and include the inter-prediction information in the bitstream.

[0082] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and a prediction result selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) is configured to operate on the residual data to encode the residual data and generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intra-encoder (722) and the inter-encoder (730), as appropriate. For example, the inter-encoder (730) can generate decoded blocks based on the decoded residual data and inter-prediction information, and the intra-encoder (722) can generate decoded blocks based on the decoded residual data and intra-prediction information. The decoded blocks are processed appropriately to generate decoded pictures, which can be buffered in a memory circuit (not shown) and, in some examples, used as reference pictures.

[0083] The entropy encoder (725) is configured to format the bitstream to include the encoded block. The entropy encoder (725) is configured to include various information in accordance with an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. It should be noted that, according to the disclosed subject matter, no residual information is present when coding a block in a merged sub-mode of either an inter-mode or a bi-prediction mode.

[0084] 8 shows a diagram of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) is used in place of the video decoder (410) of the example of FIG. 4.

[0085] In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter-decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-decoder (872) coupled together as shown in Figure 8.

[0086] The entropy decoder (871) can be configured to recover, from a coded picture, specific symbols that represent the syntax elements of which the coded picture is composed. Such symbols can include, for example, prediction information (e.g., intra-prediction information or inter-prediction information) that can identify the mode in which the block is coded (e.g., intra-mode, inter-mode, bi-prediction mode, merged sub-mode, or the latter two of another sub-mode), specific samples or metadata used for prediction by the intra decoder (872) or inter decoder (880), respectively, e.g., residual information in the form of quantized transform coefficients, etc. In one example, when the prediction mode is an inter-mode or bi-prediction mode, the inter-prediction information is provided to the inter decoder (880), and when the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra decoder (872). The residual information can undergo inverse quantization and be provided to the residual decoder (873).

[0087] The inter decoder (880) is configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0088] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0089] The residual decoder (873) is configured to perform inverse quantization to extract inverse quantized transform coefficients and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantizer parameters (QP)), which may be provided by the entropy decoder (871) (this may be only a small amount of control information, so a data path is not depicted).

[0090] The reconstruction module (874) combines, in the spatial domain, the residual output by the residual decoder (873) with the prediction results (possibly output by the inter-prediction module or the intra-prediction module) to form reconstructed blocks that may be part of a reconstructed picture, which in turn may be part of the reconstructed video. It should be noted that other appropriate operations, such as deblocking operations, may be performed to improve the appearance.

[0091] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technique. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703) may be implemented using one or more integrated circuits. 7 03), and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0092] Aspects of this disclosure provide filtering techniques for video coding / decoding.

[0093] An adaptive loop filter (ALF) with block-based filter adaptation can be applied by the encoder / decoder to reduce artifacts. For the luma component, for example, one of multiple filters (e.g., 25 filters) can be selected for a 4x4 luma block based on local gradient direction and activity.

[0094] The ALF can have any suitable shape and size. Referring to FIG. 9, the ALFs (910)-(911) have diamond shapes, such as a 5x5 diamond shape for the ALF (910) and a 7x7 diamond shape for the ALF (911). In the ALF (910), elements (920)-(932) form a diamond shape and can be used in the filtering process. Seven values ​​(e.g., C0-C6) can be used for the elements (920)-(932). In the ALF (911), elements (940)-(964) form a diamond shape and can be used in the filtering process. Thirteen values ​​(e.g., C0-C12) can be used for the elements (940)-(964).

[0095] Referring to Figure 9, in some examples, two ALFs (910)-(911) with diamond filter shapes are used. A 5x5 diamond-shaped filter (910) can be applied to a chroma component (e.g., a chroma block, saturation CB), and a 7x7 diamond-shaped filter (911) can be applied to a luma component (e.g., a luma block, luma CB). Other suitable shapes and sizes can be used in the ALFs. For example, a 9x9 diamond-shaped filter can be used.

[0096] The filter coefficients at the positions indicated by the values ​​(e.g., C0-C6 in (910) or C0-C12 in (920)) may be non-zero. Furthermore, if the ALF includes a clipping function, the clipping values ​​at those positions may be non-zero.

[0097] For luminance block classification, a 4x4 block (or luminance block, luminance CB) can be categorized or classified into one of several (e.g., 25) classes. The classification index C is calculated using Equation (1) by quantizing the directionality parameter D and the activity value A.

number

number

number

number

[0098] To reduce the complexity of the block classification described above, a subsampled 1-D Laplacian calculation can be applied. Figures 10A-10D show the gradient g in the vertical direction (Figure 10A), horizontal direction (Figure 10B), and two diagonal directions d1 (Figure 10C) and d2 (Figure 10D), respectively. v , g h , g d1 , and g d2 10A shows an example of the subsampled positions used to compute the vertical gradient g. The same subsampled positions can be used for gradient computations in different directions. In Figure 10A, the label "V" indicates the vertical gradient g.v In Figure 10B, the label "H" indicates the subsampled position for computing the horizontal gradient g h In Figure 10C, the label "D1" indicates the subsampled position for computing the diagonal gradient g d1 In Figure 10D, the label "D2" indicates the subsampled position for computing the diagonal gradient g d2 indicates the subsampled positions for computing

[0099] horizontal gradient g h and the vertical gradient g v Maximum value of

number

number

number

number

number

number

number

number

number

number

number

[0100] The activity value A is

number

number

[0101] For chroma components in a picture, no block classification is applied, and therefore a single set of ALF coefficients can be applied per chroma component.

[0102] A geometric transformation can be applied to the filter coefficients and the corresponding filter clipping values ​​(also called clipping values). Before filtering a block (e.g., a 4x4 luminance block), for example, the gradient values ​​(e.g., g v , g h , g d1 , and / or g d2Depending on the filter coefficients f(k,l), a geometric transformation such as a rotation or a diagonal and vertical flip can be applied to the filter coefficients f(k,l) and the corresponding filter clipping values ​​c(k,l). The geometric transformation applied to the filter coefficients f(k,l) and the corresponding filter clipping values ​​c(k,l) can be equivalent to applying a geometric transformation to the samples within the region supported by the filter. The geometric transformation can make the different blocks to which the ALF is applied more similar by aligning their respective directionality.

[0103] Three geometric transformations can be performed, including diagonal flip, vertical flip, and rotation, as described by equations (9)-(11), respectively. f D (k,l)=f(l,k), c D (k,l)=c(l,k) Equation (9) f V (k,l)=f(k,Kl-1), c V (k,l)=c(k,Kl-1) Equation (10) f R (k,l)=f(Kl-1,k), c R (k,l)=c(Kl-1,k) Equation (11) where K is the size of the ALF or filter, and 0≦k, l≦K-1 are the coordinates of the coefficients. For example, the filter f or clipping value matrix (or clipping matrix) c has position (0,0) in the upper left corner and position (K-1,K-1) in the lower right corner. Transforms can be applied to the filter coefficients f(k,l) and clipping values ​​c(k,l) depending on the gradient values ​​calculated for the block. An example of the relationship between the transforms and the four gradients is summarized in Table 1.

[0104] [Table 1]

[0105] In some embodiments, the ALF filter parameters are signaled within an adaptive parameter set (APS) for a picture. In the APS, one or more sets (e.g., up to 25 sets) of luma filter coefficients and clipping value indices may be signaled. In one example, a set of the one or more sets may include luma filter coefficients and one or more clipping value indices. One or more sets (e.g., up to 8 sets) of chroma filter coefficients and clipping value indices may be signaled. To reduce signaling overhead, filter coefficients of different classifications (e.g., having different classification indices) for the luma component may be merged. In the slice header, the index of the APS used for the current slice may be signaled.

[0106] In one embodiment, a clipping value index (also referred to as a clipping index) can be decoded from the APS. The clipping value index can be used to determine a corresponding clipping value, for example, based on a relationship between the clipping value index and the corresponding clipping value. The relationship can be predefined and stored in the decoder. In one example, the relationship is described by a table, such as a luma table (e.g., used for luma CB) of clipping value indexes and corresponding clipping values, or a saturation table (e.g., used for saturation CB) of clipping value indexes and corresponding clipping values. The clipping value may depend on the bit depth B. The bit depth B may refer to an internal bit depth, a bit depth of reconstructed samples in the CB to be filtered, etc. In some examples, the table (e.g., luma table, saturation table) is obtained using Equation (12).

number

[0107] [Table 2]

[0108] The slice header for the current slice may signal one or more APS indices (e.g., up to seven APS indices) to specify the luma filter sets that can be used for the current slice. The filtering process may be controlled at one or more appropriate levels, such as the picture level, slice level, or CTB level. In one embodiment, the filtering process may be further controlled at the CTB level. A flag may be signaled to indicate whether an ALF is applied to the luma CTB. The luma CTB may select a filter set from among multiple fixed filter sets (e.g., 16 fixed filter sets) and a filter set (also referred to as a signaled filter set) signaled in the APS. A filter set index may be signaled to the luma CTB to indicate the filter set to be applied (e.g., a filter set among the multiple fixed filter sets and the signaled filter set). The multiple fixed filter sets may be predefined and hard-coded in the encoder and decoder and may be referred to as predefined filter sets.

[0109] For the chroma component, an APS index can be signaled in the slice header to indicate the chroma filter set used for the current slice. At the CTB level, if there is more than one chroma filter set in an APS, a filter set index can be signaled per chroma CTB.

[0110] The filter coefficients may be quantized with a norm equal to 128. To reduce multiplication complexity, bitstream adaptation may be applied so that coefficient values ​​for non-center positions may be within the range of -27 to 27-1, inclusive. In one example, center position coefficients are not signaled in the bitstream and may be assumed to be equal to 128.

[0111] In some embodiments, the syntax and semantics of clipping indexes and clipping values ​​are defined as follows: alf_luma_clip_idx[sfIdx][j] can be used to specify the clipping index of the clipping value to use before multiplying the j-th coefficient of the signaled luma filter indicated by sfIdx. Bitstream conformance requirements may include that the value of alf_luma_clip_idx[sfIdx][j], where sfIdx=0 to alf_luma_num_filters_signalled_minus1 and j=0 to 11, should be in the range of 0 to 3, inclusive. The luma filter clipping value AlfClipL[adaptation_parameter_set_id] with element AlfClipL[adaptation_parameter_set_id][filtIdx][j], where filtIdx=0 to NumAlfFilters-1 and j=0 to 11, can be derived as specified in Table 2 depending on bitDepth set equal to BitDepthY and clipIdx set equal to alf_luma_clip_idx[alf_luma_coeff_delta_idx[filtIdx]][j]. alf_chroma_clip_idx[altIdx][j] can be used to specify the clipping index of the clipping value to use before multiplying the jth coefficient of the alternative chroma filter with index altIdx. Bitstream conformance requirements may include that the value of alf_chroma_clip_idx[altIdx][j], where altIdx=0 to alf_chroma_num_alt_filters_minus1, j=0 to 5, should be in the range 0 to 3, inclusive. The chroma filter clipping value AlfClipC[adaptation_parameter_set_id][altIdx] with element AlfClipC[adaptation_parameter_set_id][altIdx][j], where altIdx=0 to alf_chroma_num_alt_filters_minus1, j=0 to 5, can be derived as specified in Table 2 depending on bitDepth set equal to BitDepthC and clipIdx set equal to alf_chroma_clip_idx[altIdx][j].

[0112] In one embodiment, the filtering process can be described as follows: On the decoder side, when ALF is enabled for CTB, samples R(i,j) in a CU (or CB) can be filtered, and filtered sample values ​​R′(i,j) are obtained using equation (13) as shown below: In one example, each sample in a CU is filtered.

number

[0113] In a nonlinear ALF, multiple sets of clipping values ​​can be provided in Table 3. In one example, the luma set includes four clipping values ​​{1024, 181, 32, 6}, and the chroma set includes four clipping values ​​{1024, 161, 25, 4}. The four clipping values ​​in the luma set can be selected by approximately equally dividing the full range of sample values ​​(encoded with 10 bits) for the luma block (e.g., 1024) in the logarithmic domain. The range can be from 4 to 1024 for the chroma set.

[0114] [Table 3]

[0115] The selected clipping value may be coded within the "alf_data" syntax element as follows: An appropriate coding scheme (e.g., Golomb coding) may be used to code the clipping index corresponding to the selected clipping value as shown in Table 3. The coding scheme may be the same coding scheme used to code the filter set index.

[0116] In one embodiment, a virtual boundary filtering process can be used to reduce the line buffer requirements of ALF. Therefore, modified block classification and filtering can be employed for samples near CTU boundaries (e.g., horizontal CTU boundaries). The virtual boundary (1130) is defined as the horizontal CTU boundary (1120) as shown in FIG. 11A. samples " can be defined as a line by shifting the sample by N samples can be a positive integer. In one example, N samples is equal to 4 for the luminance component, and N samples is equal to 2 for the saturation component.

[0117] Referring to Figure 11A, modified block classification can be applied to the luma component. In one example, a 1D Laplacian gradient calculation for a 4x4 block (1110) above a virtual boundary (1130) uses only samples above the virtual boundary (1130). Similarly, referring to Figure 11B, a 1D Laplacian gradient calculation for a 4x4 block (1111) below a virtual boundary (1131) shifted from the CTU boundary (1121) uses only samples below the virtual boundary (1131). The quantization of the activity value A can be scaled accordingly by taking into account the reduction in the number of samples used in the 1D Laplacian gradient calculation.

[0118] For the filtering process, symmetric padding operations at the virtual boundary can be used for both luma and chroma components. Figures 12A-12F show examples of such modified ALF filtering for luma components at the virtual boundary. When a sample being filtered is located below the virtual boundary, adjacent samples located above the virtual boundary can be padded. When a sample being filtered is located above the virtual boundary, adjacent samples located below the virtual boundary can be padded. Referring to Figure 12A, adjacent sample C0 can be padded with sample C2 located below the virtual boundary (1210). Referring to Figure 12B, adjacent sample C0 can be padded with sample C2 located above the virtual boundary (1220). Referring to Figure 12C, adjacent samples C1-C3 can be padded with samples C5-C7 located below the virtual boundary (1230), respectively. Referring to Figure 12D, adjacent samples C1-C3 can be padded with samples C5-C7 located above the virtual boundary (1240), respectively. Referring to Figure 12E, adjacent samples C4-C8 can be padded with samples C10, C11, C12, C11, and C10, respectively, located below the virtual boundary (1250). Referring to Figure 12F, adjacent samples C4-C8 can be padded with samples C10, C11, C12, C11, and C10, respectively, located above the virtual boundary (1260).

[0119] In some instances, the above description can be appropriately adapted when a sample and an adjacent sample are located to the left (or right) and right (or left) of a virtual boundary.

[0120] According to aspects of the present disclosure, pictures can be partitioned based on a filtering process to improve coding efficiency. In some examples, a CTU is also referred to as a largest coding unit (LCU). In one example, a CTU or LCU can have a size of 64x64 pixels. In some embodiments, an LCU-aligned picture quadtree partition can be used for filtering-based partitioning. In some examples, a coding unit-synchronized picture quadtree-based adaptive loop filter can be used. For example, a luma picture can be divided into several multi-level quadtree partitions, with the boundaries of each partition aligned with the boundaries of the LCUs. Each partition has its own filtering process and may therefore be referred to as a filter unit (FU).

[0121] In some examples, a two-pass encoding flow may be used. In the first pass of the two-pass encoding flow, a quadtree division pattern of the picture and a best filter for each FU may be determined. In some embodiments, the determination of the quadtree division pattern of the picture and the determination of the best filter for the FU are based on filtering distortion. The filtering distortion may be estimated by a fast filtering distortion estimation (FFDE) technique during the determination process. The picture is partitioned using a quadtree partition. The reconstructed picture may be filtered according to the determined quadtree division pattern and the selected filters of all FUs.

[0122] In the second pass of the two-pass encoding flow, the CU-synchronized ALF on / off control is performed. According to the ALF on / off result, the initially filtered picture is partially restored by the reconstructed picture.

[0123] Specifically, in some examples, a top-down partitioning procedure is employed to divide an image into multi-level quadtree partitions using a rate-distortion criterion. Each partition is called a filter unit (FU). The partitioning process aligns the quadtree partitions to the boundaries of LCUs. The coding order of the FUs follows the z-scan order.

[0124] 13 illustrates an example partition according to some embodiments of the present disclosure. In the example of FIG. 13, a picture (1300) is divided into 10 FUs, with the coding order being FU0, FU1, FU2, FU3, FU4, FU5, FU6, FU7, FU8, and FU9.

[0125] Figure 14 shows a quadtree partitioning pattern (1400) for a picture (1300). In the example of Figure 14, a partition flag is used to indicate the partition pattern of the picture. For example, a "1" indicates that quadtree partitioning is performed on the block, and a "0" indicates that no further partitioning is performed. In some examples, the smallest size FU has the LCU size, and no partition flag is required for the smallest size FU. The partition flag is coded and transmitted in z-order as shown in Figure 14.

[0126] In some examples, the filter for each FU is selected from two filter sets based on a rate-distortion criterion. The first set has half-symmetric square and diamond filters derived for the current FU. The second set comes from a time-delay filter buffer, which stores filters previously derived for FUs of the previous picture. The filter with the smallest rate-distortion cost of these two sets can be selected for the current FU. Similarly, if the current FU is not the smallest FU and can be further divided into four child FUs, the rate-distortion costs of the four child FUs are calculated. By recursively comparing the rate-distortion costs for the division and no division cases, a picture quadtree division pattern can be determined.

[0127] In some examples, a maximum quadtree division level may be used to limit the maximum number of FUs. In one example, when the maximum quadtree division level is 2, the maximum number of FUs is 16. Furthermore, during quadtree division determination, correlation values ​​for deriving Wiener coefficients for the 16 FUs at the lowest quadtree level (smallest FUs) can be reused. The remaining FUs can derive their Wiener filters from the correlations of the 16 FUs at the lowest quadtree level. Thus, in that example, only one frame buffer access is performed to derive filter coefficients for all FUs.

[0128] After the quadtree division pattern is determined, CU-synchronized ALF on / off control can be performed to further reduce filtering distortion. By comparing the filtering distortion and non-filtering distortion in each leaf CU, the leaf CU can explicitly switch ALF on / off in its local region. In some examples, coding efficiency can be further improved by redesigning the filter coefficients according to the ALF on / off results.

[0129] The inter-component filtering process may apply an inter-component filter, such as an inter-component adaptive loop filter (CC-ALF). The inter-component filter may use the luma sample value of the luma component (e.g., luma CB) to refine the chroma component (e.g., chroma CB corresponding to luma CB). In one example, the luma CB and chroma CB are included in a CU.

[0130] FIG. 15 illustrates an inter-component filter (e.g., CC-ALF) used to generate a chroma component according to one embodiment of the present disclosure. In some examples, FIG. 15 illustrates a filtering process for a first chroma component (e.g., first chroma CB), a second chroma component (e.g., second chroma CB), and a luma component (e.g., luma CB). The luma component can be filtered by a sample adaptive offset (SAO) filter (1510) to generate an SAO-filtered luma component (1541). The SAO-filtered luma component (1541) can be further filtered by an ALF luma filter (1516) to become a filtered luma CB (1561) (e.g., “Y”).

[0131] The first chroma component can be filtered by the SAO filter (1512) and the ALF chroma filter (1518) to generate a first intermediate component (1552). Furthermore, the SAO-filtered luma component (1541) can be filtered by an inter-component filter (e.g., CC-ALF) for the first chroma component (1521) to generate a second intermediate component (1542). A filtered first chroma component (1562) (e.g., “Cb”) can then be generated based on at least one of the second intermediate component (1542) and the first intermediate component (1552). In one example, the filtered first chroma component (1562) (e.g., “Cb”) can be generated by combining the second intermediate component (1542) and the first intermediate component (1552) with an adder (1522). The inter-component adaptive loop filtering process for the first chroma component may include steps performed by the CC-ALF (1521) and steps performed by, for example, an adder (1522).

[0132] The above description can be adapted to the second chroma component. The second chroma component can be filtered by the SAO filter (1514) and the ALF chroma filter (1518) to generate a third intermediate component (1553). Furthermore, the SAO-filtered luma component (1541) can be filtered by an inter-component filter (e.g., CC-ALF) for the second chroma component (1531) to generate a fourth intermediate component (1543). A filtered second chroma component (1563) (e.g., “Cr”) can then be generated based on at least one of the fourth intermediate component (1543) and the third intermediate component (1553). In one example, the filtered second chroma component (1563) (e.g., “Cr”) can be generated by combining the fourth intermediate component (1543) and the third intermediate component (1553) with an adder (1532). In one example, the inter-component adaptive loop filtering process for the second chroma component may include steps performed by a CC-ALF (1531) and steps performed by, for example, an adder (1532).

[0133] The inter-component filters (e.g., CC-ALF(1521), CC-ALF(1531)) can operate by applying a linear filter with any suitable filter shape to the luma component (or luma channel) to refine each chroma component (e.g., first chroma component, second chroma component).

[0134] FIG. 16 shows an example of a filter (1600) according to one embodiment of the present disclosure. The filter (1600) may include non-zero filter coefficients and zero filter coefficients. The filter (1600) has a diamond shape (1620) formed by filter coefficients (1610) (shown as solid circles). In one example, non-zero filter coefficients in the filter (1600) are included in the filter coefficients (1610), and filter coefficients not included in the filter coefficients (1610) are zero. Thus, non-zero filter coefficients in the filter (1600) are included in the diamond shape (1620), and filter coefficients not included in the diamond shape (1620) are zero. In one example, the number of filter coefficients in the filter (1600) is equal to the number of filter coefficients (1610), which is 18 in the example shown in FIG. 16.

[0135] The CC-ALF may include any suitable filter coefficients (also referred to as CC-ALF filter coefficients). Referring back to Figure 15, the CC-ALF (1521) and the CC-ALF (1531) may have the same filter shape, such as the diamond shape (1620) shown in Figure 16, and the same number of filter coefficients. In one example, the values ​​of the filter coefficients in the CC-ALF (1521) are different from the values ​​of the filter coefficients in the CC-ALF (1531).

[0136] In general, filter coefficients in CC-ALF (e.g., non-zero filter coefficients) can be transmitted, for example, in APS. In one example, the filter coefficients are multiplied by a factor (e.g., 2 10) and may be rounded for fixed-point representation. The application of CC-ALF is controlled for variable block sizes and may be signaled by a context coding flag (e.g., a CC-ALF enable flag) received for each block of samples. Context coding flags such as the CC-ALF enable flag may be signaled at any appropriate level, such as the block level. The block size along with the CC-ALF enable flag may be received at the slice level for each chroma component. In some examples, block sizes (in chroma samples) of 16x16, 32x32, and 64x64 may be supported.

[0137] 17 shows an example syntax for CC-ALF according to some embodiments of the present disclosure. In the example of FIG. 17, alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is an index that indicates whether an inter-component Cb filter is used, and if so, the index of the inter-component Cb filter. For example, when alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is equal to 0, no cross-component Cb filter is applied to the block of Cb color component samples at luma location (xCtb, yCtb), and when alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is not equal to 0, alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is the index for the filter to be applied. For example, the alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]th cross-component Cb filter is applied to the block of Cb color component samples at luminance location (xCtb, yCtb).

[0138] Furthermore, in the example of FIG. 17, alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is used to indicate whether an inter-component Cr filter is used, and if so, is the index of the inter-component Cr filter. For example, when alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is equal to 0, no inter-component Cr filter is applied to the block of Cr color component samples at luma location (xCtb, yCtb), and when alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is not equal to 0, alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is the index of the inter-component Cr filter. ctb_ The cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]th cross-component Cr filter may be applied to a block of Cr color component samples at luma location (xCtb, yCtb).

[0139] In some examples, a chroma subsampling technique is used, so that the number of samples in each of the chroma blocks can be less than the number of samples in the luma block. The chroma subsampling format (also referred to as the chroma subsampling format and specified, for example, by chroma_format_idc) can indicate a chroma horizontal subsampling factor (e.g., SubWidthC) and a chroma vertical subsampling factor (e.g., SubHeightC) between each of the chroma blocks and the corresponding luma block. In one example, the chroma subsampling format is 4:2:0, so that the chroma horizontal subsampling factor (e.g., SubWidthC) and the chroma vertical subsampling factor (e.g., SubHeightC) are 2, as shown in Figures 18A-18B. In one example, the chroma subsampling format is 4:2:2, so that the chroma horizontal subsampling factor (e.g., SubWidthC) is 2 and the chroma vertical subsampling factor (e.g., SubHeightC) is 1. In one example, the chroma subsampling format is 4:4:4, and therefore the chroma horizontal subsampling factor (e.g., SubWidthC) and the chroma vertical subsampling factor (e.g., SubHeightC) are 1. The chroma sample type (also called the chroma sample position) may indicate the relative position of a chroma sample in a chroma block relative to at least one corresponding luma sample in a luma block.

[0140] 18A-18B illustrate exemplary locations of chroma samples relative to luma samples, according to an embodiment of the present disclosure. Referring to FIG. 18A, luma samples (1801) are located in rows (1811)-(1818). The luma samples (1801) shown in FIG. 18A may represent a portion of a picture. In one example, a luma block (e.g., luma CB) includes a luma sample (1801). The luma block may correspond to two chroma blocks with a chroma subsampling format of 4:2:0. In one example, each chroma block includes a chroma sample (1803). Each chroma sample (e.g., chroma sample (1803(1))) corresponds to four luma samples (e.g., luma samples (1801(1)) to (1801(4))). In one example, the four luma samples are the top left sample (1801(1)), the top right sample (1801(2)), the bottom left sample (1801(3)), and the bottom right sample (1801(4)). A chroma sample (e.g., (1803(1))) corresponds to the top left sample (1801(1)) and the bottom left sample (18 The chroma sample type of the chroma block with chroma sample (1803) located at the center left position between the top left sample (1801(1)) and the bottom left sample (1801(3)) can be referred to as chroma sample type 0. Chroma sample type 0 indicates relative position 0, which corresponds to the center left position halfway between the top left sample (1801(1)) and the bottom left sample (1801(3)). The four luma samples (e.g., (1801(1)) through (1801(4))) can be referred to as neighboring luma samples of chroma sample (1803)(1).

[0141] In one example, each chroma block includes a chroma sample (1804). The above description referring to the chroma sample (1803) may be adapted to the chroma sample (1804), and therefore detailed descriptions may be omitted for brevity. Each of the chroma samples (1804) may be located at a central position of four corresponding luma samples, and the chroma sample type of a chroma block having the chroma sample (1804) may be referred to as chroma sample type 1. Chroma sample type 1 indicates relative position 1, which corresponds to the central position of the four luma samples (e.g., (1801(1)) to (1801(4))). For example, one of the chroma samples (1804) may be located in the central portion of the luma samples (1801(1)) to (1801(4)).

[0142] In one example, each saturation block includes a saturation sample (1805). Each saturation sample (1805) may be located at the top left position, which is the same as the top left sample of the four corresponding luma samples (1801). The saturation sample type of a saturation block having a saturation sample (1805) may be referred to as saturation sample type 2. Therefore, each saturation sample (1805) is located at the same position as the top left sample of the four luma samples (1801) corresponding to the respective saturation sample. The saturation sample type 2 indicates a relative position 2, which corresponds to the top left position of the four luma samples (1801). For example, one of the saturation samples (1805) may be located at the top left position of luma samples (1801(1)) to (1801(4)).

[0143] In one example, each chroma block includes chroma samples (1806). Each of the chroma samples (1806) may be located at an upper center position between a corresponding upper-left sample and a corresponding upper-right sample, and the chroma sample type of the chroma block having the chroma sample (1806) may be referred to as chroma sample type 3. Chroma sample type 3 indicates relative position 3, which corresponds to an upper center position between the upper-left sample and the upper-right sample. For example, one of the chroma samples (1806) may be located at an upper center position of the luma samples (1801(1)) to (1801(4)).

[0144] In one example, each chroma block includes a chroma sample (1807). Each chroma sample (1807) may be located at the bottom left position, which is the same position as the bottom left sample of the four corresponding luma samples (1801). The chroma sample type of a chroma block having a chroma sample (1807) may be referred to as chroma sample type 4. Therefore, each chroma sample (1807) is located at the same position as the bottom left sample of the four luma samples (1801) corresponding to the respective chroma sample. Chroma sample type 4 indicates relative position 4, which corresponds to the bottom left position of the four luma samples (1801). For example, one of the chroma samples (1807) may be located at the bottom left position of luma samples (1801(1)) to (1801(4)).

[0145] In one example, each chroma block includes chroma samples (1808). Each of the chroma samples (1808) is located at a lower center position between the lower left sample and the lower right sample, and the chroma sample type of the chroma block including the chroma samples (1808) can be referred to as chroma sample type 5. Chroma sample type 5 indicates relative position 5, which corresponds to a lower center position between the lower left sample and the lower right sample of the four luma samples (1801). For example, one of the chroma samples (1808) can be located between the lower left sample and the lower right sample of luma samples (1801(1)) to (1801(4)).

[0146] In general, any suitable chroma sample type can be used for a chroma subsampling format. Chroma sample types 0 through 5 are exemplary chroma sample types described for chroma subsampling format 4:2:0. Additional chroma sample types can be used for chroma subsampling format 4:2:0. Additionally, other chroma sample types and / or variations of chroma sample types 0 through 5 can be used for other chroma subsampling formats, such as 4:2:2, 4:4:4, etc. In one example, a chroma sample type combining chroma samples (1805) and (1807) is used for chroma subsampling format 4:2:2.

[0147] In one example, the luminance blocks are each composed of the top two samples (e.g., (1801(1)) to (1801(4))) of the four luminance samples (e.g., (1801(1)) to (1801(4))). 1 (2))), and alternating rows such as rows (1811)-(1812) containing the bottom two samples (e.g., (1801(3))-(1801(4))) of the four luminance samples (e.g., (1801(1))-(1801(4))). Thus, rows (1811), (1813), (1815), and (1817) can be referred to as the current rows (also called the top field), and rows (1812), ( The four luminance samples (e.g., 1801(1)) through 1801(4)) are located in the current row (e.g., 1811) and the next row (e.g., 1812). Relative positions 2 through 3 are located in the current row, relative positions 0 through 1 are located between each current row and the respective next row, and relative positions 4 through 5 are located in the next row.

[0148] Chroma samples 1803, 1804, 1805, 1806, 1807, or 1808 are located in rows 1851-1854 within each chroma block. The specific locations of rows 1851-1854 may depend on the chroma sample's chroma sample type. For example, for chroma samples 1803-1804 with chroma sample types 0-1, respectively, row 1851 is located between rows 1811-1812. For chroma samples 1805-1806 with chroma sample types 2-3, respectively, row 1851 is in the same position as the current row 1811. For saturation samples 1807-1808, which have respective saturation sample types 4-5, row 1851 is in the same position as the next row 1812. The above description can be appropriately adapted to rows 1852-1854, and a detailed description will be omitted for the sake of brevity.

[0149] Any suitable scanning method may be used to display, store, and / or transmit the luma blocks and corresponding chroma blocks described above in Figure 18A. In one example, progressive scanning is used.

[0150] Interlaced scanning can be used, as shown in Figure 18B. As described above, the chroma subsampling format is 4:2:0 (e.g., chroma_format_idc equals 1). In one example, the variable chroma location type (e.g., ChromaLocType) indicates either the current row (e.g., ChromaLocType is chroma_sample_loc_type_top_field) or the next row (e.g., ChromaLocType is chroma_sample_loc_type_bottom_field). The current rows (1811), (1813), (1815), and (1817) and the next rows (1812), (1814), (1816), and (1818) can be scanned separately, for example, the current rows (1811), (1813), (1815), and (1817) can be scanned first, followed by the next rows (1812), (1814), (1816), and (1818). The current row can include luminance samples (1801), and the next row can include luminance samples (1802).

[0151] Similarly, corresponding chroma blocks can be interlaced. Rows 1851 and 1853, which contain chroma samples 1803, 1804, 1805, 1806, 1807, or 1808 with no fill, can be referred to as the current row (or current chroma row), and rows 1852 and 1854, which contain chroma samples 1803, 1804, 1805, 1806, 1807, or 1808 with gray fill, can be referred to as the next row (or next chroma row). In one example, during interlaced scanning, rows 1851 and 1853 are scanned first, followed by rows 1852 and 1854.

[0152] In some examples, a constrained directional enhancement filtering technique can be used. The use of an in-loop constrained directional enhancement filter (CDEF) can remove coding artifacts while preserving image details. In one example (e.g., HEVC), a sample adaptive offset (SAO) algorithm can achieve a similar objective by defining signal offsets for different classes of pixels. Unlike SAO, the CDEF is a nonlinear spatial filter. In some examples, the CDEF can be constrained to be easily vectorizable (i.e., implementable with single instruction multiple data (SIMD) operations). Note that other nonlinear filters, such as median filters, bilateral filters, etc., cannot be treated similarly.

[0153] In some cases, the amount of ringing artifacts in a coded image tends to be roughly proportional to the quantization step size. While the amount of detail is a characteristic of the input image, the smallest detail retained in the quantized image also tends to be proportional to the quantization step size. For a given quantization step size, the amplitude of the ringing is generally smaller than the amplitude of the detail.

[0154] CDEF can be used to identify the orientation of each block and then adaptively filter along the identified orientation and to a lesser extent along orientations rotated 45 degrees from the identified orientation. In some examples, the encoder can look up the filter strength, or the filter strength can be signaled explicitly, allowing for a high degree of control over the blurring.

[0155] Specifically, in some examples, the direction search is performed on the restored pixels immediately after the deblocking filter. Because these pixels are available to the decoder, the direction can be searched by the decoder; therefore, in one example, the direction does not require signaling. In some examples, the direction search can operate on a specific block size, such as an 8x8 block, which is small enough to properly handle non-linear edges but large enough to reliably estimate direction when applied to a quantized image. Having a consistent directionality over an 8x8 region also facilitates vectorization of the filter. In some examples, each block (e.g., 8x8) can be compared to a fully directional block to determine the difference. A fully directional block is a block in which all pixels along a line in a certain direction have the same value. In one example, a difference measure between the block and the fully directional block, such as sum of squared differences (SSD) or root mean square (RMS) error, can be calculated. The fully directional block with the smallest difference (e.g., minimum SSD, minimum RMS, etc.) can then be determined, and the direction of the determined fully directional block can be the direction that best matches the pattern within the block.

[0156] FIG. 19 shows an example of a direction search according to one embodiment of the present disclosure. In this example, block (1910) is an 8x8 block that has been restored and output from the deblocking filter. In the example of FIG. 19, the direction search can determine a direction from eight directions indicated by (1920) for block (1910). Eight fully directional blocks (1930) are formed corresponding to each of the eight directions (1920). A fully directional block corresponding to a direction is a block whose pixels along the line of that direction have the same value. Furthermore, difference measures such as SSD and RMS error can be calculated for each of block (1910) and fully directional block (1930). In the example of FIG. 19, the RMS error is indicated by (1940). As indicated by (1943), the RMS error between block (1910) and fully directional block (1933) is the smallest, and therefore direction (1923) is the direction that best matches the pattern in block (1910).

[0157] After the direction of the block is identified, a nonlinear low-pass directional filter can be determined. For example, the filter taps of the nonlinear low-pass directional filter can be aligned along the identified direction to reduce ringing while preserving the directional edge or directional pattern. However, in some examples, directional filtering alone may not be sufficient to reduce ringing. In one example, additional filter taps are used for pixels that are not aligned with the identified direction. To reduce the risk of blurring, the extra filter taps are treated more conservatively. For this reason, the CDEF includes first-order and second-order filter taps. In one example, a complete 2-D CDEF filter is expressed as Equation (14):

number

[0158] In some examples, in addition to the deblocking operation, in-loop restoration schemes are used in post-deblocking video coding, generally to remove noise and delineate and improve edge quality. In one example, the in-loop restoration schemes are switchable within a frame for tiles of appropriate size. The in-loop restoration schemes are based on a separable symmetric Wiener filter, a dual self-guided filter with subspace projection, and a domain transform recursive filter. Because content statistics can change significantly within a frame, the in-loop restoration schemes are integrated within a switchable framework that can trigger different schemes within different regions of a frame.

[0159] A separable symmetric Wiener filter can be one of the in-loop restoration methods. In some examples, every pixel in a degraded frame can be restored as a non-causally filtered version of the pixels in a w × w window around it, where w = 2r + 1 is odd with respect to the integer r. The 2D filter taps are expressed as w in column vectorized form. 2 If the vector is represented by a 1 × 1 element vector F, then the direct LMMSE optimization is F=H -1 leading to filter parameters given by M, where H=E[XX T ] is the autocovariance of x and w in a w × w window around the pixel 2 is a column vectorized version of the samples, where M=E[YX T] is the cross-correlation of x and the scalar source sample y to be estimated. In one example, the encoder can estimate H and M from realizations in the deblocked frame and the source, and send the resulting filter F to the decoder. However, doing so would 2 Not only does transmitting these taps incur a significant bitrate cost, but non-separable filtering makes decoding prohibitively complex. In some embodiments, several additional constraints are placed on the nature of F. The first constraint constrains F to be separable, so that the filtering can be implemented as a separable horizontal and vertical w-tap convolution. The second constraint constrains each of the horizontal and vertical filters to be symmetric. The third constraint assumes that both the horizontal and vertical filter coefficients sum to one.

[0160] Dual self-guided filtering with subspace projection can be one of the in-loop restoration methods. Self-guided filtering is an image filtering technique in which a locally linear model, given by equation (15), is used to calculate the filtered output y from the unfiltered samples x, where y=Fx+G Equation (15) Here, F and G are determined based on the statistics of the guidance image in the neighborhood of the degraded image and the filtered pixel. If the guide image is the same as the degraded image, the resulting so-called self-guided filtering has the effect of edge-preserving smoothing. In one example, a specific form of self-guided filtering can be used. The specific form of self-guided filtering depends on two parameters: radius r and noise parameter e, and is enumerated as follows: 1. The mean μ and variance σ of pixels in a (2r+1) × (2r+1) window around every pixel 2 This step can be efficiently implemented with box filtering based on integral imaging. 2. f=σ for all pixels2 / (σ 2 +e), and calculate g = (1 - f)μ. 3. Calculate F and G for every pixel as the average of the f and g values ​​in a 3x3 window around the pixel you want to use.

[0161] The specific form of self-guided filtering is controlled by r and e, with larger r resulting in larger spatial variance and larger e resulting in larger range variance.

[0162] Figure 20 shows an example showing subspace projection in some cases. As shown in Figure 20, even if neither of the reconstructions X1, X2 is close to the source Y, a suitable multiplier {α, β} can bring them quite close to the source as long as they are somewhat moved in the right direction.

[0163] In some examples (e.g., HEVC), a filtering technique called sample adaptive offset (SAO) can be used. In some examples, SAO is applied to the reconstructed signal after the deblocking filter. SAO can use an offset value provided in the slice header. In some examples, for luma samples, the encoder can determine whether to apply (enable) SAO to the slice. When SAO is enabled, the current picture allows a recursive division of the coding unit into four sub-regions, and each sub-region can select an SAO type from multiple SAO types based on features within the sub-region.

[0164] FIG. 21 shows a table (2100) of multiple SAO types according to one embodiment of the present disclosure. Table (2100) shows SAO types 0 to 6. Note that SAO type 0 is used to indicate that no SAO is applied. Furthermore, each SAO type, SAO type 1 to SAO type 6, includes multiple categories. SAO can reduce distortion by classifying reconstructed pixels of a subregion into categories and adding an offset to pixels of each category within the subregion. In some examples, edge characteristics can be used to classify pixels in SAO types 1 to 4, and pixel intensity can be used to classify pixels in SAO types 5 and 6.

[0165] Specifically, in one embodiment, such as SAO Types 5-6, a band offset (BO) can be used to classify all pixels in a subregion into multiple bands. Each band of the multiple bands includes pixels within the same intensity interval. In some examples, the intensity range is equally divided into multiple intervals, such as 32 intervals from 0 to the maximum intensity value (e.g., 255 for 8-bit pixels), and each interval is associated with an offset. Furthermore, in one example, the 32 bands are divided into two groups, such as a first group and a second group. The first group includes the central 16 bands (e.g., 16 intervals in the middle of the intensity range), and the second group includes the remaining 16 bands (e.g., 8 intervals at the low end of the intensity range and 8 intervals at the high end of the intensity range). In one example, only the offset of one of the two groups is transmitted. In some embodiments, when a pixel classification operation in BO is used, the five most significant bits of each pixel can be directly used as a band index.

[0166] Additionally, in one embodiment, such as SAO Types 1-4, edge offset (EO) can be used to determine pixel classification and offset. For example, pixel classification can be determined based on a one-dimensional three-pixel pattern taking into account edge directionality information.

[0167] Figure 22 shows an example of a three-pixel pattern for pixel classification at edge offsets in some examples. In the example of Figure 22, the first pattern (2210) (shown by three gray pixels) is referred to as a 0-degree pattern (the 0-degree pattern is associated with a horizontal direction), the second pattern (2220) (shown by three gray pixels) is referred to as a 90-degree pattern (the 90-degree pattern is associated with a vertical direction), the third pattern (2230) (shown by three gray pixels) is referred to as a 135-degree pattern (the 135-degree pattern is associated with a 135-degree diagonal direction), and the fourth pattern (2240) (shown by three gray pixels) is referred to as a 45-degree pattern (the 45-degree pattern is associated with a 45-degree diagonal direction). In one example, one of the four directivity patterns shown in Figure 22 can be selected taking into account edge directivity information of the subregion. The selection can be transmitted in one example as side information within the coded video bitstream. The pixels within the sub-region can then be classified into multiple categories by comparing each pixel with its two neighboring pixels in the direction associated with the directional pattern.

[0168] 23 is a table (2300) for pixel classification rules for edge offsets in some examples. Specifically, pixel c (also shown in each pattern in FIG. 22) is compared with two adjacent pixels (also shown in gray in each pattern in FIG. 22), and pixel c can be classified into one of categories 0 to 4 based on the comparison according to the pixel classification rules shown in FIG. 23.

[0169] In some embodiments, decoder-side SAO can operate independently of the largest coding unit (LCU) (e.g., CTU) to conserve line buffers. In some examples, pixels in the top and bottom rows within each LCU are not SAO processed when the 90-degree, 135-degree, and 45-degree classification patterns are selected, and pixels in the leftmost and rightmost columns within each LCU are not SAO processed when the 0-degree, 135-degree, and 45-degree patterns are selected.

[0170] FIG. 24 shows an example (2400) of syntax that may need to be signaled for a CTU when parameters are not merged from adjacent CTUs. For example, the syntax element sao_type_idx[cldx][rx][ry] can be signaled to indicate the SAO type of the subregion. The SAO type may be BO (Band Offset) or EO (Edge Offset). When sao_type_idx[cldx][rx][ry] has a value of 0, it indicates that SAO is OFF. Values ​​of 1 to 4 indicate that one of four EO categories corresponding to 0°, 90°, 135°, and 45° is used. A value of 5 indicates that BO is used. In the example of FIG. 24, each of the BO and EO types has four SAO offset values ​​to be signaled: sao_offset[cIdx][rx][ry][0] through sao_offset[cIdx][rx][ry][3].

[0171] In general, the filtering process can use a reconstructed sample of a first color component as an input (e.g., Y or Cb or Cr, or R or G or B) to generate an output, and the output of the filtering process is applied to a second color component, which can be the same color component as the first color component or another color component different from the first color component.

[0172] In a related example of cross-component filtering (CCF), filter coefficients are derived based on several mathematical equations. The derived filter coefficients are signaled from the encoder side to the decoder side, and the derived filter coefficients are used to generate offsets using a linear combination. The generated offsets are then added to the reconstructed samples as a filtering process. For example, an offset is generated based on a linear combination of the filtering coefficients with the luma samples, and the generated offsets are added to the reconstructed chroma samples. The related example of CCF is based on the assumption of a linear mapping relationship between the reconstructed luma sample values ​​and the delta values ​​between the original chroma samples and the reconstructed chroma samples. However, the mapping between the reconstructed luma sample values ​​and the delta values ​​between the original chroma samples and the reconstructed chroma samples does not necessarily follow a linear mapping process. Therefore, the coding performance of CCF may be limited under the assumption of a linear mapping relationship.

[0173] In some examples, the nonlinear mapping technique can be used in inter-component filtering and / or same-color component filtering without significant signaling overhead. In one example, the nonlinear mapping technique can be used in inter-component filtering to generate inter-component sample offsets. In another example, the nonlinear mapping technique can be used in same-color component filtering to generate local sample offsets.

[0174] For convenience, a filtering process using a nonlinear mapping technique can be referred to as sample offset with nonlinear mapping (SO-NLM). SO-NLM in the inter-component filtering process can be referred to as inter-component sample offset (CCSO). SO-NLM in same-color component filtering can be referred to as local sample offset (LSO). A filter using a nonlinear mapping technique can be referred to as a nonlinear mapping-based filter. Nonlinear mapping-based filters can include CCSO filters, LSO filters, etc.

[0175] In one example, CCSO and LSO can be used as loop filtering to reduce distortion of reconstructed samples. CCSO and LSO do not rely on the assumption of linear mapping used in the associated exemplary CCF. For example, CCSO does not rely on the assumption of a linear mapping relationship between luma reconstructed sample values ​​and delta values ​​between original chroma samples and chroma reconstructed samples. Similarly, LSO does not rely on the assumption of a linear mapping relationship between reconstructed sample values ​​of color components and delta values ​​between original samples of color components and reconstructed samples of color components.

[0176] In the following description, an SO-NLM filtering process is described that uses a reconstructed sample of a first color component as an input (e.g., Y or Cb or Cr, or R or G or B) to generate an output, and the output of the filtering process is applied to a second color component. When the second color component is the same color component as the first color component, the description is applicable to LSO, and when the second color component is different from the first color component, the description is applicable to CCSO.

[0177] In SO-NLM, a nonlinear mapping is derived on the encoder side. The nonlinear mapping is between the reconstructed sample of the first color component in the filter support region and an offset added to the second color component in the filter support region. When the second color component is the same as the first color component, the nonlinear mapping is used in LSO. When the second color component is different from the first color component, the nonlinear mapping is used in CCSO. The region of the nonlinear mapping is determined by the different combinations of processed input reconstructed samples (also called possible reconstructed sample value combinations).

[0178] The SO-NLM technique can be illustrated using a specific example, in which a reconstruction sample from a first color component located within a filter support area (also called a "filter support region") is determined. The filter support area is an area to which a filter can be applied, and the filter support area can have any suitable shape.

[0179] FIG. 25 shows an example of a filter support area (2500) according to some embodiments of the present disclosure. The filter support area (2500) includes four restored samples of a first color component: P0, P1, P2, and P3. In the example of FIG. 25, the four restored samples may form a cross in the vertical and horizontal directions, with the center of the cross being the location for the sample to be filtered. The sample at the center and of the same color component as P0-P3 is denoted by C. The sample at the center and of the second color component is denoted by F. The second color component may be the same as the first color component P0-P3 or may be different from the first color component P0-P3.

[0180] FIG. 26 shows an example of another filter support area (2600) according to some embodiments of the present disclosure. The filter support area (2600) includes four restored samples P0, P1, P2, and P3 of a first color component that form a square. In the example of FIG. 26, the center position of the square is the position of the sample to be filtered. The sample at the center and of the same color component as P0-P3 is represented by C. The sample at the center and of the second color component is represented by F. The second color component may be the same as the first color component P0-P3 or may be different from the first color component P0-P3.

[0181] The reconstructed samples are input to the SO-NLM filter and processed appropriately to form the filter taps. In one example, the positions of the reconstructed samples that are input to the SO-NLM filter are called filter tap positions. In a specific example, the reconstructed samples are processed in two steps:

[0182] In the first step, the delta values ​​between P0 to P3 and C are calculated. For example, m0 represents the delta value between P0 and C, m1 represents the delta value between P1 and C, m2 represents the delta value between P2 and C, and m3 represents the delta value between P3 and C.

[0183] In a second step, the delta values ​​m0-m3 are further quantized, and the quantized values ​​are denoted as d0, d1, d2, and d3. In one example, the quantized value may be one of -1, 0, or 1 based on the quantization process. For example, when m is less than -N (N is a positive value and is called the quantization step size), the value m may be quantized to -1; when m is in the range [-N,N], the value m may be quantized to 0; and when m is greater than N, the value m may be quantized to 1. In some examples, the quantization step size N may be one of 4, 8, 12, 16, etc.

[0184] In some embodiments, the quantized values ​​d0-d3 are filter taps and can be used to identify one combination within a filter domain. For example, the filter taps d0-d3 can form a combination within a filter domain. Each filter tap can have three quantized values, so when four filter taps are used, the filter domain contains 81 (3 x 3 x 3 x 3) combinations.

[0185] 27A-27C show a table 2700 having 81 combinations according to one embodiment of the present disclosure. The table 2700 includes 81 rows corresponding to the 81 combinations. In each row corresponding to a combination, the first column includes an index of the combination, the second column includes a value of the filter tap d0 for the combination, the third column includes a value of the filter tap d1 for the combination, the fourth column includes a value of the filter tap d2 for the combination, the fifth column includes a value of the filter tap d3 for the combination, and the sixth column includes an offset value associated with the combination for the nonlinear mapping. In one example, once the filter taps d0-d3 are determined, an offset value (denoted by s) associated with the combination of d0-d3 can be determined according to the table 2700. In one example, the offset values ​​s0-s80 are integers such as 0, 1, -1, 3, -3, 5, -5, and -7.

[0186] In some embodiments, the final filtering process of the SO-NLM can be applied as shown in equation (16): f'=clip(f+s) Equation (16) where f is the reconstruction sample of the second color component to be filtered, and s is an offset value determined according to the filter taps that are the processing result of the reconstruction sample of the first color component, such as using table (2700). The sum of the reconstruction sample F and the offset value s is further clipped to a range associated with the bit depth to determine the final filtered sample f' of the second color component.

[0187] Please note that in the case of LSO, the second color component in the above description is the same as the first color component, and in the case of CCSO, the second color component in the above description may be different from the first color component.

[0188] It should be noted that the above description may be adjusted for other embodiments of the present disclosure.

[0189] In some examples, at the encoder side, the encoding device may derive a mapping between the reconstructed samples of the first color component within the filter support region and the offsets added to the reconstructed samples of the second color component. The mapping may be any suitable linear or nonlinear mapping. Then, the filtering process may be applied at the encoder side and / or the decoder side based on the mapping. For example, the mapping may be appropriately notified to the decoder (e.g., the mapping may be included in the coded video bitstream transmitted from the encoder side to the decoder side), and then the decoder may perform the filtering process based on the mapping.

[0190] The performance of nonlinear mapping-based filters, such as CCSO filters and LSO filters, depends on the filter shape configuration. The filter shape configuration (also called filter shape) of a filter can refer to the characteristics of the pattern formed by the filter tap positions. The pattern can be defined by various parameters such as the number of filter taps, the geometric shape of the filter tap positions, and the distance of the filter tap positions to the center of the pattern. Using a fixed filter shape configuration can limit the performance of nonlinear mapping-based filters.

[0191] As illustrated by Figures 24 and 25 and Figures 27A-27C, some examples use a 5-tap filter design for filter shape configuration of a nonlinear mapping-based filter. The 5-tap filter design may use tap positions P0, P1, P2, P3, and C. The 5-tap filter design for filter shape configuration may result in a look-up table (LUT) with 81 entries, as illustrated in Figures 27A-27C. The LUT of sample offsets needs to be signaled from the encoder side to the decoder side, and signaling the LUT contributes to a large amount of signaling overhead and may affect coding efficiency using a nonlinear mapping-based filter. According to some aspects of the present disclosure, the number of filter taps may be different from 5. In some examples, the number of filter taps may be reduced, which may further incorporate information within the filter support area and improve coding efficiency.

[0192] In some examples, the filter shape configurations in a group for nonlinear mapping-based filters each have three filter taps.

[0193] 28 illustrates seven filter shape configurations of three filter taps in one example. Specifically, a first filter shape configuration includes three filter taps at positions labeled "1" and "c," with position "c" being the center position of position "1," a second filter shape configuration includes three filter taps at positions labeled "2" and position "c," with position "c" being the center position of position "2," a third filter shape configuration includes three filter taps at positions labeled "3" and position "c," with position "c" being the center position of position "3," and a fourth filter shape configuration is labeled "4" and position "c." The fifth filter shape configuration includes three filter taps at positions labeled "5" and "c", with position "c" being the center position of position "5", the sixth filter shape configuration includes three filter taps at positions labeled "6" and position "c", with position "c" being the center position of position "6", and the seventh filter shape configuration includes three filter taps at positions labeled "7" and position "c", with position "c" being the center position of position "7".

[0194] According to some aspects of the present disclosure, the nonlinear mapping-based filter can be used with other in-loop filters in a loop filter chain. The location of the nonlinear mapping-based filter can affect the coding efficiency of the nonlinear mapping-based filter.

[0195] Figure 29 shows a block diagram of a loop filter chain (2900) in some examples. The loop filter chain (2900) includes multiple filters connected in series within a filter chain. In one example, the loop filter chain (2900) can be used as the loop filter unit (556). The loop filter chain (2900) can be used in an encoding loop or a decoding loop before storing reconstructed pictures in a decoded picture buffer, such as the reference picture memory (557). The loop filter chain (2900) receives input reconstructed samples from a previous processing module and applies filters to the reconstructed samples to generate output reconstructed samples.

[0196] The loop filter chain (2900) can include any suitable filter. In the example of FIG. 29, the loop filter chain (2900) includes a deblocking filter (labeled DEBLOCKING), a constrained directional enhancement filter (labeled CDEF), and an in-loop reconstruction filter (labeled LR) connected in the chain. The loop filter chain (2900) has an input node (2901), an output node (2909), and multiple intermediate nodes (2902)-(2903). The input node (2901) of the loop filter chain (2900) receives input reconstruction samples from a previous processing module, which are provided to the deblocking filter. The intermediate node (2902) receives the reconstruction samples from the deblocking filter (after being processed by the deblocking filter) and provides the reconstruction samples to the CDEF for further filtering. The intermediate node (2903) receives the reconstructed samples from the CDEF (after being processed by the CDEF) and provides the reconstructed samples to the LR filter for further filtering. The output node (2909) receives the output reconstructed samples from the LR filter (after being processed by the LR filter). The output reconstructed samples can be provided to other processing modules, such as a post-processing module, for further processing.

[0197] Note that the following description illustrates a technique for using a nonlinear mapping-based filter based on a loop filter chain (2900). The technique for using a nonlinear mapping-based filter can be used in other suitable loop filter chains.

[0198] According to one aspect of the present disclosure, the nonlinear mapping-based filter may be serially coupled with other filters in a loop filter chain, and the input and output of the nonlinear mapping-based filter may be located at the same node of the loop filter chain, with no other filters present between the input and output of the nonlinear mapping-based filter. For example, the nonlinear mapping-based filter may receive a reconstructed sample at a node of the loop filter chain, determine a sample offset based on the reconstructed sample at the node of the loop filter chain, and then apply the sample offset at the reconstructed sample at the node of the loop filter chain.

[0199] 30A-30D show examples of loop filter chains that include a nonlinear mapping-based filter coupled in series with other filters in the loop filter chain.

[0200] FIG. 30A shows an example of a loop filter chain (3000A) in one example. The loop filter chain (3000A) can be used in place of the loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3000A), a nonlinear mapping-based filter (labeled SO-NLM) is applied at an input node. Specifically, the input (also referred to as a first reconstructed sample) of the nonlinear mapping-based filter is an input reconstructed sample (to the loop filter chain (3000A)) at a first node (3011A). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with an intermediate reconstructed sample at a second node (3012A) to generate an output (also referred to as a second reconstructed sample) at a third node (3013A), which is provided to a deblocking filter for further filtering. In the example of FIG. 30A, the first node (3011A) and the second node (3012A) are the same node, and the intermediate reconstructed sample is the first reconstructed sample.

[0201] FIG. 30B shows an example of a loop filter chain (3000B) in one example. The loop filter chain (3000B) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3000B), a nonlinear mapping-based filter (labeled SO-NLM) is applied at an intermediate node of the loop filter chain (3000B). Specifically, the input (also referred to as the first reconstructed sample) of the nonlinear mapping-based filter is the reconstructed sample (generated by the deblocking filter) at the first node (3011B). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample at the second node (3012B) to generate an output (also referred to as the second reconstructed sample) at the third node (3013B), which is provided to the CDEF for further filtering. In the example of FIG. 30B, the first node (3011B) and the second node (3012B) are the same node, and the intermediate reconstructed sample is the first reconstructed sample.

[0202] FIG. 30C shows an example of a loop filter chain (3000C) in one example. The loop filter chain (3000C) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3000C), a nonlinear mapping-based filter (labeled SO-NLM) is applied at an intermediate node of the loop filter chain (3000C). Specifically, the input (also referred to as the first reconstructed sample) of the nonlinear mapping-based filter is the reconstructed sample (generated by the CDEF) at the first node (3011C). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample at the second node (3012C) to generate an output (also referred to as the second reconstructed sample) at the third node (3013C), which is provided to the LR filter for further filtering. In the example of FIG. 30C, the first node (3011C) and the second node (3012C) are the same node, and the intermediate reconstructed sample is the first reconstructed sample.

[0203] FIG. 30D shows an example of a loop filter chain (3000D) in one example. The loop filter chain (3000D) can be used in place of the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3000D), a nonlinear mapping-based filter (labeled SO-NLM) is applied at an output node. Specifically, the input (also referred to as the first reconstructed sample) of the nonlinear mapping-based filter is the reconstructed sample (generated by the LR filter) at the first node (3011D). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample at the second node (3012D) to generate an output (also referred to as the second reconstructed sample) at the third node (3013D), which is the output of the loop filter chain (3000D). In the example of FIG. 30D, the first node (3011D) and the second node (3012D) are the same node, and the intermediate reconstructed sample is the first reconstructed sample.

[0204] According to one aspect of the present disclosure, the nonlinear mapping-based filter can be coupled to one or more filters in parallel in a loop filter chain, with at least one filter between the input and output of the nonlinear mapping-based filter. For example, the nonlinear mapping-based filter receives a reconstructed sample at a first node of the loop filter chain, determines a sample offset based on the reconstructed sample at the first node of the loop filter chain, and then applies the sample offset at the reconstructed sample at a second node of the loop filter chain. The reconstructed sample at the second node of the loop filter chain can be obtained by applying one or more filters to the reconstructed sample at the first node of the loop filter chain.

[0205] In some examples, the input of the nonlinear mapping-based filter is the reconstructed samples located after the deblocking filter and before the CDEF, and the output of the nonlinear mapping-based filter is applied to the reconstructed samples after the CDEF and before the LR filter or after the LR filter.

[0206] FIG. 31A shows an example of a loop filter chain (3100A) including a nonlinear mapping-based filter coupled in parallel with a CDEF. The loop filter chain (3100A) can be used in place of the loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3100A), a nonlinear mapping-based filter (labeled SO-NLM) is applied between two intermediate nodes. Specifically, the input (also referred to as the first reconstructed sample) of the nonlinear mapping-based filter is the reconstructed sample (generated by the deblocking filter) at the first node (3111A). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample (generated by the CDEF) at the second node (3112A) to generate an output (also referred to as the second reconstructed sample) at the third node (3113A), which is provided to the LR filter for further filtering. In the example of Figure 31A, the CDEF is between the first node (3111A) and the second node (3112A). The intermediate reconstructed samples are the output of the CDEF.

[0207] FIG. 31B shows an example of a loop filter chain (3100B) including a nonlinear mapping-based filter coupled in parallel with a CDEF and an LR filter. The loop filter chain (3100B) can be used in place of the loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3100B), a nonlinear mapping-based filter (labeled SO-NLM) is applied between an intermediate node and an output node. Specifically, the input (also referred to as the first reconstructed sample) of the nonlinear mapping-based filter is the reconstructed sample (generated by the deblocking filter) at the first node (3111B). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample (generated by the LR filter) at the second node (3112B) to generate an output (also referred to as the second reconstructed sample) at the third node (3113B), which is the output of the loop filter chain (3100B). In the example of Figure 31B, the CDEF and LR filter are between the first node (3111B) and the second node (3112B). The intermediate reconstructed samples are the output of the LR filter.

[0208] In some examples, the input of the nonlinear mapping-based filter is the reconstructed samples located before the deblocking filter, and the output of the nonlinear mapping-based filter is applied to the reconstructed samples after the deblocking filter but before the CDEF, after the CDEF but before the LR filter, or after the LR filter.

[0209] FIG. 32A shows an example of a loop filter chain (3200A) including a nonlinear mapping-based filter coupled in parallel with a deblocking filter. The loop filter chain (3200A) can be used in place of the loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3200A), a nonlinear mapping-based filter (labeled SO-NLM) is applied between the input node and an intermediate node of the loop filter chain. Specifically, the input (also referred to as the first reconstructed sample) of the nonlinear mapping-based filter is the input reconstructed sample (input of the loop filter chain (3200A)) at the first node (3211A). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample (generated by the deblocking filter) at the second node (3212A) to generate an output (also referred to as the second reconstructed sample) at the third node (3213A), which is provided to the CDEF for further filtering. In the example of Figure 32A, the deblocking filter is between the first node (3211A) and the second node (3212A). The intermediate reconstructed samples are the output of the deblocking filter.

[0210] FIG. 32B shows an example of a loop filter chain (3200B) including a nonlinear mapping-based filter coupled in parallel with a deblocking filter and a CDEF. The loop filter chain (3200B) can be used in place of the loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3200B), a nonlinear mapping-based filter (labeled SO-NLM) is applied between the input node and an intermediate node of the loop filter chain. Specifically, the input (also referred to as the first reconstructed sample) of the nonlinear mapping-based filter is the input reconstructed sample (input of the loop filter chain (3200B)) at the first node (3211B). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample (generated by the CDEF) at the second node (3212B) to generate an output (also referred to as the second reconstructed sample) at the third node (3213B), which is provided to the LR for further filtering. In the example of Figure 32B, the deblocking filter and CDEF are between the first node (3211B) and the second node (3212B). The intermediate reconstructed samples are the output of the CDEF.

[0211] FIG. 32C shows an example of a loop filter chain (3200C) including a nonlinear mapping-based filter coupled in parallel with a deblocking filter, a CDEF, and an LR filter. The loop filter chain (3200C) can be used in place of the loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3200C), a nonlinear mapping-based filter (labeled SO-NLM) is applied between the input node and the output node of the loop filter chain. Specifically, the input (also called the first reconstructed sample) of the nonlinear mapping-based filter is applied to the input reconstructed sample (the first reconstructed sample) at the first node (3211C) of the loop filter chain (3200C). C) is the input). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample (generated by the LR filter) at the second node (3212C) to generate an output (also called the second reconstructed sample) at the third node (3213C), which is the output of the loop filter chain (3200C). In the example of Figure 32C, the deblocking filter, CDEF, and LR filter are between the first node (3211C) and the second node (3212C). The intermediate reconstructed sample is the output of the LR filter.

[0212] In some examples, the input of the nonlinear mapping-based filter is the reconstructed samples located after the CDEF and before the LR filter, and the output of the nonlinear mapping-based filter is applied to the reconstructed samples after the LR filter.

[0213] Figure 33 shows an example of a loop filter chain (3300) including a nonlinear mapping-based filter coupled in parallel with an LR filter. The loop filter chain (3300) can be used in place of the loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3300), a nonlinear mapping-based filter (labeled SO-NLM) is applied between an intermediate node and an output node of the loop filter chain. Specifically, the input (also referred to as the first reconstructed sample) of the nonlinear mapping-based filter is the reconstructed sample (generated by the CDEF) at the first node (3311). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the intermediate reconstructed sample (generated by the LR filter) at the second node (3312) to generate an output (also referred to as the second reconstructed sample) at the third node (3313), which is the output of the loop filter chain (3300). In the example of Figure 33, the LR filter is between the first node (3311) and the second node (3312). The intermediate reconstructed samples are the output of the LR filter.

[0214] According to another aspect of the present disclosure, multiple nonlinear mapping-based filters can be applied simultaneously at multiple locations in the loop filter chain, each of which can be configured as any of the examples shown in Figures 30A-30D, 31A-31B, 32A-32C, and 33.

[0215] FIG. 34 shows an example of a loop filter chain (3400) including a first nonlinear mapping-based filter denoted by SO-NLM1 and a second nonlinear mapping-based filter denoted by SO-NLM2. The loop filter chain (3400) can be used in place of the loop filter chain (2900) in an encoding or decoding device. The first nonlinear mapping-based filter is coupled in parallel with the CDEF in a configuration similar to the example shown in FIG. 31A. The second nonlinear mapping-based filter is coupled in series with other filters in the loop filter chain (3400) in a configuration similar to the example shown in FIG. 30D. In one example, the first and second nonlinear mapping-based filters are the same type of filter, such as CCSO or LSO. In another example, the first and second nonlinear mapping-based filters are different types of filters, such as one being CCSO and the other being LSO.

[0216] FIG. 35 shows an example of a loop filter chain (3500) including a first nonlinear mapping-based filter denoted by SO-NLM1 and a second nonlinear mapping-based filter denoted by SO-NLM2. The loop filter chain (3500) can be used in place of the loop filter chain (2900) in an encoding or decoding device. The first nonlinear mapping-based filter is coupled in parallel with the LR filter in a configuration similar to the example shown in FIG. 33. The second nonlinear mapping-based filter is coupled in series with other filters in the loop filter chain (3500) in a configuration similar to the example shown in FIG. 30B. In one example, the first and second nonlinear mapping-based filters are the same type of filter, such as CCSO or LSO. In another example, the first and second nonlinear mapping-based filters are different types of filters, such as one being CCSO and the other being LSO.

[0217] FIG. 36 shows a flowchart outlining a process (3600) according to one embodiment of the present disclosure. The process (3600) can be used for video filtering. When the term block is used, the block may be interpreted as a prediction block, a coding unit, a luma block, a chroma block, etc. In various embodiments, the process (3600) is performed by a processing circuit, such as a processing circuit within the terminal devices (310), (320), (330), and (340), a processing circuit performing the functions of the video encoder (403), a processing circuit performing the functions of the video decoder (410), a processing circuit performing the functions of the video decoder (510), or a processing circuit performing the functions of the video encoder (603). In some embodiments, the process (3600) is implemented in software instructions, and thus, the processing circuit performs the process (3600) when the processing circuit executes the software instructions. The process begins at (S3601) and proceeds to (S3610).

[0218] At (S3610), a first offset value for applying the nonlinear mapping-based filter is based on a first reconstructed sample at a first node along the loop filter chain.

[0219] At (S3620), the first offset value is applied to the intermediate reconstructed sample at a second node along the loop filter chain to generate a second reconstructed sample at a third node along the loop filter chain.

[0220] In one example, the nonlinear mapping-based filter is a component-to-component sample offset (CCSO) filter, and the intermediate reconstructed samples and the first reconstructed samples are samples of different color components.

[0221] In another example, the nonlinear mapping-based filter is a local sample offset (LSO) filter, and the intermediate reconstruction samples and the first reconstruction samples are samples of the same color component.

[0222] In some embodiments, the first node and the second node are of the same node, which may be an input node of the loop filter chain, or an output node of the loop filter chain, or an intermediate node of the loop filter chain.

[0223] In one example, the first reconstruction sample is generated by a processing module prior to the deblocking filter. In another example, the first reconstruction sample is generated by the deblocking filter. In another example, the first reconstruction sample is generated by a constrained directional extension filter. In another example, the first reconstruction sample is generated by a loop reconstruction filter.

[0224] In some embodiments, the first node and the second node are from different nodes. In some examples, the first restored sample is generated by a processing module prior to the deblocking filter, and the intermediate restored sample is generated by at least one of the deblocking filter, the constrained directional enhancement filter, or the loop restoration filter. In some examples, the first restored sample is generated by the deblocking filter, and the intermediate restored sample is generated by at least one of the constrained directional enhancement filter or the loop restoration filter. In some examples, the first restored sample is generated by the constrained directional enhancement filter, and the intermediate restored sample is generated by the loop restoration filter.

[0225] The process (3600) proceeds to (S3699) and ends.

[0226] It should be noted that in some examples, the nonlinear mapping-based filter is a component-to-component sample offset (CCSO) filter, and in some other examples, the nonlinear mapping-based filter is a local sample offset (LSO) filter.

[0227] The process (3600) may be adapted as appropriate. Steps of the process (3600) may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0228] The embodiments of the present disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), the encoder, and the decoder may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0229] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 37 illustrates a computer system (3700) suitable for implementing some embodiments of the disclosed subject matter.

[0230] Computer software can be coded using any suitable machine or computer language that can be assembled, compiled, linked, or similar mechanisms to create code containing instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., directly, or via interpretation, microcode execution, etc.

[0231] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0232] 37 for computer system (3700) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (3700).

[0233] The computer system (3700) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (such as keystrokes, swipes, or data glove movements), audio input (such as voice or clapping), visual input (such as gestures), or olfactory input (not depicted). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (such as voice, music, or ambient sounds), images (such as scanned images or photographic images obtained from a still camera), or video (such as two-dimensional video, three-dimensional video, including stereoscopic video).

[0234] The input human interface devices may include one or more of a keyboard (3701), a mouse (3702), a trackpad (3703), a touchscreen (3710), a data glove (not shown), a joystick (3705), a microphone (3706), a scanner (3707), and a camera (3708) (only one of each is depicted).

[0235] The computer system (3700) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (3710), data gloves (not shown), or joystick (3705), although haptic feedback devices that do not function as input devices may also be present), audio output devices (such as speakers (3709) or headphones (not depicted)), visual output devices (such as screens (3710), including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capability and each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or three-dimensional or higher-dimensional output via means such as stereographic output, virtual reality glasses (not depicted), holographic displays, and smoke tanks (not depicted)), and printers (not depicted).

[0236] The computer system (3700) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (3720) with CD / DVD or similar media (3721), thumb drives (3722), removable hard drives or solid state drives (3723), legacy magnetic media (not depicted) such as tape and floppy disks, and specialized ROM / ASIC / PLD-based devices (not depicted) such as security dongles.

[0237] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0238] The computer system (3700) may also include an interface (3754) to one or more communication networks (3755). The network may be, for example, wireless, wired, or optical. The network may further be local, wide area, metropolitan, vehicular, industrial, real-time, delay-tolerant, or the like. Examples of networks include local area networks such as Ethernet and wireless LAN; cellular networks including GSM, 3G, 4G, 5G, LTE, and the like; wired or wireless wide area digital networks for TV, including cable TV, satellite TV, and terrestrial broadcast TV; vehicular and industrial networks including CAN Bus; and the like. Certain networks typically require an external network interface adapter attached to a particular general-purpose data port (e.g., a USB port on the computer system (3700)) or peripheral bus (3749); other networks are typically integrated into the core of the computer system (3700) by attaching to a system bus (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system), as described below. Using any of these networks, the computer system (3700) can communicate with other entities. Such communication can be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to a specific CANbus device), or two-way with other computer systems using, for example, local or wide-area digital networks. Specific protocols and protocol stacks can be used with each of these networks and network interfaces described above.

[0239] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (3740) of the computer system (3700).

[0240] The core (3740) may include specialized programmable processing devices in the form of one or more central processing units (CPUs) (3741), graphics processing units (GPUs) (3742), field programmable gate arrays (FPGAs) (3743), task-specific hardware accelerators (3744), graphics adapters (3750), etc. These devices may be connected via a system bus (3748), along with read-only memory (ROM) (3745), random access memory (3746), and internal mass storage (3747), such as an internal, non-user-accessible hard drive or SSD. In some computer systems, the system bus (3748) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (3748) or via a peripheral bus (3749). In one example, a display (3710) may be connected to the graphics adapter (3750). Architectures for peripheral buses include PCI, USB, etc.

[0241] The CPU (3741), GPU (3742), FPGA (3743), and accelerator (3744) can combine to execute specific instructions that may constitute the aforementioned computer code. That computer code can be stored in ROM (3745) or RAM (3746). Persistent data can be stored, for example, in internal mass storage (3747), while transient data can also be stored in RAM (3746). Rapid storage and retrieval from any of the memory devices can be enabled using cache memory, which can be closely associated with one or more of the CPU (3741), GPU (3742), mass storage (3747), ROM (3745), RAM (3746), etc.

[0242] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0243] By way of example, and not limitation, a computer system (3700) having an architecture, and specifically a core (3740), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be the user-accessible mass storage introduced above, as well as media associated with specific storage of the core (3740) that is non-transitory in nature, such as the core's internal mass storage (3747) or ROM (3745). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (3740). The computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause the core (3740), and specifically the processor (including a CPU, GPU, FPGA, etc.) therein, to perform particular processes or particular portions of particular processes described herein, including defining data structures stored in RAM (3746) and modifying such data structures according to software-defined processes. Additionally, or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (3744)) that can operate in place of or together with software to perform particular processes or particular portions of particular processes described herein. Where appropriate, references to software can encompass logic, and vice versa. Where appropriate, references to computer-readable media can encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software. Appendix A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding MPM: Most Probable Mode WAIP: Wide-angle Intra Prediction SEI: Supplemental Extended Information VUI: Video Usability Information GOP: Group of Pictures TU: Conversion unit PU: Prediction Unit CTU: Coding Tree Unit CTB: coding tree block PB: Predicted Block HRD: Hypothetical Reference Decoder SDR: Standard Dynamic Range SNR: Signal to Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Area SSD: Solid State Drive IC: Integrated Circuit CU: Coding Unit PDPC: Position-dependent predictive coupling ISP: Intra-subpartition SPS: Sequence parameter settings

[0244] While this disclosure describes several exemplary embodiments, there are alterations, substitutions, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0245] 101 Samples 102 Arrow 103 Arrow 104 Square Blocks 180 Schematic 201 Current Block 300 Communication Systems 310 Terminal Devices 320 terminal devices 330 Terminal Devices 340 Terminal Devices 350 Communication Network 400 Communication Systems 401 Video Source 402 Video Picture Stream 403 Video Encoder 404 Encoded Video Data, Encoded Video Bitstream 405 Streaming Server 406 Client Subsystem 407 Encoded Video Data, Input Copy 408 Client Subsystem 409 Encoded Video Data, Copy 410 Video Decoder 411 Video Picture Output Stream 412 Display 413 Capture Subsystem 420 Electronic Devices 430 Electronic Devices 501 Channel 510 Video Decoder 512 rendering device 515 buffer memory 520 Parser 521 Symbol 530 Electronic Devices 531 Receiver 551 Scaler / Descaler Unit 552 Intra-picture prediction unit 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter Unit 557 Reference Picture Memory 558 Current Picture Buffer 601 Video Sources 603 Video Encoder, Video Coder 620 Electronic Devices 630 Source Coder 632 Coding Engine 633 Local Video Decoder 634 Reference Picture Cache, Reference Picture Memory 635 Predictor 640 Transmitter 643 coded video sequence 645 Entropy Coder 650 Controller 660 Communication Channels 703 Video Encoder 721 General-purpose controller 722 Intra Encoder 723 Residual Calculator 724 Residual Encoder 725 Entropy Encoder 726 Switch 728 Residual Decoder 730 Interencoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Recovery Module 880 Interdecoder 910 Diamond Shape Filter 911 Diamond Shape Filter 920~932 elements 940~964 elements 1110 4x4 Block 1111 4x4 blocks 1120 Horizontal CTU Boundary 1121 CTU boundary 1130 Virtual Boundary 1131 Virtual Boundary 1210 Virtual Boundary 1220 Virtual Boundary 1230 Virtual Boundary 1240 Virtual Boundary 1250 Virtual Boundary 1260 Virtual Boundary 1300 Pictures 1400 Quadtree Partitioning Pattern 1510 Sample Adaptive Offset (SAO) Filter 1512 SAO filter 1514 SAO filter 1516 ALF Luminance Filter 1518 ALF Chroma Filter 1521 CC-ALF 1522 Adder 1531 CC-ALF 1532 Adder 1541 SAO filtered luminance component 1542 Second intermediate component 1543 Fourth intermediate component 1552 First intermediate component 1553 Third intermediate component 1561 Filtered Luminance CB 1562 filtered first chroma component 1563 filtered second chroma component 1600 filters 1610 filter coefficients 1620 Diamond shape 1801 luminance samples 1801(1) top left sample, luminance sample 1801(2) Upper right sample, luminance sample 1801(3) Lower left sample, luminance sample 1801(4) bottom right sample, luminance sample 1803 saturation samples 1803(1) Saturation Samples 1804 saturation samples 1805 saturation samples 1806 saturation samples 1807 Saturation Samples 1808 saturation samples 1910 Block 1920 direction 1923 direction 1930 Fully directional block 1933 Fully directional block 2100 SAO type table 2210 First Pattern 2220 Second Pattern 2230 Third Pattern 2240 Fourth Pattern 2300 Table for pixel classification rules 2400 Syntax Example 2500 filter support area 2600 filter support area 2700 table 2900 Loop Filter Chain 2901 Input Node 2902 intermediate nodes 2903 intermediate nodes 2909 Output Node 3000A Loop Filter Chain 3000B Loop Filter Chain 3000C Loop Filter Chain 3000D Loop Filter Chain 3011A First Node 3012A Second Node 3013A Third Node 3011B First Node 3012B Second Node 3013B Third Node 3011C First Node 3012C Second Node 3013C Third Node 3011D First Node 3012D Second Node 3013D Third Node 3100A Loop Filter Chain 3100B Loop Filter Chain 3111A First Node 3112A Second Node 3113A Third Node 3111B First Node 3112B Second Node 3113B Third Node 3200A Loop Filter Chain 3200B Loop Filter Chain 3200C Loop Filter Chain 3211A First Node 3212A Second Node 3213A Third Node 3211B First Node 3212B Second Node 3213B Third Node 3211C First Node 3212C Second Node 3213C Third Node 3300 Loop Filter Chain 3311 First Node 3312 Second Node 3313 Third Node 3400 Loop Filter Chain 3500 Loop Filter Chain 3600 processes 3700 Computer Systems 3701 Keyboard 3702 Mouse 3703 Trackpad 3705 Joystick 3706 Microphone 3707 Scanner 3708 Camera 3709 Speaker 3710 Touchscreen 3720 CD / DVD ROM / RW 3721 CD / DVD or similar media 3722 thumb drive 3723 Removable Hard Drive or Solid State Drive 3740 cores 3741 Central Processing Unit (CPU) 3742 Graphics Processing Unit (GPU) 3743 Field Programmable Gate Area (FPGA) 3744 Hardware Accelerator 3745 Read-Only Memory (ROM) 3746 Random Access Memory 3747 Internal Mass Storage 3748 System Bus 3749 Peripheral bus 3750 graphics adapter 3754 Interface 3755 Communication Networks

Claims

1. 1. A method for filtering in video coding, comprising: determining, by a processor, a first offset value by applying a component-to-component sample offset (CCSO) filter to a first reconstructed luma sample at a first node along a loop filter chain including a plurality of video filters, wherein applying the CCSO filter uses a lookup table to determine the first offset value corresponding to a combination of quantized differences between a current chroma sample and each of the first reconstructed luma samples; adding, by the processor, the first offset value to an intermediate reconstructed sample at a second node along the loop filter chain to generate a second reconstructed sample, the second node being a node downstream of a Constrained Directivity Enhancement Filter (CDEF) in the loop filter chain; Including, the CCSO and CDEF are connected in either series or parallel in the loop filter chain; When the CCSO and the CDEF are connected in series, the first node is a downstream node of the CDEF; When the CCSO and the CDEF are connected in parallel, the first node is a downstream node of the CDEF.

2. The method of claim 1 , wherein the intermediate reconstructed sample and the first reconstructed luma sample are samples of different color components.

3. The method of claim 1 , wherein the first node is located before a deblocking filter in the loop filter chain.

4. The method of claim 1, wherein when the CCSO and the CDEF are connected in series, the first node and the second node are the same node.

5. The first reconstructed luminance sample is Processing modules before the deblocking filter, the deblocking filter, or Loop Recovery Filter The method of claim 1, wherein the signal is generated by at least one of:

6. The method of claim 1, wherein when the CCSO and the CDEF are connected in parallel, the first node and the second node are different nodes.

7. The first reconstructed luma samples are generated by a processing module prior to a deblocking filter, and the intermediate reconstructed samples are The method of claim 1 , wherein the signal is generated by a loop reconstruction filter.

8. The method of claim 1 , wherein the first reconstructed luma sample is generated by a deblocking filter and the intermediate reconstructed sample is generated by a loop reconstruction filter.

9. The method of claim 1 , wherein the first reconstructed luma sample is generated by the CDEF and the intermediate reconstructed sample is generated by a loop reconstruction filter.

10. a processor; a memory coupled to the processor, The memory stores instructions configured to, when executed by the processor, cause the method of any one of claims 1 to 9 to be performed. Device.

11. A program for carrying out the method according to any one of claims 1 to 9.

12. 10. A non-transitory computer-readable storage medium storing instructions configured to cause the method of any one of claims 1 to 9 to be performed.

Images