Methods, apparatus, and programs for boundary processing in video coding.

Abstract

Aspects of this disclosure provide methods and apparatus for filtering in video encoding / decoding. [Solution] In some examples, the device for video coding includes a processing circuit. The processing circuit buffers a first boundary pixel value of a first reconstructed sample at a first node along the loop filter chain. The first node relates to a nonlinear mapping-based filter applied in the loop filter chain before the loop restoration filter. The first boundary pixel value is the value of a pixel at the frame boundary. Based on the buffered first boundary pixel value, the processing circuit applies the loop restoration filter to the reconstructed sample to be filtered.

Description

[Technical Field]

[0001] [References] This application claims priority to U.S. Patent Application No. 17 / 448,469, “Method and Apparatus for Boundary Handling in Video Coding,” filed on 22 September 2021, which claims priority to U.S. Provisional Application No. 63 / 187,213, “On Loop Restoration Boundary Handling,” filed on 11 May 2021. The entire disclosure of the prior applications is incorporated herein by reference.

[0002] [Technical field] This disclosure generally describes embodiments related to video coding. [Background technology]

[0003] The background information provided herein is intended to provide an overview of the context of this disclosure. The work of the inventors named in this application, to the extent described in this background section, and aspects of this description that may otherwise not qualify as prior art at the time of filing, are not considered prior art to this disclosure, either expressly or implicitly.

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luminance samples and associated chrominance samples. The series of pictures can have a fixed or variable picture rate (informally also known as the frame rate), for example, a picture rate of 60 pictures per second or 60 Hz. Uncompressed video has specific bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at a frame rate of 60 Hz) requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires a storage space of more than 600 GB.

[0005] One purpose of video encoding and decoding can be the reduction of redundancy in the input video signal by compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, sometimes by more than two orders of magnitude. Both reversible compression and irreversible compression, as well as combinations thereof, can be used. Reversible compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed signal. When using irreversible compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal useful for its intended purpose. In the case of video, irreversible compression is widely used. The amount of allowable distortion depends on the application. For example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect that higher allowable / tolerable distortion can result in a higher compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transformation, quantization, and entropy coding.

[0007] Video codec technology can include techniques known as intra coding. In intra coding, sample values are represented without reference to samples from previously reconstructed reference pictures or other data. In some video codecs, a picture is spatially divided into blocks of samples. If all blocks of samples are encoded in an intra mode, that picture can be an intra picture. Intra pictures and their derivatives such as independent decoder refresh pictures can be used to reset the decoder state and thus can be used as the first picture in an encoded video bitstream and video session or as a still image. The samples of an intra block can be subjected to a transform, and the transform coefficients can be quantized prior to entropy coding. Intra prediction can be a technique that minimizes sample values in a pre - transform region. In some cases, the smaller the DC value and AC coefficients after the transform, the fewer bits are required with a given quantization step size to represent the block after entropy coding.

[0008] Traditional intra coding, such as that known from MPEG - 2 generation coding techniques, does not use intra prediction. However, some newer video compression techniques include techniques that attempt, for example, from surrounding sample data and / or metadata obtained during the encoding / decoding of spatially neighboring and previously decoded in decoding order blocks of data. Such techniques are hereinafter referred to as "intra prediction" techniques. Note that in at least some cases, intra prediction uses only reference data from the currently reconstructed picture and does not use reference data from reference pictures.

[0009] Various forms of intra-prediction are possible. If two or more such techniques are available in a given video coding technique, the techniques used can be coded in intra-prediction mode. In certain cases, a mode may have submodes and / or parameters, which can be coded individually or included in a mode codeword. The choice of codeword for a given mode / submode / parameter combination can affect the coding efficiency gain through intra-prediction, and similarly, the entropy coding technique used to convert the codeword to a bitstream can also have an effect.

[0010] A mode of intra-prediction was introduced in H.264, refined in H.265, and further refined in newer coding techniques such as Joint Exploration Models (JEM), Versatile Video Coding (VVC), and Benchmark Sets (BMS). A predictor block can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of neighboring samples are copied to the predictor block according to a certain direction. The reference to the direction used can be encoded in the bitstream or may be predicted itself.

[0011] Referring to Figure 1A, in the lower right, a subset of nine predictor directions known from the 33 possible predictor directions of H.265 (corresponding to 33 of the 35 intra-modes, or angular modes) is depicted. The point where the arrows converge (101) represents the predicted sample. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from the upper right sample(s) at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from the lower left sample(s) of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] Continuing to refer to Figure 1A, a 4x4 sample square block (104) is depicted in the upper left (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block is 4x4 sample size, S44 is in the lower right. Furthermore, a reference sample is shown that follows a similar numbering scheme. The reference sample is labeled with R and its Y position (e.g., row index) and X position (column index) relative to block (104). In both H.264 and H.265, the predicted sample is a neighborhood of the block being reconstructed, and therefore there is no need to use negative values.

[0013] Intra-picture prediction can function by copying a reference sample value from a neighboring sample that is assigned by the prediction direction, which is signaled. For example, suppose the encoded video bitstream includes signaling for this block indicating the prediction direction, which is aligned with arrow (102). That is, the sample is predicted from the upper right prediction sample(s) at an angle of 45 degrees from the horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the direction is not divisible by 45 degrees, the values ​​of multiple reference samples can be combined, for example, by interpolation, to calculate the reference sample.

[0015] With the advancement of video coding technology, the number of possible directions has increased. H.264 (2003) could represent nine different directions. This increased to 33 in H.265 (2013), and as of the time of this disclosure, JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding are used to represent those more likely directions with a small number of bits, while accepting a penalty for less likely directions. Furthermore, the direction itself can sometimes be predicted from the neighboring directions used in neighboring, already decoded blocks.

[0016] Figure 1B shows a schematic diagram (180) illustrating 65 intra-prediction directions by JEM to show the number of prediction directions increasing over time.

[0017] The mapping of intra-predicted direction bits in an encoded video bitstream representing direction can vary depending on the video encoding technique, ranging from a simple direct mapping of the predicted direction to an intra-predicted mode, to complex adaptive schemes involving codewords, most probable modes, and similar techniques. However, in any case, there may be certain directions in video content that are statistically less likely than others. Since the goal of video compression is to reduce redundancy, in a well-functioning video encoding technique, these less likely directions are represented by more bits than the more likely directions.

[0018] Motion compensation may be an irreversible compression technique and may be used for predicting a newly reconstructed picture or part thereof after blocks of sample data from a previously reconstructed picture or part thereof (reference picture) have been spatially shifted in the direction indicated by a motion vector (MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two or three dimensions, X and Y, where the third dimension is an indication of the reference picture used (which may indirectly be the time dimension).

[0019] In some video compression techniques, a motion vector (MV) applicable to a given region of sample data can be predicted from other MVs, for example, from MVs related to another region of sample data that is spatially adjacent to the region being reconstructed and precedes that MV in the decoding order. This significantly reduces the amount of data required to encode the MV, thereby eliminating redundancy and increasing compression. MV prediction works because, for example, when encoding an input video signal derived from a camera (known as natural video), there is statistical certainty that regions larger than the region to which a single MV is applicable move in a similar direction, and therefore, in certain cases, can be predicted using similar motion vectors derived from neighboring MVs. As a result, the MV found for a given region will be similar to or identical to the MV predicted from the surrounding MVs, and it can be represented with fewer bits than what would be used if the MV were encoded directly after entropy coding. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself may be irreversible, for example, due to rounding errors when calculating the predictor from some surrounding MVs.

[0020] H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016) describes various MV prediction mechanisms. Of the many MV prediction mechanisms provided by H.265, this specification will describe a technique hereafter referred to as "spatial merge".

[0021] Referring to Figure 2, the current block (201) contains samples found by the encoder during the motion search process, which are predictable from the previous block of the same size but spatially shifted. Instead of directly encoding its MV, the MV can be derived from metadata associated with one or more reference pictures, for example, from the most recent reference picture (in decoding order), using the MV associated with one of the five surrounding samples labeled A0, A1, and B0, B1, B2 (202-206, respectively). In H.265, the MV prediction can use predictors from the same reference pictures used by the neighboring blocks. [Overview of the Initiative]

[0022] Aspects of this disclosure provide methods and apparatus for filtering in video coding / decoding. In some examples, the apparatus for video coding includes a processing circuit. The processing circuit buffers a first boundary pixel value of a first reconstructed sample at a first node along a loop filter chain. The first node relates to a nonlinear mapping-based filter applied in the loop filter chain before the loop-reconstruction filter. In one example, the first boundary pixel value is the value of a pixel at the frame boundary. Based on the buffered first boundary pixel value, the processing circuit applies a loop-reconstruction filter to the reconstructed sample to be filtered.

[0023] In some examples, the nonlinear mapping-based filter is a cross-component sample offset (CCSO) filter. In other examples, the nonlinear mapping-based filter is a local sample offset (LSO) filter.

[0024] In some examples, the first reconstructed sample at the first node is the input to a nonlinear mapping-based filter. In some examples, the processing circuit buffers the second boundary pixel value of the second reconstructed sample at the second node along the loop filter chain. The second reconstructed sample at the second node is generated after the application of the sample offset generated by the nonlinear mapping-based filter. The second boundary pixel value is the pixel value at the frame boundary. The processing circuit can then apply the loop reconstruction filter to the reconstructed sample to be filtered, based on the buffered first boundary pixel value and the buffered second boundary pixel value.

[0025] In one example, the processing circuit combines the sample offset generated by a nonlinear mapping-based filter with the output of a constraint directional enhancement filter to generate a second reconstructed sample. In another example, the processing circuit combines the sample offset generated by a nonlinear mapping-based filter with the first reconstructed sample to generate an intermediate reconstructed sample, and then applies a constraint directional enhancement filter to the intermediate reconstructed sample to generate a second reconstructed sample.

[0026] In some examples, the processing circuit buffers the second boundary pixel value of the second reconstructed sample at a second node along the loop filter, and combines the second reconstructed sample at the second node with the sample offset generated by the nonlinear mapping-based filter to generate the reconstructed sample to be filtered. The second boundary pixel value is the pixel value at the frame boundary. The processing circuit then applies the loop reconstruction filter to the reconstructed sample to be filtered based on the buffered first boundary pixel value and the buffered second boundary pixel value.

[0027] In some examples, the processing circuit buffers a second boundary pixel value from a second reconstructed sample generated by a deblocking filter, and applies a constraint direction reinforcement filter to the second reconstructed sample to generate an intermediate reconstructed sample. The second boundary pixel value is the pixel value at the frame boundary. The processing circuit combines the intermediate reconstructed sample with a sample offset generated by a nonlinear mapping-based filter to generate a first reconstructed sample. A loop reconstruction filter can then be applied based on the buffered first boundary pixel value and the buffered second boundary pixel value.

[0028] In some examples, the processing circuit clips the reconstructed samples to be filtered before applying the loop reconstruction filter. In some examples, the processing circuit clips the intermediate reconstructed samples before combining them with the sample offset generated by the nonlinear mapping-based filter.

[0029] Aspects of this disclosure also provide non-temporary computer-readable media that, when executed by a computer, store instructions causing the computer to perform one of the methods for video encoding / decoding. [Brief explanation of the drawing]

[0030] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the following detailed description and accompanying drawings. [Figure 1A] This is a schematic diagram of an exemplary subset of intra-predictive modes. [Figure 1B] This is an illustrative diagram of an example of an intra-prediction direction. [Figure 2] This is a schematic diagram of the current block and its surrounding spatial merge candidates in one example. [Figure 3] This is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Figure 4] This is a schematic diagram of a simplified block diagram of a communication system (400) according to one embodiment. [Figure 5] This is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 6] This is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 7] A block diagram of an encoder according to another embodiment is shown. [Figure 8] A block diagram of a decoder according to another embodiment is shown. [Figure 9] An example of a filter shape according to an embodiment of this disclosure is shown. [Figure 10A] Examples of subsampled locations used to calculate the gradient according to embodiments of this disclosure are shown. [Figure 10B] Examples of subsampled locations used to calculate the gradient according to embodiments of this disclosure are shown. [Figure 10C] Examples of subsampled locations used to calculate the gradient according to embodiments of this disclosure are shown. [Figure 10D] Examples of subsampled locations used to calculate the gradient according to embodiments of this disclosure are shown. [Figure 11A] An example of a virtual boundary filtering process according to an embodiment of this disclosure is shown. [Figure 11B]An example of a virtual boundary filtering process according to an embodiment of this disclosure is shown. [Figure 12A] An example of symmetrical padding behavior at a virtual boundary according to embodiments of this disclosure is shown. [Figure 12B] An example of symmetrical padding behavior at a virtual boundary according to embodiments of this disclosure is shown. [Figure 12C] An example of symmetrical padding behavior at a virtual boundary according to embodiments of this disclosure is shown. [Figure 12D] An example of symmetrical padding behavior at a virtual boundary according to embodiments of this disclosure is shown. [Figure 12E] An example of symmetrical padding behavior at a virtual boundary according to embodiments of this disclosure is shown. [Figure 12F] An example of symmetrical padding behavior at a virtual boundary according to embodiments of this disclosure is shown. [Figure 13] Examples of picture segmentation according to some embodiments of this disclosure are shown. [Figure 14] This shows quadtree partitioning patterns for pictures in several examples. [Figure 15] This disclosure shows a cross-component filter according to one embodiment. [Figure 16] An example of a filter shape according to one embodiment of this disclosure is shown. [Figure 17] Examples of syntax for cross-component filters according to some embodiments of this disclosure are shown. [Figure 18A] The following shows an exemplary position of the chromatic sample relative to the luma sample according to embodiments of the present disclosure. [Figure 18B] The following shows an exemplary position of the chromatic sample relative to the luma sample according to embodiments of the present disclosure. [Figure 19] An example of direction finding according to one embodiment of this disclosure is shown. [Figure 20] Examples illustrating subspace projections in several cases are shown. [Figure 21] A table of multiple sample adaptation offset (SAO) types according to one embodiment of this disclosure is shown. [Figure 22] Examples of patterns for pixel classification at edge offsets in several cases are shown. [Figure 23] A table is shown for pixel classification rules for edge offsets in several examples. [Figure 24] Here are some examples of syntax that can be transmitted as signals. [Figure 25] Examples of filter support regions in some embodiments of this disclosure are shown. [Figure 26] Examples of other filter support regions in some embodiments of this disclosure are shown below. [Figure 27A] A portion of the table showing 81 combinations according to one embodiment of the present disclosure is shown. [Figure 27B] A portion of the table showing 81 combinations according to one embodiment of the present disclosure is shown. [Figure 27C] A portion of the table showing 81 combinations according to one embodiment of the present disclosure is shown. [Figure 28] This example shows a configuration of seven filter shapes for three filter taps. [Figure 29] Block diagrams of loop filter chains in several examples are shown. [Figure 30] An example of a loop filter chain including a nonlinear mapping-based filter is shown. [Figure 31] Here is another example of a loop filter chain that includes a nonlinear mapping-based filter. [Figure 32] Here is another example of a loop filter chain that includes a nonlinear mapping-based filter. [Figure 33] Here is another example of a loop filter chain that includes a nonlinear mapping-based filter. [Figure 34] Here is another example of a loop filter chain that includes a nonlinear mapping-based filter. [Figure 35] Here is another example of a loop filter chain that includes a nonlinear mapping-based filter. [Figure 36]A flowchart outlining the process according to one embodiment of this disclosure is shown. [Figure 37] This is a schematic diagram of a computer system according to one embodiment. [Modes for carrying out the invention]

[0031] Figure 3 shows a simplified block diagram of a communication system (300) according to one embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, over a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected over the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) perform one-way transmission of data. For example, terminal device (310) may encode video data (e.g., a stream of video pictures captured by terminal device (310)) for transmission to the other terminal device (320) over the network (350). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. Terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to restore the video pictures, and display the video pictures according to the restored video data. One-way data transmission can be common in media service applications and the like.

[0032] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, for example, during a video conference. For bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., a stream of video pictures captured by the terminal device) for transmission over the network (350) to the other terminal device of terminal devices (330) and (340). Each of the terminal devices (330) and (340) may receive encoded video data transmitted by the other terminal device of terminal devices (330) and (340), decode the encoded video data to restore the video pictures, and display the video pictures on an accessible display device according to the restored video data.

[0033] In the example in Figure 3, terminal devices (310), (320), (330), and (340) may be represented as a server, a personal computer, and a smartphone, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure find applications in laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (350) represents any number of networks that transmit encoded video data between terminal devices (310), (320), (330), and (340), including, for example, wired and / or wireless communication networks. Communication network (350) may exchange data over circuit-switched and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of network (350) may not be important to the operation of this disclosure unless described below.

[0034] Figure 4 shows the arrangement of a video encoder and video decoder in a streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications, such as video conferencing, digital TV, and storage of compressed video on digital media including CDs, DVDs, and memory sticks.

[0035] The streaming system may include a video source (401), such as a digital camera, and may also include a capture subsystem (413) that generates, for example, a stream of uncompressed video pictures (402). In one example, the stream of video pictures (402) includes samples captured by the digital camera. The stream of video pictures (402), which is drawn as a thick line to emphasize its high data volume compared to encoded video data (404) (or encoded video bitstream), may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or realize aspects of the disclosed subject, as will be described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), which is drawn as a thin line to emphasize its lower data volume compared to the stream of video pictures (402), may be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) in Figure 4, can access a streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). Client subsystem (406) may include a video decoder (410) within, for example, an electronic device (430). The video decoder (410) decodes the input copy (407) of the encoded video data and generates an output stream (411) of a video picture that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T Recommendation H.265.For example, a video coding standard under development is informally known as Multipurpose Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0036] It should be noted that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0037] Figure 5 shows a block diagram of a video decoder (510) according to one embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used in place of the video decoder (310) in the example of Figure 4.

[0038] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510). In the same or different embodiments, one encoded video sequence at a time, and the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device that stores encoded video data. The receiver (531) may receive the encoded video data together with other data, such as encoded audio data and / or auxiliary data streams, which may carry their respective usage entities (not shown). The receiver (531) can isolate the encoded video sequences from other data. As a measure against network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter "Parser"). In certain applications, the buffer memory (515) is part of the video decoder (510). In other applications, it may be outside the video decoder (510) (not shown). In other applications, for example, to counter network jitter, there may be a buffer memory (not shown) outside the video decoder (510), and further, for example, to handle playback timing, there may be another buffer memory (515) inside the video decoder (510). If the receiver (531) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isochronous network, the buffer memory (515) may not be necessary or may be small. For use in best-effort packet networks such as the Internet, the buffer memory (515) may be required, may be relatively large, may be advantageously adaptive in size, and may be implemented at least partially outside the video decoder (510) in the operating system or similar element (not shown).

[0039] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the encoded video sequence. These categories of symbols may include information used to manage the operation of the video decoder (510) and potentially information for controlling a rendering device, such as a render device (512) (e.g., a display screen). The rendering device may be coupled to an electronic device (530), rather than being an integral part of the electronic device (530), as shown in Figure 5. Control information for the rendering device(s) may be in the form of Supplementary Enhancement Information (SEI messages) or Video Usability Information (VUI) parameter set fragments (not shown). The parser (520) can parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may follow video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, context-sensitive or non-context-sensitive arithmetic coding, etc. The parser (520) can extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to the group. Subgroups can include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The parser (520) can also extract information such as transform coefficients, quantizer parameter values, and motion vectors from the encoded video sequence.

[0040] The parser (520) can perform an entropy decoding / parse operation on the video sequence received from the buffer memory (515), thereby generating a symbol (521).

[0041] The reconstruction of symbol (521) can involve multiple different units, depending on the type of encoded video picture or its parts (e.g., inter and intra picture, inter and intra block) and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the encoded video sequence by parser (520). The flow of such subgroup control information between parser (520) and the multiple units described below is not depicted for clarity.

[0042] In addition to the functional blocks already described, the video decoder (510) can be conceptually divided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these units can interact closely with each other and be at least partially integrated. However, for the purpose of describing the subject matter being disclosed, the conceptual subdivision into functional units described below is appropriate.

[0043] The first unit is the scaler / inverse unit (551). The scaler / inverse unit (551) receives quantized transformation coefficients and control information from the parser (520) as a symbol (singular or plural) (521). The control information includes which transformation to use, block size, quantization coefficients, quantization scaling matrix, etc. The scaler / inverse unit (551) can output a block containing sample values ​​that can be input to the tallyer (555).

[0044] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-encoded blocks, i.e., blocks that do not use prediction information from previously reconstructed pictures but can use prediction information from previously reconstructed portions of the current picture. Such prediction information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates blocks of the same size and shape as the block being reconstructed, using surrounding already reconstructed information taken from the current picture buffer (558). The current picture buffer (558) buffers, for example, partially reconstructed current pictures and / or fully reconstructed current pictures. The aggregater (555) may, in some cases, add the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) for each sample.

[0045] In other cases, the output samples of the scaler / inverse unit (551) may relate to intercoded and potentially motion-compensated blocks. In such cases, the motion-compensated prediction unit (553) may access the reference picture memory (557) to retrieve samples to be used for prediction. After motion-compensating the retrieved samples according to symbols (521) relating to the blocks, these samples can be added by the aggregater (555) to the output of the scaler / inverse unit (in this case, called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion-compensated unit (553) retrieves prediction samples can be controlled by motion vectors available to the motion-compensated unit (553) in the form of symbols (521). These symbols may have, for example, X, Y, and reference picture components. Motion compensation may include interpolation of sample values ​​retrieved from the reference picture memory (557) when accurate motion vectors less than or equal to a sample are used, motion vector prediction mechanisms, etc.

[0046] The output samples from the tallyer (555) can be subjected to various loop filtering techniques within the loop filter unit (556). The video compression technique may include in-loop filtering techniques. The in-loop filtering technique is controlled by parameters contained in the encoded video sequence (also called the encoded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but can also respond to metadata obtained during decoding of earlier parts (in decoding order) of the encoded picture or encoded video sequence, as well as to previously reconstructed and loop-filtered sample values.

[0047] The output of the loop filter unit (556) can be a sample stream, which can be output to the render device (512) and can also be stored in the reference picture memory (557) for use in future interpicture prediction.

[0048] Once an encoded picture is fully reconstructed, it can be used as a reference picture for future predictions. For example, once an encoded picture corresponding to the current picture is fully reconstructed and that encoded picture is identified as a reference picture (e.g., by a parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a fresh current picture buffer can be reallocated before the reconstruction of subsequent encoded pictures begins.

[0049] The video decoder (510) can perform decoding operations according to a specified video compression technique in a standard such as ITU-T Recommendation H.265. The encoded video sequence can conform to the syntax defined by the video compression technique or standard used, in the sense that the encoded video sequence conforms to the syntax and profile documented in the video compression technique or standard. Specifically, a profile can select certain tools from all the tools available in the video compression technique or standard, as tools that are only available for use under that profile. Conformance may also require that the complexity of the encoded video sequence be within the range defined by the level of the video compression technique or standard. In some cases, the level constrains the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further constrained through the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management, which are signaled in the encoded video sequence.

[0050] In one embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s) or more. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, a temporal, spatial, or signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, forward error correction codes, etc.

[0051] Figure 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of Figure 4.

[0052] The video encoder (603) can receive video samples from a video source (601) (which is not part of the electronic device (620) in the example in Figure 6) that can capture video images to be encoded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0053] The video source (601) can provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream, which can have any preferred bit depth (e.g., 8 bits, 10 bits, 12 bits, ...), any color space (e.g., BT.601 YCrCB, RGB, ...), and any preferred sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) may be a storage device storing pre-prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that give motion when viewed in sequence. The picture itself may be organized as a spatial array of pixels, and each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description will focus on samples.

[0054] According to one embodiment, the video encoder (603) can encode and compress the pictures of a source video sequence in real time or under any other temporal constraints required by the application to obtain an encoded video sequence (643). Enforcing an appropriate encoding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units, such as those described below. Such couplings are not depicted for clarity. Parameters set by the controller (650) may include parameters related to rate control (picture skip, quantizer, lambda value of rate-distortion optimization technique, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other preferred functions relating to the video encoder (603) optimized for certain system designs.

[0055] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. In a drastically simplified explanation, in one example, the encoding loop may include a source encoder (630) (for example, responsible for generating symbols such as a symbol stream based on the input picture to be encoded and one or more reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in a similar manner to that a (remote) decoder would also generate (in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since decoding of the symbol stream yields bit-accurate results regardless of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the encoder's prediction unit "sees" the exact same sample values ​​as the reference picture samples that the decoder "sees" when using the predictions during decoding. This fundamental principle of reference picture synchronization (and the resulting drift, for example, when synchronization cannot be maintained due to channel errors) is also used in several related techniques.

[0056] The operation of the "local" decoder (633) may be the same as that of the "remote" decoder, such as the video decoder (410), which has already been described in detail above in relation to Figure 5. However, referring briefly to Figure 5, since symbols are available and the encoding / decoding of symbols to an encoded video sequence by the entropy encoder (645) and parser (420) may be reversible, the entropy decoding section of the video decoder (410), which includes the buffer memory (415) and parser (420), does not need to be fully implemented in the local decoder (633).

[0057] An observation that can be made at this point is that any decoder technique present within the decoder, except for parse / entropy decoding, must exist in substantially the same functional form within the corresponding encoder. Therefore, the subject matter disclosed focuses on decoder operation. The description of encoder techniques can be omitted, as it is the inverse of the comprehensively described decoder techniques. More detailed explanations are necessary only in certain areas, which are provided below.

[0058] During operation, in some examples, the source encoder (630) can perform motion-compensated predictive coding, predictively coding the input picture by referencing one or more previously coded pictures from a video sequence designated as “reference pictures”. In this way, the coding engine (632) codes the difference between the pixel blocks of the input picture and the pixel blocks of the reference picture(s) that may be selected as predictive references for the input picture.

[0059] The local video decoder (633) can decode the encoded video data of a picture that may be designated as a reference picture, based on the symbols generated by the source encoder (630). The operation of the encoding engine (632) can, advantageously, be a lossy process. When the encoded video data can be decoded by the video decoder (not shown in Figure 6), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) can replicate the decoding process that the video decoder may perform on the reference picture and have the reconstructed reference picture stored in the reference picture cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture that has common content (unless there are transmission errors) as the reconstructed reference picture that would be obtained by the far-end video decoder.

[0060] The predictor (635) can perform a predictive search on the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) can search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that can function as appropriate predictive references for the new picture. The predictor (635) may operate on a sample block-by-pixel-block basis to find appropriate predictive references. In some cases, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory (634), as determined by the search results obtained by the predictor (635).

[0061] The controller (650) may manage the encoding operation of the source encoder (630), including, for example, setting parameters and subgroup parameters used to encode the video data.

[0062] The outputs of all the above functional units can undergo entropy coding in the entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into coded video sequences by lossless compression of the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0063] The transmitter (640) can buffer the encoded video sequence generated by the entropy encoder (645) and prepare it for transmission over the communication channel (660). The communication channel (660) may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) can merge the encoded video data from the video encoder (630) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0064] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain encoded picture type to each encoded picture. The encoded picture type may affect the encoding technique that can be applied to each picture. For example, a picture may often be assigned as one of the following picture types:

[0065] An intra-picture (I-picture) can be encoded and decoded without using other pictures in the sequence as a source for prediction. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art will recognize these variations of I-pictures, as well as their respective uses and characteristics.

[0066] A prediction picture (P-picture) may be encoded and decoded using intra-prediction or inter-prediction, which uses up to one motion vector and reference index to predict the sample values ​​of each block.

[0067] A bidirectional predictive picture (B-picture) may be encoded and decoded using intra-prediction or inter-prediction, employing up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-predictive picture may use three or more reference pictures and associated metadata for the reconstruction of a single block.

[0068] A source picture is typically divided spatially into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block can be encoded. Blocks can be predictively encoded by referencing other (already encoded) blocks, as determined by the encoding assignment applied to each picture in the block. For example, a block of picture I may be non-predictively encoded, or it may be predictively encoded by referencing already encoded blocks of the same picture (spatial prediction or intra-prediction). A pixel block of picture P may be predictively encoded via spatial prediction or temporal prediction by referencing one previously encoded reference picture. A block of picture B may be predictively encoded via spatial prediction or temporal prediction by referencing one or two previously encoded reference pictures.

[0069] The video encoder (603) can perform encoding operations in accordance with a specified video encoding technique or standard, such as ITU-T Recommendation H.265. In this operation, the video encoder (603) can perform various compression operations, including predictive encoding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0070] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and the like.

[0071] A video may be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes (temporal or other) correlations between pictures. In one example, a specific picture to be encoded / decoded, called the current picture, is divided into blocks. If a block in the current picture is analogous to a reference block in a previously encoded and still-buffered reference picture in the video, that block in the current picture can be encoded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension to identify the reference picture if multiple reference pictures are used.

[0072] In some embodiments, a dual prediction technique can be used in interpicture prediction. According to the dual prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture, both of which precede the current picture in the video in decoding order (however, in display order, they may be past and future, respectively). A block in the current picture can be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. A block can be predicted by a combination of the first and second reference blocks.

[0073] Furthermore, merge mode techniques can be used in interpicture prediction to improve coding efficiency.

[0074] According to some embodiments of this disclosure, predictions such as interpicture prediction and intrapicture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and those CTUs in a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU contains three coding tree blocks (CTBs), which are one lumen CTB and two chroma CTBs. Each CTU can be recursively quadtree-partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, or four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In one example, each CU is parsed to determine a prediction type for that CU, such as an inter-prediction type or an intra-prediction type. A CU is divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Generally, each PU contains a Luma prediction block (PB) and two Chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. Using a Luma prediction block as an example of a prediction block, the prediction block contains a matrix of values ​​(e.g., Luma values) for pixels, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0075] Figure 7 shows a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values ​​in the current video picture within a sequence of video pictures, and to encode the processing block into an encoded picture which is part of an encoded video sequence. In one example, the video encoder (703) is used instead of the video encoder (403) in the example of Figure 4.

[0076] In the HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block with 8x8 samples. The video encoder (703) determines, for example using rate-distortion optimization, which mode—intra-mode, inter-mode, or bi-prediction mode—best encodes the processing block. If the processing block is encoded in intra-mode, the video encoder (703) may use the intra-prediction technique to encode the processing block into an encoded picture. If the processing block is encoded in inter-mode or bi-prediction mode, the video encoder (703) may use the inter-prediction technique or the bi-prediction technique, respectively, to encode the processing block into an encoded picture. In some video encoding techniques, merge mode may be an inter-picture prediction submode in which the motion vector is derived from one or more motion vector predictors, but without the benefit of encoded motion vector components outside of those predictors. In some other video encoding techniques, there may be motion vector components applicable to the target block. In one example, the video encoder (703) includes other components such as a mode determination module (not shown) for determining the mode of the processing block.

[0077] In the example shown in Figure 7, the video encoder (703) includes an interencoder (730), an intraencoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), all coupled together as shown in Figure 7.

[0078] The interencoder (730) is configured to receive a sample of the current block (e.g., a processing block), compare the block with one or more reference blocks in the reference picture (e.g., blocks in the previous and subsequent pictures), generate interprediction information (e.g., a description of redundant information by the intercoding technique, motion vectors, merge mode information), and, based on the interprediction information, compute an interprediction result (e.g., a predicted block) using any preferred technique. In some examples, the reference picture is a decoded reference picture, decoded based on encoded video information.

[0079] The intra encoder (722) is configured to receive a sample of the current block (e.g., a processing block), and optionally compare the block to a block already encoded in the same picture to generate quantized coefficients after transformation, and optionally also generate intra prediction information (e.g., intra prediction direction information using one or more intra coding techniques). In one example, the intra encoder (722) also calculates an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same picture.

[0080] The general controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines the mode of a block and provides control signals to the switch (726) based on that mode. For example, if the mode is intra-mode, the general controller (721) controls the switch (726) to select the intra-mode result for use by the residual calculator (723), select the intra-prediction information, and control the entropy encoder (725) to include the intra-prediction information in the bitstream. If the mode is inter-mode, the general controller (721) controls the switch (726) to select the inter-prediction result for use by the residual calculator (723), select the inter-prediction information, and control the entropy encoder (725) to include the inter-prediction information in the bitstream.

[0081] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra-encoder (722) or inter-encoder (730). The residual encoder (724) is configured to encode the residual data based on the residual data to generate conversion coefficients. In one example, the residual encoder (724) is configured to convert the residual data from the spatial domain to the frequency domain to generate conversion coefficients. The conversion coefficients are then subjected to a quantization process to obtain quantized conversion coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform to generate decoded residual data. The decoded residual data can be suitably used by the intra-encoder (722) and inter-encoder (730). For example, an interencoder (730) can generate a decoded block based on decoded residual data and interprediction information, and an intraencoder (722) can generate a decoded block based on decoded residual data and intraprediction information. The decoded block is suitably processed to generate a decoded picture, which is buffered in a memory circuit (not shown) and can be used as a reference picture in some examples.

[0082] The entropy encoder (725) is configured to format the bitstream to include the encoded blocks. The entropy encoder (725) is configured to include various information according to a preferred standard such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other preferred information in the bitstream. Note that, according to the disclosed subject matter, if the blocks are encoded in either the inter-mode or bi-prediction mode merge submode, residual information is not present.

[0083] Figure 8 shows a diagram of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive an encoded picture, which is part of an encoded video sequence, and to decode the encoded picture to produce a reconstructed picture. In one example, the video decoder (810) is used instead of the video decoder (410) in the example of Figure 4.

[0084] In the example shown in Figure 8, the video decoder (810) includes an entropy decoder (871), an interdecoder (880), a residual decoder (873), a reconfiguration module (874), and an intradecoder (872) coupled together as shown in Figure 8.

[0085] The entropy decoder (871) can be configured to reconstruct from the encoded picture a certain kind of symbol representing the syntax elements that make up the encoded picture. Such symbols may include, for example, the mode in which the block is encoded (e.g., intra-mode, inter-mode, bi-prediction mode, merge sub-mode, or two of the latter in another sub-mode), prediction information (e.g., intra-prediction information or inter-prediction information, etc.) that can identify certain samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), respectively, and residual information in the form of quantized transformation coefficients, etc. In one example, if the prediction mode is inter or bi-prediction mode, inter-prediction information is provided to the inter-decoder (880). If the prediction type is intra-prediction type, intra-prediction information is provided to the intra-decoder (872). The residual information may undergo inverse quantization and be provided to the residual decoder (873).

[0086] The interdecoder (880) is configured to receive interprediction information and generate interprediction results based on said interprediction information.

[0087] The intra decoder (872) is configured to receive intra prediction information and generate prediction results based on said intra prediction information.

[0088] The residual decoder (873) is configured to perform inverse quantization to extract the dequantized conversion coefficients, process the dequantized conversion coefficients, and convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (including quantizer parameters (QP)), which may be provided by the entropy decoder (871) (this is only low-volume control information, so no data path is drawn).

[0089] The reconstruction module (874) is configured to combine the residuals output by the residual decoder (873) and the prediction results (output by the intra or interprediction module as applicable) in the spatial domain to form a reconstructed block, the reconstructed block may be part of a reconstructed picture, and the reconstructed picture may be part of a reconstructed video. Note that other preferred operations, such as deblocking operations, may be performed to improve visual quality.

[0090] The video encoders (403), (603), (703), and video decoders (410), (510), (810) can be implemented using any preferred technique. In one embodiment, the video encoders (403), (603), (703), and video decoders (410), (510), (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), (703), and video decoders (410), (510), (810) can be implemented using one or more processors that execute software instructions.

[0091] Aspects of this disclosure provide filtering techniques for video encoding / decoding.

[0092] To reduce artifacts, an adaptive loop filter (ALF) with block-based filter adaptation can be applied by the encoder / decoder. For the luma component, one of several filters (e.g., 25 filters) can be selected for a 4x4 luma block, for example, based on the direction and activity of the local gradient.

[0093] ALF can have any suitable shape and size. Referring to Figure 9, ALF(910)~(911) have diamond shapes such as a 5x5 diamond shape for ALF(910) and a 7x7 diamond shape for ALF(911). In ALF(910), elements (920)~(932) form a diamond shape that can be used in the filtering process. For elements (920)~(932), seven values ​​(e.g., C0~C6) can be used. In ALF(911), elements (940)~(964) form a diamond shape that can be used in the filtering process. For elements (940)~(964), thirteen values ​​(e.g., C0~C12) can be used.

[0094] Referring to Figure 9, in some examples, two ALFs (910) and (911) with diamond filter shapes are used. The 5x5 diamond-shaped filter (910) can be applied to the chromatic component (chromatic block, chromatic CB, etc.), while the 7x7 diamond-shaped filter (911) can be applied to the luma component (luma block, luma CB, etc.). Other suitable shapes and sizes can be used for the ALF. For example, a 9x9 diamond-shaped filter can be used.

[0095] The filter coefficients at the positions indicated by those values ​​(for example, C0-C6 in (910) or C0-C12 in (920)) may be non-zero. Furthermore, if the ALF includes a clipping function, the clipping values ​​at those positions may also be non-zero.

[0096] For the block classification of the luma component, a 4x4 block (or luma block, luma CB) can be categorized or classified as one of several (e.g., 25) classes. The classification index C is the directional parameter and the quantized value of the activity value A.

number

number

number

number

[0097] To reduce the complexity of the block classification described above, a subsampled 1-D Laplacian calculation can be applied. Figures 10A-10D show the gradients g in the vertical (Figure 10A), horizontal (Figure 10B), and two diagonal directions d1 (Figure 10C) and d2 (Figure 10D). v , g h , g d1 and g d2 Examples of subsampled locations used to calculate each of these are shown. The same subsampled locations can be used to calculate gradients in different directions. In Figure 10A, the label "V" represents the vertical gradient g.v shows the subsampling positions for calculating. In FIG. 10B, the label "H" represents the horizontal gradient g h shows the subsampling positions for calculating. In FIG. 10C, the label "D1" represents the diagonal gradient g of d1 d1 shows the subsampling positions for calculating. In FIG. 10D, the label "D2" represents the diagonal gradient g of d2 d2 shows the subsampling positions for calculating.

[0098] The gradients g in the horizontal and vertical directions v and g h The maximum value g among them h,v max and the minimum value g h,v min can be set as follows.

Equation

Equation

[0099] The activity value A can be calculated as follows:

number

number

[0100] Since block classification is not applied to the chroma components within a picture, a single set of ALF coefficients can be applied to each chroma component.

[0101] Geometric transformations can be applied to the filter coefficients and the corresponding filter clipping values ​​(also called clipping values). Before filtering a block (e.g., a 4x4 Luma block), for example, the gradient values ​​calculated for that block (e.g., g v , g h , g d1 , and / or g d2Depending on the filter coefficients, geometric transformations such as rotation or diagonal and vertical inversions can be applied to the filter coefficients f(k,l) and the corresponding filter clipping values ​​c(k,l). The geometric transformations applied to the filter coefficients f(k,l) and the corresponding filter clipping values ​​c(k,l) may be equivalent to applying geometric transformations to the samples within the region supported by the filter. Geometric transformations can make different blocks to which ALF is applied more similar by aligning their respective orientations.

[0102] The three geometric transformations, including diagonal flip, vertical flip, and rotation, can be performed as shown in equations (9) to (11), respectively.

number

[0103] In some embodiments, ALF filter parameters are transmitted in an Adaptive Parameter Set (APS) for the picture. The APS can transmit one or more sets (up to 25 sets) of lumina filter coefficients and clipping value indices. In one example, one of these sets may include lumina filter coefficients and one or more clipping value indices. One or more sets (up to 8 sets) of chroma filter coefficients and clipping value indices can also be transmitted. To reduce transmission overhead, filter coefficients for different classifications of the lumina component (e.g., with different classification indices) can be merged. In the slice header, the APS index used for the current slice can be transmitted.

[0104] In one embodiment, a clipping value index (also referred to as the clipping index) can be decoded from the APS. The clipping value index can be used, for example, to determine a corresponding clipping value based on the relationship between the clipping value index and the corresponding clipping value. This relationship can be predefined and stored in the decoder. In one example, this relationship is described by tables such as a luma table (e.g., used for luma CBs) for clipping value indices and corresponding clipping values, and a chroma table (e.g., used for chroma CBs) for clipping value indices and corresponding clipping values. The clipping value may depend on a bit depth B, where B can refer to the internal bit depth, the bit depth of the reconstructed sample in the filtered CB, and so on. In some examples, the tables (e.g., luma table, chroma table) are obtained using equation (12).

number

[0105] To specify the set of Luma filters available for the current slice, one or more APS indices (up to seven APS indices) may be signaled in the slice header for the current slice. The filtering process can be controlled at one or more appropriate levels, such as picture level, slice level, CTB level, and / or others. In some embodiments, the filtering process can be further controlled at the CTB level. A flag is signaled indicating whether the ALF is applied to the Luma CTB. The Luma CTB can select a set of filters from a plurality of fixed filter sets (e.g., 16 fixed filter sets) and a set of filters signaled in the APS (also referred to as the signaled filter set). A filter set index is signaled for the Luma CTB to indicate the filter set to be applied (e.g., a set of filters from the plurality of fixed filter sets and the signaled filter set). The plurality of fixed filter sets are predefined and hardcoded in the encoder and decoder and can be called a predefined filter set.

[0106] For chroma components, the APS index can be signaled in the slice header to indicate the chroma filter set used for the current slice. At the CTB level, if the APS has multiple chroma filter sets, the filter set index can be signaled for each chroma CTB.

[0107] The filter coefficients can be quantized to a norm equal to 128. To reduce the complexity of multiplication, bitstream fit can be applied so that the coefficient values ​​at non-center positions are within the range of -27 to 27-1. In one example, the coefficients at center positions are not transmitted in the bitstream and can be considered equal to 128.

[0108] In some embodiments, the syntax and meaning of the clipping index and clipping value are defined as follows: alf_luma_clip_idx[sfIdx][j] can be used to specify the clipping index of the clipping value to use before multiplying by the j-th coefficient of the signal-transmitted Luma filter indicated by sfIdx. Bitstream compatibility requirements may include the requirement that the value of alf_luma_clip_idx[sfIdx][j] is in the range of 0 to 3 for sfIdx=0 to alf_luma_num_filters_signalled_minus1 and j=0 to 11. The luma filter clipping value AlfClipL[adaptation_parameter_set_id] for element AlfClipL[adaptation_parameter_set_id][filtIdx][j] with filtIdx=0~NumAlfFilters-1 and j=0~11 can be derived as specified in Table 2, depending on bitDepth set to equal BitDepthY and clipIdx set to equal alf_luma_clip_idx[alf_luma_coeff_delta_idx][filtIdx][j]. alf_chroma_clip_idx[altIdx][j] can be used to specify the clipping index of the clipping value to use before multiplying the j-th coefficient of the alternative chroma filter by the index altIdx. Bitstream compatibility requirements may include the requirement that the value of alf_chroma_clip_idx[altIdx][j] is in the range of 0 to 3 for altIdx=0 to alf_chroma_num_alt_filters_minus1 and j=0 to 5. The chroma filter clipping value AlfClipC[adaptation_parameter_set_id][altIdx] with elements AlfClipC[adaptation_parameter_set_id][altIdx][j] for altIdx=0~alf_chroma_num_alt_filters_minus1 and j=0~5 can be derived as specified in Table 2, depending on bitDepth set to equal BitDepthC and clipIdx set to equal alf_chroma_clip_idx[altIdx][j].

[0109] In one embodiment, the filtering process can be described as follows: On the decoder side, when ALF is enabled for CTB, the sample R(i,j) in CU (or CB) can be filtered, and as a result, the filtered sample value R'(i,j) is obtained as shown below using equation (13). In one example, each sample in CU is filtered.

number

[0110] In a nonlinear ALF, multiple sets of clipping values ​​can be provided in Table 3. For example, the luma set contains four clipping values ​​{1024, 181, 32, 6}, and the chroma set contains four clipping values ​​{1024, 161, 25, 4}. The four clipping values ​​in the luma set can be selected by dividing the entire range (e.g., 1024) of sample values ​​(encoded in 10 bits) for the luma block into approximately equal logarithmic regions. This range can be between 4 and 1024 for the chroma set. [Table 3]

[0111] The selected clipping values ​​can be encoded using the "alf_data" syntax element as follows. Using an appropriate encoding scheme (e.g., Golomb encoding), the clipping index corresponding to the selected clipping values, as shown in Table 3, can be encoded. The encoding scheme may be the same one used to encode the filter set index.

[0112] In one embodiment, a virtual boundary filtering process can be used to reduce the ALF line buffer requirements. Thus, modified block classification and filtering can be used for samples near the CTU boundary (e.g., horizontal CTU boundary). The virtual boundary (1130) is defined as "N" as shown in Figure 11A, with respect to the horizontal CTU boundary (1120). samples This can be defined by shifting only the samples. Here, N samples can be a positive integer. For example, N samples For the luma component, it is equal to 4, N samples For the chroma component, it is equal to 2.

[0113] Referring to Figure 11A, a modified block classification can be applied to the Luma component. For example, for a 1D Laplacian gradient calculation of a 4x4 block (1110) above a virtual boundary (1130), only samples on the virtual boundary (1130) are used. Similarly, referring to Figure 11B, for a 1D Laplacian gradient calculation of a 4x4 block (1111) below a virtual boundary (1131) shifted from the CTU boundary (1121), only samples below the virtual boundary (1131) are used. By considering the reduction in the number of samples used in the 1D Laplacian gradient calculation, the quantization of the activation value A can be appropriately scaled.

[0114] For filtering, symmetric padding operations at the virtual boundary can be used for both the luma and chroma components. Figures 12A-12F show examples of such modified ALF filtering for the luma component at the virtual boundary. If the sample to be filtered is located below the virtual boundary, neighboring samples located above the virtual boundary can be padded. If the sample to be filtered is located above the virtual boundary, neighboring samples located below the virtual boundary can be padded. Referring to Figure 12A, neighboring sample C0 can be padded by sample C2 located below the virtual boundary (1210). Referring to Figure 12B, neighboring sample C0 can be padded by sample C2 located above the virtual boundary (1220). Referring to Figure 12C, neighboring samples C1-C3 can each be padded by samples C5-C7 located below the virtual boundary (1230). Referring to Figure 12D, neighboring samples C1-C3 can each be padded by samples C5-C7 located above the virtual boundary (1240). Referring to Figure 12E, neighboring samples C4-C8 can be padded by samples C10, C11, C12, C11, and C10, respectively, which are located below the virtual boundary (1250). Referring to Figure 12F, neighboring samples C4-C8 can be padded by samples C10, C11, C12, C11, and C10, respectively, which are located above the virtual boundary (1260).

[0115] In some examples, the above explanation can be appropriately applied when the sample and neighboring samples are located to the left (or right) and right (or left) of the virtual boundary.

[0116] According to certain aspects of this disclosure, a picture can be partitioned based on a filtering process to improve encoding efficiency. In some examples, the CTU is also called the Largest Encoding Unit (LCU). In one example, the CTU or LCU may have a size of 64 x 64 pixels. In some embodiments, an LCU-aligned picture quadtree partition can be used for partitioning based on filtering. In some examples, an encoding unit-synchronous picture quadtree-based adaptive loop filter can be used. For example, a Luma picture can be partitioned into several multi-level quadtree partitions, with each partition boundary aligned to the LCU boundary. Each partition has its own filtering process and is therefore called a filter unit (FU).

[0117] In some examples, a two-pass coding flow can be used. In the first pass of the two-pass coding flow, the quadtree partitioning pattern of the picture and the best filter for each FU can be determined. In some embodiments, the determination of the quadtree partitioning pattern of the picture and the best filter for each FU is based on filtering distortion. Filtering distortion can be estimated by a fast filtering distortion estimation (FFDE) technique in the decision process. The picture is partitioned using quadtree partitions. The reconstructed picture can be filtered according to the determined quadtree partitioning pattern and the selected filters for all FUs.

[0118] In the second pass of the two-pass encoding flow, CU-synchronous ALF on / off control is performed. Depending on the ALF on / off result, the initial filtered picture is partially recovered by the reconstructed picture.

[0119] Specifically, in some examples, a top-down partitioning strategy is employed to divide a picture into multilevel quadtree partitions by using a rate-distortion criterion. Each partition is called a filter unit (FU). The partitioning process aligns the quadtree partitions to LCU boundaries. The encoding order of the FUs follows the z-scan order.

[0120] Figure 13 shows examples of partitioning according to some embodiments of the present disclosure. In the example in Figure 13, picture (1300) is partitioned into 10 FUs, with the encoding order being FU0, FU1, FU2, FU3, FU4, FU5, FU6, FU7, FU8, and FU9.

[0121] Figure 14 shows a quadtree partitioning pattern (1400) for a picture (1300). In the example in Figure 14, partitioning flags are used to indicate the picture partitioning pattern. For example, "1" indicates that a quadtree partition will be performed on the block, and "0" indicates that the block will not be partitioned further. In some examples, the smallest FU has an LCU size, and partitioning flags are not needed for the smallest FU. The partitioning flags are encoded in z order and transmitted, as shown in Figure 14.

[0122] In some examples, the filter for each FU is selected from two sets of filters based on a rate-distortion criterion. The first set contains the 1 / 2 symmetric square and rhombic filters derived for the current FU. The second set comes from a time-delay filter buffer, which stores the filters previously derived for the FUs of previous pictures. The filter with the minimum rate-distortion cost from these two sets can be selected for the current FU. Similarly, if the current FU is not the smallest FU and can be further divided into four child FUs, the rate-distortion costs of the four child FUs are calculated. By recursively comparing the rate-distortion costs of the divided and undivided cases, the picture quadtree partitioning pattern can be determined.

[0123] In some examples, a maximum quadtree partitioning level can be used to limit the maximum number of FUs. In one example, if the maximum quadtree partitioning level is 2, the maximum number of FUs is 16. Furthermore, during the quadtree partitioning decision, correlation values ​​can be reused to derive the Wiener coefficients of the 16 FUs (minimum FUs) at the lowest quadtree level. The remaining FUs can have their Wiener filters derived from the correlations of the 16 FUs at the lowest quadtree level. Therefore, in this example, only one framebuffer access is performed to derive the filter coefficients for all FUs.

[0124] After the quadtree partitioning pattern is determined, CU-synchronous ALF on / off control can be implemented to further reduce filtering distortion. By comparing the filtering distortion and unfiltering distortion at each leaf CU, the leaf CU can explicitly switch ALF on or off in its local region. In some examples, encoding efficiency can be further improved by redesigning the filter coefficients according to the ALF on / off results.

[0125] The cross-component filtering process can apply cross-component filters, such as cross-component adaptive loop filters (CC-ALF). A cross-component filter can refine a chroma component (e.g., a chroma CB corresponding to a chroma CB) using the luma sample values ​​of a luma component (e.g., luma CB). In one example, both the luma CB and chroma CB are contained within the CU.

[0126] Figure 15 shows a cross-component filter (e.g., CC-ALF) used to generate a chroma component according to one embodiment of the present disclosure. In some examples, Figure 15 shows the filtering process for a first chroma component (e.g., first chroma CB), a second chroma component (e.g., second chroma CB), and a luma component (e.g., luma CB). The luma component can be filtered by a sample adaptive offset (SAO) filter (1510) to produce an SAO-filtered luma component (1541). The SAO-filtered luma component (1541) is further filtered by an ALF luma filter (1516) to become a filtered luma CB (1561) (e.g., "Y").

[0127] The first chroma component can be filtered by an SAO filter (1512) and an ALF chroma filter (1518) to generate a first intermediate component (1552). Furthermore, the SAO-filtered chroma component (1541) can be filtered with respect to the first chroma component by a cross-component filter (e.g., CC-ALF) (1521) to generate a second intermediate component (1542). Subsequently, a filtered first chroma component (1562) (e.g., "Cb") can be generated based on at least one of the second intermediate component (1542) and the first intermediate component (1552). In one example, the filtered first chroma component (1562) (e.g., "Cb") can be generated by combining the second intermediate component (1542) and the first intermediate component (1552) using an adder (1522). A cross-component adaptive loop filtering process for the first chroma component may include steps performed by a CC-ALF (1521) and steps performed by, for example, an adder (1522).

[0128] The above description can be applied to a second chroma component. The second chroma component can be filtered by an SAO filter (1514) and an ALF chroma filter (1518) to produce a third intermediate component (1553). Furthermore, the SAO-filtered chroma component (1541) can be filtered by a cross-component filter (e.g., CC-ALF) (1531) on the second chroma component to produce a fourth intermediate component (1543). Subsequently, a filtered second chroma component (1563) (e.g., "Cr") can be generated based on at least one of the fourth intermediate component (1543) and the third intermediate component (1553). In one example, the filtered second chroma component (1563) (e.g., "Cr") can be generated by combining the fourth intermediate component (1543) and the third intermediate component (1553) using an adder (1532). In one example, a cross-component adaptive loop filtering process for a second chroma component may include steps performed by a CC-ALF (1531) and steps performed by, for example, an adder (1532).

[0129] Cross-component filters (e.g., CC-ALF(1521), CC-ALF(1531)) can operate by applying a linear filter with any suitable filter shape to the luma component (or luma channel) to refine each chroma component (e.g., the first chroma component, the second chroma component).

[0130] Figure 16 shows an example of a filter (1600) according to one embodiment of the present disclosure. The filter (1600) may include non-zero filter coefficients and zero filter coefficients. The filter (1600) has a diamond shape (1620) (shown as a black-filled circle) formed by the filter coefficients (1610). In one example, the non-zero filter coefficients in the filter (1600) are included in the filter coefficients (1610), and the filter coefficients not included in the filter coefficients (1610) are zero. Thus, the non-zero filter coefficients in the filter (1600) are included in the diamond shape (1620), and the filter coefficients not included in the diamond shape (1620) are zero. In one example, the number of filter coefficients in the filter (1600) is equal to the number of filter coefficients (1610), which is 18 in the embodiment shown in Figure 16.

[0131] A CC-ALF can include any suitable filter coefficients (also called CC-ALF filter coefficients). Referring back to Figure 15, CC-ALF(1521) and CC-ALF(1531) can have the same filter shape and the same number of filter coefficients as the diamond shape (1620) shown in Figure 16. In one example, the values ​​of the filter coefficients in CC-ALF(1521) are different from the values ​​of the filter coefficients in CC-ALF(1531).

[0132] In general, filter coefficients within a CC-ALF (e.g., non-zero filter coefficients) can be transmitted, for example, within an APS. In one example, the filter coefficients are a certain coefficient (e.g., 2 10It can be scaled by and rounded for fixed-point representation. The application of CC-ALF can be controlled by a variable block size and can be signaled by a context-encoded flag (e.g., a CC-ALF enable flag) received for each block of samples. Context-encoded flags, such as the CC-ALF enable flag, can be signaled at any appropriate level, such as the block level. The block size, along with the CC-ALF enable flag, can be received at the slice level for each chroma component. In some examples, block sizes of 16x16, 32x32, and 64x64 (in chroma samples) can be supported.

[0133] Figure 17 shows syntax examples for CC-ALF according to several embodiments of the present disclosure. In the examples in Figure 17, alf_ctb_cross_component_cross_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is an index indicating whether a cross-component Cb filter is used, and if so, the index of that cross-component Cb filter. For example, if alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is equal to 0, the cross-component Cb filter is not applied to the block of Cb color component samples at the luma position (xCtb, yCtb). If alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is not equal to 0, then alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is the index for the filter to be applied. For example, the cross-component Cb filter of the order alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is applied to the block of Cb color component samples at the luma position (xCtb, yCtb).

[0134] Furthermore, in the example in Figure 17, alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is used to indicate whether a cross-component Cr filter is used, and is the index of the cross-component Cr filter to be used. For example, if alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is equal to 0, the cross-component Cr filter is not applied to the block of Cr color component samples at the Luma position (xCtb, yCtb). If alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is not equal to 0, then alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is the index of the cross-component Cr filter. For example, the cross-component Cr filter of the order alf_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] can be applied to a block of Cr color component samples at the luma position (xCtb, yCtb).

[0135] In some examples, chroma subsampling techniques are used, so that the number of samples in each chroma block can be less than the number of samples in the luma block. The chroma subsampling format (also called the chroma subsampling format, specified, for example, by chroma_format_idc) can specify the chroma horizontal subsampling coefficient (e.g., SubWidthC) and the chroma vertical subsampling coefficient (e.g., SubHeightC) between each chroma block and its corresponding luma block. In one example, the chroma subsampling format is 4:2:0, so the chroma horizontal subsampling coefficient (e.g., SubWidthC) and the chroma vertical subsampling coefficient (e.g., SubHeightC) are 2, as shown in Figures 18A-18B. In another example, the chroma subsampling format is 4:2:2, so the chroma horizontal subsampling coefficient (e.g., SubWidthC) is 2 and the chroma vertical subsampling coefficient (e.g., SubHeightC) is 1. In one example, the chroma subsampling format is 4:4:4, and therefore the chroma horizontal subsampling coefficient (e.g., SubWidthC) and the chroma vertical subsampling coefficient (e.g., SubHeightC) are 1. The chroma sample type (also called the chroma sample position) can indicate the relative position of a chroma sample within a chroma block to at least one corresponding chroma sample within that chroma block.

[0136] Figures 18A–18B show exemplary positions of chroma samples relative to chroma samples according to embodiments of the present disclosure. Referring to Figure 18A, chroma sample (1801) is located in rows (1811)–(1818). The chroma sample (1801) shown in Figure 18A may represent a portion of a picture. In one example, a chroma block (e.g., chroma CB) contains chroma sample (1801). A chroma block may correspond to two chroma blocks with a chroma subsampling format of 4:2:0. In one example, each chroma block contains chroma sample (1803). Each chroma sample (e.g., chroma sample (1803(1))) corresponds to four chroma samples (e.g., chroma samples (1801(1))–(1801(4))). In one example, the four luma samples are the top-left sample (1801(1)), the top-right sample (1801(2)), the bottom-left sample (1801(3)), and the bottom-right sample (1801(4)). The chroma sample (e.g., (1803(1))) is located in the left-center position between the top-left sample (1801(1)) and the bottom-left sample (1801(3)), and the chroma sample type of the chroma block having chroma sample (1803) can be referred to as chroma sample type 0. Chroma sample type 0 indicates relative position 0, which corresponds to the left-center position midway between the top-left sample (1801(1)) and the bottom-left sample (1801(3)). The four luma samples (e.g., (1801(1)) to (1801(4))) can be referred to as neighboring luma samples of chroma sample (1803)(1).

[0137] In one example, each chroma block contains a chroma sample (1804). The above description of chroma sample (1803) can be applied to chroma sample (1804), and therefore, for brevity, a detailed description can be omitted. Each chroma sample (1804) can be located at the central position of the four corresponding luma samples, and the chroma sample type of a chroma block having chroma samples (1804) can be referred to as chroma sample type 1. Chroma sample type 1 indicates relative position 1 corresponding to the central positions of the four luma samples (e.g., (1801(1)) to (1801(4))). For example, one of the chroma samples (1804) can be located in the central part of luma samples (1801(1)) to (1801(4)).

[0138] In one example, each chroma block contains a chroma sample (1805). Each chroma sample (1805) can be located in the upper left position, which is copositional with the upper left sample of the four corresponding luma samples (1801). The chroma sample type of a chroma block containing chroma samples (1805) can be referred to as chroma sample type 2. Thus, each chroma sample (1805) is copositional with the upper left sample of the four luma samples (1801) corresponding to each chroma sample. Chroma sample type 2 indicates relative position 2, which corresponds to the upper left position of the four luma samples (1801). For example, one of the chroma samples (1805) can be located in the upper left position of luma samples (1801(1)) to (1801(4)).

[0139] In one example, each chroma block contains a chroma sample (1806). Each chroma sample (1806) can be positioned in the upper-center position between the corresponding upper-left sample and the corresponding upper-right sample, and the chroma sample type of a chroma block containing chroma samples (1806) can be referred to as chroma sample type 3. Chroma sample type 3 indicates relative position 3, which corresponds to the upper-center position between the upper-left sample (and the upper-right sample). For example, one of the chroma samples (1806) can be positioned in the upper-center position between chroma samples (1801(1))~(1801(4)).

[0140] In one example, each chroma block contains a chroma sample (1807). Each chroma sample (1807) can be located in a lower-left position, which is coposition with the lower-left sample of the four corresponding luma samples (1801). The chroma sample type of a chroma block containing chroma samples (1807) can be referred to as chroma sample type 4. Thus, each chroma sample (1807) is coposition with the lower-left sample of the four luma samples (1801) corresponding to each chroma sample. Chroma sample type 4 indicates relative position 4, which corresponds to the lower-left position of the four luma samples (1801). For example, one of the chroma samples (1807) can be located in a lower-left position of luma samples (1801(1)) to (1801(4)).

[0141] In one example, each chroma block contains a chroma sample (1808). Each chroma sample (1808) is located in the lower-center position between the lower-left and lower-right samples, and the chroma sample type of a chroma block containing chroma samples (1808) can be referred to as chroma sample type 5. Chroma sample type 5 indicates a relative position 5 corresponding to the lower-center position between the lower-left and lower-right samples of the four chroma samples (1801). For example, one of the chroma samples (1808) may be located between the lower-left and lower-right samples of chroma samples (1801(1)) to (1801(4)).

[0142] In general, any suitable chroma sample type can be used for a chroma subsampling format. Chroma sample types 0-5 are exemplary chroma sample types described in chroma subsampling format 4:2:0. Additional chroma sample types may be used for chroma subsampling format 4:2:0. Furthermore, other chroma sample types and / or variations of chroma sample types 0-5 can be used for other chroma subsampling formats such as 4:2:2 and 4:4:4. For example, a chroma sample type combining chroma samples (1805) and (1807) is used for chroma subsampling format 4:2:2.

[0143] For example, a luma block can be thought of as having alternating rows such as (1811) to (1812) that contain the top two samples (e.g., (1801(1)) to (1801(2))) and the bottom two samples (e.g., (1801(3)) to (1801(4))) of four luma samples (e.g., (1801(1)) to (1801(4))). Thus, rows (1811), (1813), (1815), and (1817) can be called the current row (also called the top field), and rows (1812), (1814), (1816), and (1818) can be called the next row (also called the bottom field). The four luma samples (e.g., (1801(1)) to (1801(4))) are located on the current line (e.g., (1811)) and the next line (e.g., (1812)). Relative positions 2 and 3 are located on the current line, relative positions 0 and 1 are located between each current line and its respective next line, and relative positions 4 and 5 are located on the next line.

[0144] Chroma samples (1803), (1804), (1805), (1806), (1807), or (1808) are located in rows (1851) to (1854) within each chroma block. The specific position in rows (1851) to (1854) may depend on the type of chroma sample. For example, for chroma samples (1803) to (1804) having chroma sample types 0 to 1, row (1851) is located between rows (1811) and (1812). For chroma samples (1805) to (1806) having chroma sample types 2 to 3, row (1851) is co-located with the current row (1811). For chromatic samples (1807)-(1808) having chromatic sample types 4-5, row (1851) is copositioned with the following row (1812). The above explanation can be appropriately applied to rows (1852)-(1854), and a detailed explanation is omitted for brevity.

[0145] Using any suitable scanning method, the luma blocks and corresponding chroma blocks described above can be displayed, stored, and / or transmitted in Figure 18A. In one example, sequential scanning is used.

[0146] Interlaced scanning can be used, as shown in Figure 18B. As previously mentioned, the chroma subsampling format is 4:2:0 (e.g., chroma_format_idc is equal to 1). In one example, the variable chroma location type (e.g., ChromaLocType) indicates the current row (e.g., ChromaLocType is chroma_sample_loc_type_top_field) or the next row (e.g., ChromaLocType is chroma_sample_loc_type_bottom_field). The current rows (1811), (1813), (1815), and (1817) and the next rows (1812), (1814), (1816), and (1818) can be scanned separately. For example, the current rows (1811), (1813), (1815), and (1817) are scanned first, followed by the next rows (1812), (1814), (1816), and (1818). The current row may contain luma sample (1801), and the next row may contain luma sample (1802).

[0147] Similarly, corresponding chroma blocks can be interlaced. Rows (1851) and (1853) containing unfilled chroma samples (1803), (1804), (1805), (1806), (1807), or (1808) can be referred to as the current row (or current chroma row), and rows (1852) and (1854) containing gray-filled chroma samples (1803), (1804), (1805), (1806), (1807), or (1808) can be referred to as the next row (or next chroma row). In one example, during interlaced scanning, rows (1851) and (1853) are scanned first, followed by rows (1852) and (1854).

[0148] In some cases, the constrained directional enhancement filtering technique can be used. The use of a constrained directional enhancement filter (CDEF) within a loop can remove coding artifacts while preserving image detail. In one example (e.g., HEVC), the sample adaptive offset (SAO) algorithm can achieve a similar goal by defining signal offsets for different classes of pixels. Unlike SAO, CDEF is a nonlinear spatial filter. In some cases, CDEF can be constrained to be easily vectorizable (i.e., implementable with single-instruction multiple-data (SIMD) operations). Note that other nonlinear filters, such as median filters and bilateral filters, cannot be treated in the same way.

[0149] In some cases, the amount of ringing artifacts in the encoded image tends to be approximately proportional to the quantization step size. While the amount of detail is a characteristic of the input image, the minimum detail retained in the quantized image also tends to be proportional to the quantization step size. For a given quantization step size, the amplitude of ringing is generally smaller than the amplitude of detail.

[0150] CDEF can be used to identify the orientation of each block, then to adaptively filter along the identified orientation, and to filter to a lesser degree along directions rotated 45 degrees from the identified orientation. In some examples, an encoder can explore the filter intensity, which can be explicitly signaled, allowing for a high degree of control over blurring.

[0151] Specifically, in some examples, orientation search is performed on the reconstructed pixels immediately after the deblocking filter. Since these pixels are available to the decoder, the decoder can search for orientation, and thus, in one example, orientation does not require signal transmission. In some examples, orientation search can operate with a block size that is small enough to properly handle non-linear edges, such as an 8x8 block, but large enough to reliably estimate orientation when applied to a quantized image. Also, having a constant orientation across the 8x8 area facilitates the vectorization of the filter. In some examples, each block (e.g., 8x8) can be compared to a perfectly oriented block to determine the difference. A perfectly oriented block is one in which all pixels along a line in one direction have the same value. In one example, an index of the difference between the block and the perfectly oriented block, such as the sum of squared differences (SSD) or root mean square (RMS) error, can be calculated. Next, a block with perfect orientation that has the smallest difference (e.g., smallest SSD, smallest RMS, etc.) can be determined, and the orientation of the determined block with perfect orientation can be the orientation that best matches the pattern within that block.

[0152] Figure 19 shows an example of direction finding according to one embodiment of the present disclosure. In this example, block (1910) is an 8x8 block that has been reconstructed and output from a deblocking filter. In the example in Figure 19, direction finding can determine the direction of block (1910) from eight directions indicated by (1920). Eight perfectly oriented blocks (1930) are formed, each corresponding to one of the eight directions (1920). A perfectly oriented block corresponding to a given direction is a block in which pixels along the line of that direction have the same value. Furthermore, it is possible to calculate an index of the difference between block (1910) and each of the perfectly oriented blocks (1930), such as SSD, RMS error. In the example in Figure 19, the RMS error is indicated by (1940). As indicated by (1943), the RMS error between block (1910) and the perfectly oriented block (1933) is smallest, and therefore direction (1923) is the direction that best matches the pattern of block (1910).

[0153] After the orientation of the block is identified, a nonlinear low-pass directional filter can be determined. For example, the filter taps of a nonlinear low-pass directional filter can be aligned along the identified orientation to reduce ringing while preserving directional edges or patterns. However, in some cases, directional filtering alone may not sufficiently reduce ringing. In one example, additional filter taps are also used for pixels that do not exist along the identified orientation. To reduce the risk of blurring, the additional filter taps are treated more conservatively. For this reason, the CDEF includes first-order and second-order filter taps. In one example, a complete 2-D CDEF filter can be expressed as equation (14).

number

[0154] In some cases, in-loop restoration schemes are used in video encoding after deblocking to generally denoise beyond the deblocking operation and improve edge quality. In one example, the in-loop restoration scheme is switchable within a frame for each appropriately sized tile. The in-loop restoration scheme is based on a separable symmetric Wiener filter, a dual self-guided filter with subspace projection, and a domain transform recursive filter. Since content statistics can change substantially within a frame, the in-loop restoration scheme is integrated into a switchable framework in which different schemes can be triggered in different regions of the frame.

[0155] According to one aspect of the disclosure, in-loop restoration (LR) (also called LR filtering) can use neighboring pixels during LR filtering, such as a window of pixels around the pixel to be filtered. In order to enable LR filtering at frame edges, in some examples, some boundary pixel values ​​of pixels at the frame boundary are copied to a boundary buffer before LR filtering is applied. In some examples, two copies of boundary pixels are stored in the boundary buffer. In one example, a first copy step is applied after the deblocking filter and before the CDEF, and the copy of the boundary pixel values ​​of the pixels at the frame boundary by the first copy step is denoted as COPY0. A second copy step is applied after the CDEF and before the LR filter, and the copy of the boundary pixel values ​​of the pixels at the frame boundary by the second copy step is denoted as COPY1. The boundary buffer is then used for padding boundary pixels during the LR filtering process, for example, when applying LR filtering to frame edges. The process of copying boundary pixel values ​​to the boundary buffer is denoted as boundary processing.

[0156] A separable symmetric Wiener filter can be one of the in-loop reconstruction schemes. In some examples, all pixels in a degraded frame can be reconstructed as a non-causally filtered version of the pixels in the surrounding w×w window, where w=2r+1 is odd for an integer r. The 2D filter taps are in the form of a column vectorized w 2 If represented by a vector F with 1 element, the filter parameter is F=H through direct LMMSE optimization. -1 This will be given by M. Here, H = E[XX T ] is the autocovariance of x, and w in the w×w window around the pixel. 2 This is a column vectorized version of the individual samples, where M=E[YX T] is the cross-correlation of x with the scalar source sample y, which is to be estimated. In one example, the encoder can estimate H and M from the deblocked frame and realized values ​​in the source, and the resulting filter F can be sent to the decoder. However, this is because w 2 Not only does transmitting individual taps incur a substantial bitrate cost, but non-separable filtering also makes decoding excessively complex. In some embodiments, several additional constraints are imposed on the properties of F. The first constraint is that F is separable so that filtering can be implemented as separable horizontal and vertical w-tap convolutions. The second constraint is that the horizontal and vertical filters are symmetrical. The third constraint is that the sum of both the horizontal and vertical filter coefficients is assumed to be 1.

[0157] Dual self-guided filtering with subspace projection can be one of the in-loop reconstruction schemes. Guided filtering is an image filtering technique in which a local linear model, shown by equation (15) below, is used to compute a filtered output y from an unfiltered sample x. Equation (15) y = Fx + G Here, F and G are determined based on the statistics of the degraded image and the guide image of the neighboring pixels of the filtered image. If the guide image is the same as the degraded image, the resulting so-called self-guided filtering has the effect of smoothing that preserves edges. In one example, a specific form of self-guided filtering can be used. This specific form of self-guided filtering depends on two parameters, radius r and noise parameter e, and is enumerated as the following steps: 1. Mean μ and variance σ of each pixel in the (2r+1)×(2r+1) window surrounding each pixel. 2 This step can be efficiently implemented using box filtering based on integrated imaging. 2. For all pixels, f = σ2 / (σ 2 Calculate g=(1-f)μ (+e). 3. Calculate F and G for each pixel as the average of the f and g values ​​in a 3x3 window surrounding the pixel to be used.

[0158] The specific shape of the self-guided filter is controlled by r and e, where a larger r means greater spatial variance and a larger e means greater range variance.

[0159] Figure 20 shows examples illustrating subspace projections in several cases. As shown in Figure 20, neither the reconstructed X1 nor X2 is close to the source Y, but appropriate multipliers {α,β} can bring them much closer to the source Y, as long as they are moving in the somewhat correct direction.

[0160] In some cases, a technique called frame super-resolution (FSR) is used to improve the perceived quality of the decoded picture. Generally, the FSR process is applied at low bitrates and involves four steps. In the first step, the source video is downscaled on the encoder side as a non-normative procedure. In the second step, the downscaled video is encoded, followed by deblocking and CDEF filtering processes. In the third step, a linear upscaling process is applied as a normative procedure to restore the encoded video to its original spatial resolution. In the fourth step, a loop restoration filter is applied to resolve some of the lost high frequencies. In some examples, the last two steps can be collectively called the super-resolution process. Similarly, on the decoder side, the decoding, deblocking, and CDEF processes can be applied at lower spatial resolutions. The frame then passes through the super-resolution process. In some examples, the upscaling and downscaling processes are applied only to the horizontal dimension to reduce the line buffer overhead of the hardware implementation.

[0161] In some cases (e.g., HEVC), a filtering technique called Sample Adaptive Offset (SAO) can be used. In some cases, SAO is applied to the reconstructed signal after a deblocking filter. SAO can use an offset value given in the slice header. In some cases, for luma samples, the encoder can decide whether to apply (enable) SAO for a given slice. When SAO is enabled, the current picture allows the encoded unit to be recursively divided into four subregions, and each subregion can select an SAO type from several SAO types based on the features within that subregion.

[0162] Figure 21 shows Table (2100) of several SAO types according to one embodiment of the present disclosure. Table (2100) shows SAO types 0 to 6. Note that SAO type 0 is used to indicate that no SAO is applied. Also, each SAO type from SAO type 1 to SAO type 6 includes multiple categories. SAO can reduce distortion by classifying reconstructed pixels of a subregion into categories and adding an offset to the pixels of each category within the subregion. In some examples, edge characteristics can be used for pixel classification in SAO types 1 to 4, and pixel intensity can be used for pixel classification in SAO types 5 to 6.

[0163] Specifically, in certain embodiments such as SAO types 5-6, a band offset (BO) can be used to classify all pixels in a subregion into multiple bands. Each band in the multiple bands contains pixels within the same intensity interval. In some examples, the intensity range is evenly divided into multiple intervals, such as 32 intervals from zero to the maximum intensity value (e.g., 255 for 8-bit pixels), and each interval is associated with an offset. Furthermore, in one example, the 32 bands are divided into two groups, such as a first group and a second group. The first group contains the central 16 bands (e.g., 16 intervals in the middle of the intensity range), and the second group contains the remaining 16 bands (e.g., 8 intervals on the lower side of the intensity range and 8 intervals on the higher side of the intensity range). In one example, only the offset of one of the two groups is transmitted. In some embodiments, when a pixel classification operation using BOs is used, the five most significant bits of each pixel can be directly used as a band index.

[0164] Furthermore, in certain embodiments such as SAO types 1-4, edge offsets (EOs) can be used for pixel classification and offset determination. For example, pixel classification can be determined based on one-dimensional 3-pixel patterns, taking edge orientation information into account.

[0165] Figure 22 shows examples of 3-pixel patterns for pixel classification at edge offsets in several cases. In the examples in Figure 22, the first pattern (2210) (shown by three gray pixels) is called the 0-degree pattern (the horizontal direction is associated with the 0-degree pattern), the second pattern (2220) (shown by three gray pixels) is called the 90-degree pattern (the vertical direction is associated with the 90-degree pattern), the third pattern (2230) (shown by three gray pixels) is called the 135-degree pattern (the 135-degree diagonal direction is associated with the 135-degree pattern), and the fourth pattern (2240) (shown by three gray pixels) is called the 45-degree pattern (the 45-degree diagonal direction is associated with the 45-degree pattern). In one example, one of the four directional patterns shown in Figure 22 can be selected considering edge direction information for a subregion. The selection can be sent in the encoded video bitstream, in one example, as side information. Next, pixels within a subregion can be classified into several categories by comparing each pixel with two neighboring pixels in the direction associated with the directional pattern.

[0166] Figure 23 shows a table (2300) of pixel classification rules for edge offsets in several examples. Specifically, pixel c (also shown in each pattern in Figure 22) is compared with two neighboring pixels (shown in gray in each pattern in Figure 22), and pixel c can be classified into one of categories 0 to 4 based on the comparison according to the pixel classification rules shown in Figure 23.

[0167] In some embodiments, the decoder-side SAO can be operated independently of the maximum coding unit (LCU) (e.g., CTU) to save line buffers. In some examples, when a classification pattern of 90, 135, and 45 degrees is selected, the pixels in the top and bottom rows of each LCU are not SAO-processed. When a pattern of 0, 135, and 45 degrees is selected, the pixels in the leftmost and rightmost columns of each LCU are not SAO-processed.

[0168] Figure 24 shows an example of syntax (2400) where signaling may be necessary for a CTU when parameters are not merged from a neighboring CTU. For example, the syntax element sao_type_idx[cldx][rx][ry] can be signaled to indicate the SAO type of a subregion. The SAO type can be BO (band offset) or EO (edge ​​offset). A value of sao_type_idx[cldx][rx][ry] indicates that the SAO is off, values ​​from 1 to 4 indicate that one of the four EO categories corresponding to 0°, 90°, 135°, and 45° is used, and a value of 5 indicates that BO is used. In the example in Figure 24, each of the BO and EO types has four SAO offset values ​​that are signaled (sao_offset[cIdx][rx][ry][0] to sao_offset[cIdx][rx][ry][3]).

[0169] Generally, a filtering process can generate an output using a reconstructed sample of a first color component as input (e.g., Y, Cb, or Cr, or R, G, or B), and the output of the filtering process is applied to a second color component, which may be the same as the first color component or a different color component from the first color component.

[0170] In a related example of cross-component filtering (CCF), the filter coefficients are derived based on several mathematical formulas. The derived filter coefficients are transmitted from the encoder to the decoder and are used to generate an offset using a linear combination. The generated offset is then added to the reconstructed sample as part of the filtering process. For example, the offset is generated based on a linear combination of the luma sample and the filtering coefficients, and the generated offset is added to the reconstructed chroma sample. This related example of CCF is based on the assumption of a linear mapping relationship between the reconstructed luma sample value and the delta value between the original chroma sample and the reconstructed chroma sample. However, the mapping between the reconstructed luma sample value and the delta value between the original chroma sample and the reconstructed chroma sample does not necessarily follow a linear mapping process, and therefore the coding performance of CCF may be limited under the assumption of a linear mapping relationship.

[0171] In some examples, nonlinear mapping techniques can be used in cross-component filtering and / or same-color component filtering with minimal signal transfer overhead. In one example, nonlinear mapping techniques can be used in cross-component filtering to generate cross-component sample offsets. In another example, nonlinear mapping techniques can be used in same-color component filtering to generate local sample offsets.

[0172] For convenience, a filtering process that uses nonlinear mapping techniques can be referred to as sample offset by nonlinear mapping (SO-NLM). In a cross-component filtering process, SO-NLM can be called cross-component sample offset (CCSO). In color component filtering, SO-NLM can be called local sample offset (LSO). A filter that uses nonlinear mapping techniques can be called a nonlinear mapping-based filter. Nonlinear mapping-based filters can include CCSO filters, LSO filters, etc.

[0173] In one example, CCSO and LSO can be used as loop filtering to reduce distortion in the reconstructed sample. CCSO and LSO do not depend on the assumption of linear mapping used in the relevant exemplary CCF. For example, CCSO does not depend on the assumption of a linear mapping relationship between the chroma-reconstructed sample value and the delta value between the original chroma sample and the chroma-reconstructed sample. Similarly, LSO does not depend on the assumption of a linear mapping relationship between the reconstructed sample value of a color component and the delta value between the original sample of that color component and the reconstructed sample of that color component.

[0174] The following description describes the SO-NLM filtering process. This process uses a reconstructed sample of the first color component as input (e.g., Y, Cb, or Cr, or R, G, or B) to generate an output, and the output of the filtering process is applied to the second color component. If the second color component is the same as the first color component, this description applies to LSO. If the second color component is different from the first color component, this description applies to CCSO.

[0175] In SO-NLM, the nonlinear mapping is derived on the encoder side. The nonlinear mapping is between the reconstructed sample of the first color component within the filter support region and the offset applied to the second color component within the filter support region. If the second color component is the same as the first color component, the nonlinear mapping is used in LSO. If the second color component is different from the first color component, the nonlinear mapping is used in CCSO. The domain of the nonlinear mapping is determined by different combinations of processed input reconstructed samples (also called combinations of possible reconstructed sample values).

[0176] The SO-NLM technique can be explained using a specific example in which a reconstructed sample is determined from a first color component located within a filter support area (also called the "filter support region"). The filter support region is the area in which a filter can be applied, and the filter support region can have any preferred shape.

[0177] Figure 25 shows examples of filter support regions (2500) according to some embodiments of the present disclosure. The filter support region (2500) includes four reconstructed samples of the first color components P0, P1, P2, and P3. In the example of Figure 25, the four reconstructed samples can form a cross shape in the vertical and horizontal directions, and the center position of the cross shape is the position of the sample to be filtered. The sample at the center position with the same color components as P0-P3 is represented by C. The sample at the center position with the second color component is represented by F. The second color component may be the same as the first color components P0-P3, or it may be different from the first color components P0-P3.

[0178] Figure 26 shows an example of another filter support region (2600) according to some embodiments of the present disclosure. The filter support region (2600) includes four reconstructed samples P0, P1, P2, and P3 of a first color component that form a square shape. In the example of Figure 26, the center position of the square shape is the position of the sample to be filtered. A sample at the center position with the same color component as P0-P3 is represented by C. A sample at the center position with a second color component is represented by F. The second color component may be the same as the first color component of P0-P3, or it may be different from the first color component of P0-P3.

[0179] The reconstructed sample is fed into an SO-NLM filter and processed appropriately to form filter taps. In one example, the location of the reconstructed sample that is input to the SO-NLM filter is called the filter tap location. In a particular example, the reconstructed sample is processed in the following two steps:

[0180] In the first step, the delta values ​​between P0, P3, and C are calculated. For example, m0 represents the delta value between P0 and C, m1 represents the delta value between P1 and C, m2 represents the delta value between P2 and C, and m3 represents the delta value between P3 and C.

[0181] In the second step, the delta values ​​m0 to m3 are further quantized, and the quantized values ​​are represented as d0, d1, d2, and d3. In one example, the quantized values ​​can be -1, 0, or 1, depending on the quantization process. For example, if m is less than -N (where N is a positive value and is called the quantization step size), the value m can be quantized to -1; if m is in the range [-N, N], the value m can be quantized to 0; and if m is greater than N, the value m can be quantized to 1. In some examples, the quantization step size N can be one of 4, 8, 12, 16, etc.

[0182] In some embodiments, the quantized values ​​d0 to d3 are filter taps that can be used to identify a single combination within a filter domain. For example, filter taps d0 to d3 can form a combination within a filter domain. Since each filter tap can have three quantized values, if four filter taps are used, the filter domain will contain 81 (3 × 3 × 3 × 3) combinations.

[0183] Figures 27A-27C show a table (2700) having 81 combinations according to one embodiment of the present disclosure. The table (2700) contains 81 rows corresponding to the 81 combinations. In each row corresponding to a combination, the first column contains the index of the combination, the second column contains the value of filter tap d0 for the combination, the third column contains the value of filter tap d1 for the combination, the fourth column contains the value of filter tap d2 for the combination, the fifth column contains the value of filter tap d3 for the combination, and the sixth column contains the offset value associated with the combination for nonlinear mapping. In one example, when filter taps d0-d3 are determined, the offset values ​​(represented by s) associated with the combinations d0-d3 can be determined according to the table (2700). In one example, the offset values ​​s0-s80 are integers such as 0, 1, -1, 3, -3, 5, -5, -7, etc.

[0184] In some embodiments, the final filtering process of SO-NLM can be applied as shown in equation (16). f'=clip(f+s) Equation (16) Here, f is a reconstructed sample of the second color component to be filtered, and s is an offset value determined according to a filter tap, which is the result of processing the reconstructed sample of the first color component, for example using a table (2700). The sum of the reconstructed sample F and the offset value s is further clipped within a range related to the bit depth to determine the final filtered sample f' of the second color component.

[0185] Note that in the case of LSO, the second color component described above is the same as the first color component, while in the case of CCSO, the second color component described above may be different from the first color component.

[0186] The above description may be adapted for other embodiments of this disclosure.

[0187] In some examples, on the encoder side, the encoding device can derive a mapping between a reconfigured sample of a first color component within the filter support region and an offset to be applied to a reconfigured sample of a second color component. The mapping can be any suitable linear or nonlinear mapping. In this case, the filtering process can be applied on the encoder side and / or decoder side based on the mapping. For example, the mapping is appropriately communicated to the decoder (e.g., the mapping is included in the encoded video bitstream sent from the encoder side to the decoder side), and the decoder can then perform the filtering process based on the mapping.

[0188] The performance of nonlinear mapping-based filters, such as CCSO filters and LSO filters, depends on the filter geometry configuration. The filter geometry configuration (also called the filter shape) can refer to the properties of the pattern formed by the filter tap locations. The pattern can be defined by various parameters, such as the number of filter taps, the geometric shape of the filter tap locations, and the distance of the filter tap locations to the center of the pattern. Using a fixed filter geometry configuration may limit the performance of nonlinear mapping-based filters.

[0189] As shown in Figures 24 and 25 and Figures 27A-27C, some examples use a 5-tap filter design for the filter geometry configuration of a nonlinear mapping-based filter. The 5-tap filter design can use tap positions at P0, P1, P2, P3, and C. The 5-tap filter design for the filter geometry configuration can produce a lookup table (LUT) with 81 entries, as shown in Figures 27A-27C. The LUT of the sample offset needs to be signaled from the encoder side to the decoder side, and the signaling of the LUT contributes to a large portion of the signaling overhead, which can affect the coding efficiency when using a nonlinear mapping-based filter. According to some aspects of this disclosure, the number of filter taps can differ from 5. In some examples, the number of filter taps can be reduced, and information within the filter support region can still be captured, while coding efficiency can be improved.

[0190] In some examples, the filter shape configurations within a group for nonlinear mapping-based filters each have three filter taps.

[0191] Figure 28 shows seven filter shape configurations for three filter taps in one example. Specifically, the first filter shape configuration includes three filter taps at positions labeled "1" and "c", where position "c" is the center position of position "1". The second filter shape configuration includes three filter taps at positions labeled "2" and "c", where position "c" is the center position of position "2". The third filter shape configuration includes three filter taps at positions labeled "3" and "c", where position "c" is the center position of position "3". The fourth filter shape configuration includes three filter taps at positions labeled "4" and "c", where position "c" is the center position of position "4". The fifth filter shape configuration includes three filter taps at positions labeled "5" and "c", where position "c" is the center position of position "5". The sixth filter shape configuration includes three filter taps at the position labeled "6" and position "c", where position "c" is the center position of position "6". The seventh filter shape configuration includes three filter taps at the position labeled "7" and position "c", where position "c" is the center position of position "7".

[0192] Aspects of this disclosure provide techniques for integrating video processing tools such as filtering, boundary processing, and clipping tools. In some examples, nonlinear mapping-based filters (e.g., CCSO, LSO) and other tools (e.g., clipping modules) are placed after the deblocking filter and before the LR filter, and a boundary processing process that stores two copies of boundary pixels is also applied after the deblocking filter and before the LR filter. This disclosure provides various configurations for including nonlinear mapping-based filters, boundary processing modules, and / or clipping modules after the deblocking filter and before the LR filter.

[0193] Figure 29 shows block diagrams of loop filter chains (2900) in several examples. A loop filter chain (2900) includes multiple filters connected in series within a filter chain. In one example, a loop filter chain (2900) can be used like a loop filtering unit, such as a loop filter unit (556). A loop filter chain (2900) can be used in an encoding or decoding loop before the reconstructed picture is stored in a decoding picture buffer, such as a reference picture memory (557). The loop filter chain (2900) receives input reconstructed samples from a preprocessing module, applies filters to the reconstructed samples, and generates output reconstructed samples.

[0194] The loop filter chain (2900) can include any suitable filters. In the example in Figure 29, the loop filter chain (2900) includes a deblocking filter (labeled Deblocking), a constraint direction enhancement filter (labeled CDEF), and an in-loop reconstruction filter (labeled LR), all connected in a chain. The loop filter chain (2900) has an input node (2901), an output node (2909), and several intermediate nodes (2902)~(2903). The input node (2901) of the loop filter chain (2900) receives the input reconstructed sample from the preprocessing module, and the input reconstructed sample is provided to the deblocking filter. The intermediate node (2902) receives the reconstructed sample (after processing by the deblocking filter) from the deblocking filter and provides the reconstructed sample to the CDEF for further filtering. The intermediate node (2903) receives the reconstructed sample (after processing by the CDEF) from the CDEF and provides the reconstructed sample to the LR filter for further filtering. The output node (2909) receives the output reconstructed sample (after processing by the LR filter) from the LR filter. The output reconstructed sample can then be provided to other processing modules, such as a post-processing module, for further processing.

[0195] Note that the following explanation describes a technique for using nonlinear mapping-based filters in a loop filter chain (2900). This technique can also be used in other suitable loop filter chains.

[0196] According to several aspects of the disclosure, when a nonlinear mapping-based filter is used in a loop filter chain, at least one of the copies of boundary pixel values ​​(COPY0 and COPY1) used by boundary processing in the LR filter is related to the nonlinear mapping-based filter. In some examples, reconstructed samples from which boundary pixel values ​​are acquired and buffered can be input to the nonlinear mapping-based filter. In some examples, reconstructed samples from which boundary pixel values ​​are acquired and buffered can be the result of applying the nonlinear mapping-based filter. In some examples, reconstructed samples from which boundary pixel values ​​are acquired and buffered can be combined with a sample offset generated by the nonlinear mapping-based filter.

[0197] In some examples, pixels after the deblocking filter and before the CDEF or nonlinear mapping-based filter are used for COPY0, and pixels after the application of the CDEF or nonlinear mapping-based filter and before the LR filter are used for COPY1.

[0198] Figure 30 shows an example of a loop filter chain (3000) including a nonlinear mapping-based filter and a CDEF between the input and output of the nonlinear mapping-based filter. The loop filter chain (3000) can be used in place of a loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3000), a deblocking filter generates a first intermediate reconstructed sample at a first intermediate node (3011), and the first intermediate reconstructed sample is input to a CDEF and a nonlinear mapping-based filter (labeled SO-NLM). The CDEF is applied to the first intermediate reconstructed sample to generate a second intermediate reconstructed sample at a second intermediate node (3012). The nonlinear mapping-based filter generates a sample offset SO based on the first intermediate reconstructed sample. The sample offset SO is combined with the second intermediate reconstructed sample to generate a third intermediate reconstructed sample at a third intermediate node (3013). Next, the LR filter is applied to the third intermediate reconstructed sample to generate the output of the loop filter chain (3000). Furthermore, the first intermediate reconstructed sample at the first intermediate node (3011) is used to obtain a first copy COPY0 of the boundary pixels for boundary processing in the LR filter, and the third intermediate reconstructed sample at the third intermediate node (3013) is used to obtain a second copy COPY1 of the boundary pixels for boundary processing in the LR filter.

[0199] In some examples, pixels after the deblocking filter and before the application of CDEF or nonlinear mapping-based filters are used for COPY0, and pixels after the application of CDEF and before the application of nonlinear mapping-based filters are used for COPY1.

[0200] Figure 31 shows an example of a loop filter chain (3100) including a nonlinear mapping-based filter and a CDEF between the input and output of the nonlinear mapping-based filter. The loop filter chain (3100) can be used in place of a loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3100), a deblocking filter generates a first intermediate reconstructed sample at a first intermediate node (3111), and the first intermediate reconstructed sample is input to a CDEF and a nonlinear mapping-based filter (labeled SO-NLM). The CDEF is applied to the first intermediate reconstructed sample to generate a second intermediate reconstructed sample at a second intermediate node (3112). The nonlinear mapping-based filter generates a sample offset SO based on the first intermediate reconstructed sample. The sample offset SO is combined with the second intermediate reconstructed sample to generate a third intermediate reconstructed sample at a third intermediate node (3113). Next, the LR filter is applied to the third intermediate reconstruction sample to produce the output of the loop filter chain (3100). Furthermore, the first intermediate reconstruction sample at the first intermediate node (3111) is used to obtain a first copy COPY0 of the boundary pixels for boundary processing in the LR filter, and the second intermediate reconstruction sample at the second intermediate node (3012) is used to obtain a second copy COPY1 of the boundary pixels for boundary processing in the LR filter.

[0201] In some examples, a nonlinear mapping-based filter is connected in series with other filters in a loop filter chain, and there are no other filters between the input and output of the nonlinear mapping-based filter. In one example, the nonlinear mapping-based filter is applied after the CDEF. Pixels after the deblocking filter and before the CDEF are used for COPY0, and pixels after the nonlinear mapping-based filter and before the LR filter are used for COPY1.

[0202] Figure 32 shows an example of a loop filter chain (3200). The loop filter chain (3200) can be used in place of a loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3200), the CDEF is located between the first intermediate node (3211) and the second intermediate node (3212), and a nonlinear mapping-based filter (labeled SO-NLM) is applied at the second intermediate node (3212) of the loop filter chain (3200). Specifically, the deblocking filter generates the first intermediate reconstruction sample at the first intermediate node (3211), the CDEF is applied to the first intermediate reconstruction sample to generate the second intermediate reconstruction sample at the second intermediate node (3212), and the second intermediate reconstruction sample is the input to the nonlinear mapping-based filter. Based on this input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the second intermediate reconstructed sample at the second intermediate node (3212) to generate the third intermediate reconstructed sample at the third intermediate node (3213). The LR filter is applied to the third intermediate reconstructed sample to generate the output of the loop filter chain (3200). Furthermore, the first intermediate reconstructed sample at the first intermediate node (3211) is used to obtain a first copy COPY0 of the boundary pixels for boundary processing in the LR filter, and the third intermediate reconstructed sample at the third node (3213) is used to obtain a second copy COPY1 of the boundary pixels for boundary processing in the LR filter.

[0203] In one example, a nonlinear mapping-based filter is applied before the CDEF. Pixels after the deblocking filter and before the nonlinear mapping-based filter are used for COPY0, and pixels after the CDEF and before the LR filter are used for COPY1.

[0204] Figure 33 shows an example of a loop filter chain (3300). The loop filter chain (3300) can be used in place of a loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3300), a nonlinear mapping-based filter (labeled SO-NLM) is applied at the first intermediate node (3311) of the loop filter chain (3300), and the CDEF is located between the second intermediate node (3312) and the third intermediate node (3313). Specifically, the deblocking filter generates the first intermediate reconstruction sample at the first intermediate node (3311). The first intermediate reconstruction sample is the input to the nonlinear mapping-based filter. Based on this input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is combined with the first intermediate reconstruction sample at the first intermediate node (3311) to generate the second intermediate reconstruction sample at the second intermediate node (3312). CDEF is applied to the second intermediate reconstructed sample to generate the third intermediate reconstructed sample at the third intermediate node (3313). The LR filter is applied to the third intermediate reconstructed sample to generate the output of the loop filter chain (3300). Furthermore, the first intermediate reconstructed sample at the first intermediate node (3311) is used to obtain a first copy COPY0 of the boundary pixels for boundary processing in the LR filter, and the third intermediate reconstructed sample at the third intermediate node (3313) is used to obtain a second copy COPY1 of the boundary pixels for boundary processing in the LR filter.

[0205] According to one aspect of this disclosure, enabling / disabling boundary processing of an LR filter (e.g., obtaining and using COPY0 and COPY1 in an LR filter) depends on enabling / disabling other filtering tools such as nonlinear mapping-based filters, CDEF, and / or FSR in the loop filtering chain.

[0206] In some examples, enabling / disabling boundary processing depends on enabling / disabling a non-linear mapping-based filter and / or CDEF and / or FSR. For example, the flag En-Boundary is used to indicate enabling (e.g., the flag En-Boundary has a binary 1) or disabling (e.g., the flag En-Boundary has a binary 0) of boundary processing. The flag En-SO-NLM is used to indicate enabling (e.g., the flag En-SO-NLM has a binary 1) or disabling (e.g., the flag En-SO-NLM has a binary 0) of the application of the non-linear mapping-based filter. The flag En-CDEF is used to indicate enabling (e.g., the flag En-CDEF has a binary 1) or disabling (e.g., the flag En-CDEF has a binary 0) of the application of CDEF. The flag En-FSR is used to indicate enabling (e.g., the flag En-FSR has a binary 1) or disabling (e.g., the flag En-FSR has a binary 0) of the application of FSR. In that case, in one example, the flag En-Boundary is a logical combination of the flags En-SO-NLM, En-CDEF, and En-FSR. Note that any suitable logical operators such as AND, OR, NOT, etc. can be used.

[0207] In some cases, enabling / disabling boundary processing depends on enabling / disabling nonlinear mapping-based filters and / or CDEFs. For example, the flag En-Boundary is used to indicate whether boundary processing is enabled (e.g., the flag En-Boundary has binary 1) or disabled (e.g., the flag En-Boundary has binary 0). The flag En-SO-NLM is used to indicate whether the application of nonlinear mapping-based filters is enabled (e.g., the flag En-SO-NLM has binary 1) or disabled (e.g., the flag En-SO-NLM has binary 0). The flag En-CDEF is used to indicate whether the application of CDEFs is enabled (e.g., the flag En-CDEF has binary 1) or disabled (e.g., the flag En-CDEF has binary 0). In one example, the flag En-Boundary is a logical combination of the flags En-SO-NLM and En-CDEF. Note that any appropriate logical operator such as AND, OR, NOT, etc. can be used.

[0208] In some examples, enabling / disabling boundary processing depends on enabling / disabling nonlinear mapping-based filters and / or FSRs. For example, the flag En-Boundary is used to indicate whether boundary processing is enabled (e.g., the flag En-Boundary has binary 1) or disabled (e.g., the flag En-Boundary has binary 0). The flag En-SO-NLM is used to indicate whether the application of nonlinear mapping-based filters is enabled (e.g., the flag En-SO-NLM has binary 1) or disabled (e.g., the flag En-SO-NLM has binary 0). The flag En-FSR is used to indicate whether the application of FSRs is enabled (e.g., the flag En-FSR has binary 1) or disabled (e.g., the flag En-FSR has binary 0). In one example, the flag En-Boundary is a logical combination of the flags En-SO-NLM and En-FSR. Note that any appropriate logical operator such as AND, OR, NOT, etc., can be used.

[0209] In some examples, enabling / disabling boundary processing depends on a non-linear mapping-based filter. For example, the flag En-Boundary is used to indicate enabling (e.g., the flag En-Boundary has a binary value of 1) or disabling (e.g., the flag En-Boundary has a binary value of 0) of boundary processing. The flag En-SO-NLM is used to indicate enabling (e.g., the flag En-SO-NLM has a binary value of 1) or disabling (e.g., the flag En-SO-NLM has a binary value of 0) of the application of the non-linear mapping-based filter. In that case, in one example, the flag En-Boundary can be the flag En-SO-NLM, or can be the logical NOT of the flag En-SO-NLM.

[0210] In one embodiment, enabling / disabling boundary processing depends on enabling / disabling of CDEF and / or FSR.

[0211] In some examples, enabling / disabling boundary processing depends on enabling / disabling of CDEF and / or FSR. For example, the flag En-Boundary is used to indicate enabling (e.g., the flag En-Boundary has a binary value of 1) or disabling (e.g., the flag En-Boundary has a binary value of 0) of boundary processing. The flag En-CDEF is used to indicate enabling (e.g., the flag En-CDEF has a binary value of 1) or disabling (e.g., the flag En-CDEF has a binary value of 0) of the application of CDEF. The flag En-FSR is used to indicate enabling (e.g., the flag En-FSR has a binary value of 1) or disabling (e.g., the flag En-FSR has a binary value of 0) of the application of FSR. In that case, in one example, the flag En-Boundary is a logical combination of the flag En-CDEF and the flag En-FSR. Note that any suitable logical operator such as AND, OR, NOT, etc. can be used.

[0212] In some examples, enabling / disabling boundary processing depends on enabling / disabling CDEF. For example, the flag En-Boundary is used to indicate whether boundary processing is enabled (e.g., flag En-Boundary has binary 1) or disabled (e.g., flag En-Boundary has binary 0). The flag En-CDEF is used to indicate whether the application of nonlinear mapping-based filters is enabled (e.g., flag En-CDEF has binary 1) or disabled (e.g., flag En-CDEF has binary 0). In this case, for example, flag En-Boundary can be flag En-CDEF, or it can be a logical NOT of flag En-CDEF.

[0213] In one embodiment, enabling / disabling boundary processing depends on enabling / disabling FSR.

[0214] In some cases, enabling / disabling boundary processing depends on enabling / disabling FSR. For example, the flag En-Boundary is used to indicate whether boundary processing is enabled (e.g., flag En-Boundary has binary 1) or disabled (e.g., flag En-Boundary has binary 0). The flag En-FSR is used to indicate whether the application of nonlinear mapping-based filters is enabled (e.g., flag En-FSR has binary 1) or disabled (e.g., flag En-FSR has binary 0). In this case, for example, flag En-Boundary can be flag En-FSR, or it can be a logical NOT of flag En-FSR.

[0215] According to some aspects of this disclosure, pixel values ​​are clipped before the LR filter. In some examples, pixels after applying the CDEF are clipped first and designated as CLIP0. Then, pixels after applying the nonlinear mapping-based filter are clipped and designated as CLIP1. Note that in some examples, pixel values ​​are clipped to a meaningful and appropriate range for the pixel value. In one example, if the pixel value is represented by 8 bits, the pixel value may be clipped to the range [0, 255].

[0216] Note that this explanation uses a nonlinear mapping-based filter as an example to demonstrate a technique for buffering boundary pixel values ​​at different nodes along a loop filter chain, and / or a technique for clipping pixel values ​​at different nodes along a loop filter chain. This technique can be used when other coding tools, such as the cross component sample adaptive offset (CCSAO) tool, are applied after the deblocking filter and before the LR filter.

[0217] Figure 34 shows an example of a loop filter chain (3400) that includes a nonlinear mapping-based filter having a CDEF between the input and output of the nonlinear mapping-based filter. The loop filter chain (3400) can be used in place of a loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3400), a nonlinear mapping-based filter (labeled SO-NLM) is applied between two intermediate nodes. The deblocking filter generates a first intermediate reconstruction sample at the first intermediate node (3411). The first intermediate reconstruction sample is the input to both the CDEF and the nonlinear mapping-based filter. Based on the first intermediate reconstruction sample, the CDEF is applied to generate a second intermediate reconstruction sample at the second intermediate node (3412). Also based on the first intermediate reconstruction sample, the nonlinear mapping-based filter generates a sample offset (SO). The second intermediate reconstruction sample is clipped to generate CLIP0. CLIP0, combined with the sample offset, generates a third intermediate reconstructed sample at the third intermediate node (3413). This third intermediate reconstructed sample is then clipped to generate CLIP1. CLIP1 is then provided to the LR filter for further filtering.

[0218] Figure 35 shows an example of a loop filter chain (3500) that includes a nonlinear mapping-based filter having a CDEF between the input and output of the nonlinear mapping-based filter. The loop filter chain (3500) can be used as an alternative to the loop filter chain (2900) in an encoding or decoding device. In the loop filter chain (3500), a nonlinear mapping-based filter (labeled SO-NLM) is applied between two intermediate nodes. The deblocking filter generates a first intermediate reconstruction sample at the first intermediate node (3511). The first intermediate reconstruction sample is the input to both the CDEF and the nonlinear mapping-based filter. Based on the first intermediate reconstruction sample, the CDEF is applied to generate a second intermediate reconstruction sample at the second intermediate node (3512). Also, based on the first intermediate reconstruction sample, the nonlinear mapping-based filter generates a sample offset (SO). The second intermediate reconstructed sample is combined with the sample offset to generate a third intermediate reconstructed sample at the third intermediate node (3513), and the third intermediate reconstructed sample is clipped to generate CLIP0. CLIP0 is then provided to the LR filter for further filtering.

[0219] Figure 36 shows a flowchart outlining process (3600) according to one embodiment of the present disclosure. Process (3600) can be used for video filtering. Where the term "block" is used, a block may be interpreted as a prediction block, coding unit, luma block, chroma block, etc. In various embodiments, process (3600) is executed by processing circuits such as processing circuits for terminal devices (310), (320), (330), and (340), processing circuits that perform the functions of a video encoder (403), processing circuits that perform the functions of a video decoder (410), processing circuits that perform the functions of a video decoder (510), and processing circuits that perform the functions of a video encoder (603). In some embodiments, process (3600) is implemented in software instructions, so that when a processing circuit executes the software instructions, the processing circuit executes process (3600). The process begins at (S3601) and proceeds to (S3610).

[0220] In (S3610), the first boundary pixel values ​​of the first reconstruction sample at the first node along the loop filter chain are buffered. The first node relates to a nonlinear mapping-based filter applied in the loop filter chain before the loop reconstruction filter.

[0221] In some examples, the nonlinear mapping-based filter is a cross-component sample offset (CCSO) filter. In other examples, the nonlinear mapping-based filter is a local sample offset (LSO) filter.

[0222] In (S3620), the loop reconstruction filter is applied to the reconstructed sample to be filtered based on the buffered first boundary pixel values.

[0223] In some examples, the first reconstructed sample at the first node is the input to a nonlinear mapping-based filter.

[0224] In the examples in Figures 30, 31, and 33, the first node can be the first intermediate node (3011) / (3111) / (3311) in the respective descriptions of Figures 30, 31, and 33, and the first reconstructed sample can be the first intermediate reconstructed sample in the respective descriptions of Figures 30, 31, and 33.

[0225] Next, in the examples in Figures 30 and 33, the second boundary pixel value of the second reconstructed sample at the second node along the loop filter chain can be buffered, and the second reconstructed sample at the second node is generated after the application of the sample offset generated by the nonlinear mapping-based filter. The loop reconstruction filter is applied to the reconstructed sample to be filtered based on the buffered first boundary pixel value and the buffered second boundary pixel value.

[0226] In the example in Figure 30, the sample offset generated by the nonlinear mapping-based filter is combined with the output of the constraint direction enhancement filter (e.g., the second intermediate reconstruction sample in the explanation of Figure 30) to generate a second reconstruction sample (e.g., the third intermediate reconstruction sample in the explanation of Figure 30).

[0227] In the example in Figure 33, the sample offset generated by the nonlinear mapping-based filter is combined with the first reconstructed sample (e.g., the first intermediate reconstructed sample in the description of Figure 33) to generate an intermediate reconstructed sample (e.g., the second intermediate reconstructed sample in the description of Figure 33), and then the constraint direction reinforcement filter is applied to the intermediate reconstructed sample to generate a second reconstructed sample (e.g., the third intermediate reconstructed sample in the description of Figure 33).

[0228] In the example in Figure 31, the second boundary pixel value of the second reconstructed sample at the second node along the loop filter (for example, the second intermediate reconstructed sample at the second intermediate node (3122) in the description of Figure 33) is buffered. The second reconstructed sample at the second node is combined with the sample offset generated by the nonlinear mapping-based filter to generate the reconstructed sample to be filtered. The loop reconstruction filter is applied to the reconstructed sample to be filtered based on the buffered first boundary pixel value and the buffered second boundary pixel value.

[0229] In the example in Figure 32, the first reconstructed sample may be the third intermediate reconstructed sample at the third intermediate node (3213) in the description of Figure 32, which is the result of applying a nonlinear mapping-based filter. The second boundary pixel value of the second reconstructed sample (e.g., the second intermediate reconstructed sample in the description of Figure 32), generated by the deblocking filter, is then buffered. A constraint direction reinforcement filter is applied to the second reconstructed sample to generate an intermediate reconstructed sample (e.g., the second intermediate reconstructed sample in the description of Figure 32). The intermediate reconstructed sample is combined with the sample offset generated by the nonlinear mapping-based filter to generate the first reconstructed sample (e.g., the third intermediate reconstructed sample in the description of Figure 32). The loop reconstruction filter can then be applied based on the buffered first boundary pixel value and the buffered second boundary pixel value.

[0230] In some examples, the reconstructed samples to be filtered are clipped to an appropriate range such as [0, 255] for 8 bits before the application of the loop restoration filter, as in the examples of FIGS. 34 and 35. In some examples (e.g., FIG. 34), the intermediate reconstructed samples (e.g., the second intermediate reconstructed sample in the description of FIG. 34) are clipped to an appropriate range such as [0, 255] for 8 bits before combination with the sample offset generated by a non-linear mapping-based filter.

[0231] The process (3600) proceeds to (S3699) and ends.

[0232] In some examples, the non-linear mapping-based filter is a cross-component sample offset (CCSO) filter. In some examples, the non-linear mapping-based filter is a local sample offset (LSO) filter.

[0233] The process (3600) can be appropriately adapted. The steps in the process (3600) can be modified and / or omitted. Additional steps can be added. Any appropriate order of implementation can be used.

[0234] Embodiments of the present disclosure can be used separately or in combination in any order. Further, each method (or embodiment), encoder, and decoder may be implemented by a processing circuit (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-transitory computer-readable medium.

[0235] The techniques described above can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 37 shows a computer system (3700) suitable for carrying out certain embodiments of the disclosed subject matter.

[0236] Computer software can be coded using any suitable machine code or computer language, and can be subjected to assembly, compilation, linking, or similar mechanisms to create code containing instructions that can be executed directly or through interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0237] Instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.

[0238] The components shown in Figure 37 for the computer system (3700) are illustrative and not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. The configuration of the components should not be construed as having any dependence or requirement on any one or combination of components shown in the exemplary embodiments of the computer system (3700).

[0239] The computer system (3700) may include certain types of human interface input devices. Such human interface input devices may respond to input from one or more human users through, for example, haptic input (e.g., keystrokes, swipes, data glove movements), voice input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as sound (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), or video (e.g., 2D video, 3D video including stereoscopic video).

[0240] The input human interface device may include one or more of the following (only one of each is shown): keyboard (3701), mouse (3702), trackpad (3703), touchscreen (3710), data glove (not shown), joystick (3705), microphone (3706), scanner (3707), and camera (3708).

[0241] The computer system (3700) may also include certain types of human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (3710), data glove (not shown), or joystick (3705); however, there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speakers (3709), headphones (not shown)), visual output devices (e.g., screens (3710) including CRT screens, LCD screens, plasma screens, and OLED screens; each may or may not have touchscreen input functionality, each may or may not have tactile feedback functionality, and some of them may output two-dimensional visual output or higher-than-three-dimensional output through means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0242] The computer system (3700) may also include human-accessible storage devices and associated media, such as optical media including CD / DVD ROM / RW (3720) along with CD / DVD or similar media (3721), thumb drives (3722), removable hard drives or solid-state drives (3723), legacy magnetic media such as tapes and floppy disks (not shown), and specialized ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0243] Those skilled in the art should also understand that the term “computer-readable medium” as used in relation to the subject matter currently disclosed does not include a transmission medium, carrier wave, or other transient signal.

[0244] The computer system (3700) may also include an interface (3754) to one or more communication networks (3755). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, metropolitan, automotive, and industrial, real-time, latency-tolerant, etc. Examples of networks include cellular networks such as Ethernet®, Wi-Fi, GSM, 3G, 4G, 5G, LTE, etc., wide-area digital networks for wired or wireless TV including cable television, satellite television, and terrestrial television, and automotive and industrial networks including CANBus. Certain networks typically require an external network interface adapter attached to some general-purpose data port or peripheral bus (3749) (e.g., a USB port on the computer system (3700)). Others are typically integrated into the core of the computer system (3700) by attachment to a system bus, as described later (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (3700) can communicate with other entities. Such communication may be one-way, receive-only (e.g., broadcast television), one-way transmit-only (e.g., CANbus to certain CANbus devices), or bidirectional to other computer systems using local or wide-area digital networks, for example. Certain protocols and protocol stacks may be used on each of these networks and network interfaces as described above.

[0245] The aforementioned human interface device, human-accessible memory device, and network interface can be attached to the core (3740) of the computer system (3700).

[0246] The core (3740) may include one or more central processing units (CPUs) (3741), graphics processing units (GPUs) (3742), specialized programmable processing units in the form of field-programmable gate arrays (FPGAs) (3743), hardware accelerators for certain tasks (3744), graphics adapters (3750), etc. These devices may be connected via a system bus (3748) along with read-only memory (ROM) (3745), random access memory (3746), internal mass storage devices such as internal non-user-accessible hard drives and solid-state drives (SSDs) (3747). In some computer systems, the system bus (3748) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (3748) or via a peripheral bus (3749). For example, a display (3710) may be connected to a graphics adapter (3750). The architecture for peripheral buses includes PCI, USB, etc.

[0247] The CPU (3741), GPU (3742), FPGA (3743), and accelerator (3744) can execute certain instructions that can be combined to form the aforementioned computer code. This computer code can be stored in ROM (3745) or RAM (3746). Temporary data can also be stored in RAM (3746), while persistent data can be stored, for example, in an internal mass storage device (3747). High-speed storage and retrieval to any of the memory devices can be enabled by using cache memory that can be closely associated with one or more CPUs (3741), GPUs (3742), mass storage devices (3747), ROMs (3745), RAM (3746), etc.

[0248] A computer-readable medium may have computer code on it for performing various computer-implemented operations. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software field.

[0249] As an example, and not an limitation, a computer system having an architecture (3700), specifically a core (3740), can provide functionality as a result of a processor (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media related to certain types of storage of the core (3740) of a non-temporary nature, such as a mass storage device (3747) or ROM (3745) within the core. Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core (3740). The computer-readable media can include one or more memory devices or chips, depending on the specific needs. The software can cause the core (3740) and specifically the processor (including CPUs, GPUs, FPGAs, etc.) within it to execute certain processes or specific parts described herein, including defining data structures stored in RAM (3746) and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic wired within a circuit (e.g., an accelerator (3744)) or otherwise embodied, which may operate in place of, or in conjunction with, software for performing a particular process or a particular part of a particular process as described herein. References to software include logic, and vice versa as appropriate. References to computer-readable media may, as appropriate, include circuits for storing software for execution (e.g., integrated circuits (ICs)), circuits for embodying logic for execution, or both. This disclosure encompasses any preferred combination of hardware and software.

[0250] Appendix A: Acronyms JEM: Joint Exploration Model VVC: versatile video coding BMS: Benchmark set MV: Motion Vector HEVC: High Efficiency Video Coding MPM: Most Probable Mode WAIP: Wide-Angle Intra Prediction SEI: Supplementary Enhancement Information VUI: Video Usability Information GOP: Group of Pictures TU: Transform Unit PU: Prediction Unit CTU: Coding Tree Unit CTB: Coding Tree Block PB: Prediction Block HRD: Hypothetical Reference Decoder SDR: Standard Dynamic Range SNR: Signal-to-Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Areas SSD: Solid-state drive Above: Integrated Circuit CU: Coding Unit PDPC: Position Dependent Prediction Combination ISP: Intra Sub-Partition SPS: Sequence Parameter Setting HDR: High Dynamic Range SDR: Standard Dynamic Range JVET: Joint Video Exploration Team MPM: Most Probable Mode WAIP: Wide-Angle Intra Prediction CU: Coding Unit PU: Prediction Unit TU: Transform Unit CTU: Coding Tree Unit PDPC: Position Dependent Prediction Combination ISP: Intra Sub-Partition SPS: Sequence Parameter Setting PPS: Picture Parameter Set APS: Adaptation Parameter Set VPS: Video Parameter Set DPS: Decoding Parameter Set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC-ALF: Cross-Component Adaptive Loop Filter CDEF: Constrained Directional Enhancement Filter CCSO: Cross-Component Sample Offset CCSAO: Cross-Component Sample Adaptive Offset LSO: Local Sample Offset LR: Loop Restoration Filter FSR: Frame Super-Resolution AV1: AOMedia Video 1 (AOMedia Video 1) AV2: AOMedia Video 2

[0251] While this disclosure has described several exemplary embodiments, there are many modifications, substitutions, and alternative equivalents that fall within the scope of this disclosure. Therefore, those skilled in the art will understand that many systems and methods can be devised that embody the principles of this disclosure and thus fall within the spirit and scope of this disclosure, even if they are not expressly shown or described herein.

Claims

[Claim 1] A method for filtering in video coding, A processor buffers a first boundary pixel value of a first reconstructed sample at a first node along a loop filter chain, wherein the first node is associated with a nonlinear mapping-based filter applied in the loop filter chain before the loop reconstruction filter, and the first boundary pixel value is the value of a pixel at the frame boundary. The processor performs the steps of applying the loop reconstruction filter to the reconstructed sample to be filtered based on the buffered first boundary pixel values. A method that includes this.

Images