Method and apparatus for video coding

Abstract

To provide a method and an apparatus for video coding.SOLUTION: An aspect of the present disclosure includes a method, an apparatus, and a non-transitory computer-readable storage medium for video encoding / decoding. The apparatus includes a processing circuit that decodes prediction information of a current block which indicates a plurality of combinations of processed reconstructed samples and corresponding output values of filter processing. The processing circuit executes the filter processing on processed reconstructed samples of a first color component of the current block. The processed reconstructed samples of the first color are not included in the plurality of combinations of the processed reconstructed samples. The processing circuit determines an output value of the filter processing based on one of a default value, a predicted output value, and one of the output values indicated in the prediction information. The processing circuit reconstructs the current block based on the output value being applied to a second color component of the current block that is different from the first color component.SELECTED DRAWING: Figure 20

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of priority of U.S. Provisional Application No. 63 / 113,120, filed on November 12, 2020, entitled "IMPROVED CROSS - COMPONENT SAMPLE OFFSET", and claims the benefit of priority of U.S. Patent Application No. 17 / 360,840, filed on June 28, 2021, entitled "METHOD AND APPARATUS FOR VIDEO CODING". The entire disclosure of the prior applications is hereby incorporated by reference in its entirety.

[0002] This disclosure generally describes embodiments related to video coding.

Background Art

[0003] The description of the background art provided herein is for the purpose of generally presenting the context of the present disclosure. The inventors' research, insofar as this research is described in this background art section, and aspects of the description that cannot be regarded as prior art at the time of filing are not admitted as prior art to the present disclosure, either explicitly or implicitly.

[0004] Video coding and decoding can be performed using inter - picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, and each picture can have spatial dimensions of, for example, 1920×1080 luminance samples and associated chrominance samples. A series of pictures can have a fixed or variable picture rate, for example, 60 pictures per second or 60 Hz (informally also known as the frame rate). Uncompressed video has significant bit - rate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at a frame rate of 60 Hz) requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space.

[0005] One purpose of video coding and decoding can be the reduction of redundancy in the input video signal by compression. Compression can help reduce the aforementioned bandwidth or storage space requirements, in some cases by more than two orders of magnitude. Both reversible compression and irreversible compression, as well as combinations thereof, can be used. Reversible compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal. When using irreversible compression, the reconstructed signal cannot be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough to utilize the reconstructed signal for its intended purpose. In the case of video, irreversible compression is widely used. The amount of acceptable distortion depends on the application. For example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio may reflect that higher acceptable / tolerable distortion can result in a higher compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transformation, quantization, and entropy coding.

[0007] Video codec technology may include a technology known as intra coding. In intra coding, sample values are represented without referring to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. If all blocks of samples are coded in an intra mode, that picture can be an intra picture. Intra pictures, and their derivatives such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as the first picture in a coded video bitstream and video session or as a still image. Samples of an intra block can undergo a transform, and the transform coefficients can be quantized before entropy coding. Intra prediction can be a technique that minimizes sample values in a pre-transform region. In some cases, the smaller the post-transform DC value and the smaller the AC coefficients, the fewer bits are required to represent the block after entropy coding for a given quantization step size.

[0008] Conventional intra coding, such as known from MPEG-2 production coding technology, does not use intra prediction. However, some newer video compression technologies include techniques that attempt, for example, from the encoding and / or decoding of spatially adjacent and previously decoded-in-order data blocks and / or surrounding sample data and / or metadata obtained during decoding. Such techniques are hereinafter referred to as "intra prediction" techniques. It should be noted that in at least some cases, intra prediction uses only reference data from the current picture being reconstructed and does not use reference data from reference pictures.

[0009] Intra prediction can have many different forms. If more than one of such techniques can be used in a given video coding technique, the technique in use can be coded in an intra prediction mode. In some cases, the mode can have sub - modes and / or parameters, which can be coded individually or can be included in the mode codeword. Which codeword to use for a given combination of mode, sub - mode, and / or parameter can affect the coding efficiency / gain by intra prediction and thus can also affect the entropy coding technique used to convert the codeword into the bitstream.

[0010] A particular mode of intra prediction was introduced in H.264, improved in H.265, and further improved in more recent coding techniques such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). The predictor block can be formed using adjacent sample values belonging to already available samples. The sample values of the adjacent samples are copied into the predictor block according to a direction. The reference to the direction in use can be coded within the bitstream or can itself be predicted.

[0011] Referring to Figure 1A, shown at the lower right is a subset of 9 predictor directions known from 33 possible predictor directions of H.265 (corresponding to the angular modes of 33 out of 35 intra - modes). The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from samples that are at a 45 - degree angle from the horizontal and to the upper right. Similarly, arrow (103) indicates that sample (101) is predicted from samples that are at a 22.5 - degree angle from the horizontal and to the lower left of sample (101).

[0012] Referring further to FIG. 1A, in the upper left, a square block (104) of 4×4 samples (shown by the thick dashed line) is shown. The square block (104) contains 16 samples, and these 16 samples are each labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample from the top in the Y dimension and the first sample from the left in the X dimension. Similarly, sample S44 is the fourth sample within block (104) in both the Y and X dimensions. Since the block size is 4×4 samples, S44 is in the lower right. Further reference samples are shown following a similar numbering scheme. The reference samples are labeled with R for the block (104), its Y position (e.g., row index), and its X position (column index). In both H.264 and H.265, the predicted samples are adjacent to the block being reconstructed, and thus negative values need not be used.

[0013] Intra-picture prediction may function by copying reference sample values from adjacent samples as assigned by the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating a prediction direction that matches arrow (102) for this block, i.e., that the sample or samples are predicted from a predicted sample or samples at a 45-degree angle from the horizontal and up to the right. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Next, sample S44 is predicted from reference sample R08.

[0014] In certain cases, particularly when the direction does not divide evenly at 45 degrees, the values of multiple reference samples may be combined, for example by interpolation, to calculate a reference sample.

[0015] The number of possible directions has been increasing as video coding technology develops. In H.264 (2003), nine different directions can be represented. This has increased to 33 in H.265 (2013), and at the time of the present disclosure, JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and certain techniques of entropy coding are used to represent those likely directions with fewer bits, accepting a certain penalty for less likely directions. Further, the direction itself may be predicted from adjacent directions used in adjacent already decoded blocks.

[0016] FIG. 1B shows a schematic diagram (105) showing 65 intra prediction directions by JEM to show the number of prediction directions increasing over time.

[0017] The mapping of intra prediction direction bits within the coded video bitstream representing the direction can vary for each video coding technology and can include, for example, a simple direct mapping from the prediction direction to the intra prediction mode, a codeword, a complex adaptive scheme including the most likely mode, and the like. However, in all cases, there may be certain directions in the video content that are statistically less likely than certain other directions. Since the purpose of video compression is to reduce redundancy, those less likely directions are represented with more bits than the more likely directions in a well-functioning video coding technology.

[0018] Motion compensation can be an irreversible compression technique and can relate to a technique in which a block of sample data from a previously reconstructed picture or a portion thereof (reference picture) is spatially shifted in a direction indicated by a motion vector (hereinafter MV (motion vector)) and then used for prediction of a newly reconstructed picture or picture portion. In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV can have two dimensions X and Y, or three dimensions, with the third dimension being an indication of the reference picture in use (the latter can be indirectly the temporal dimension).

[0019] In some video compression techniques, the MV applicable to a particular region of sample data can be predicted from other MVs, for example, from an MV associated with another region of sample data that is spatially adjacent to the region being reconstructed and that precedes that MV in decoding order. By doing so, the amount of data required to code the MV can be significantly reduced, and as a result, redundancy is eliminated and compression is increased. For example, when coding an input video signal pulled from a camera (known as natural video), regions larger than the region to which a single MV is applicable move in a similar direction and thus, in some cases, there can be a statistical likelihood that they can be predicted using a similar MV derived from the MVs of adjacent regions. Thereby, the MV found for a given region is similar to or the same as the MV predicted from surrounding MVs, and as a result, after entropy coding, it can be represented with fewer bits than when directly coding the MV. In some cases, MV prediction can be an example of reversible compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself can be irreversible, for example, due to rounding errors when calculating a predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms provided by H.265, a technique hereinafter referred to as "spatial merge" is described in this specification.

[0021] Referring to FIG. 1C, the current block (111) may include samples found by the encoder during the motion search process that are predictable from a previous block of the same size that has been spatially shifted. Instead of directly coding the MV, the MV can be derived from metadata associated with one or more reference pictures using an MV associated with any one of five surrounding samples represented by A0, A1 and B0, B1, B2 (112 to 116 respectively), for example from the latest reference picture (in decoding order). In H.265, MV prediction can use predictors from the same reference picture that adjacent blocks are using. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0022] Aspects of the present disclosure provide an apparatus for video encoding / decoding. The apparatus includes a processing circuit that decodes prediction information for a current block within a current picture that is part of a coded video sequence. The prediction information indicates a plurality of combinations of processed reconstructed samples and corresponding output values of a filtering process. The processing circuit performs a filtering process on the processed reconstructed samples of a first color component of the current block. The processed reconstructed samples of the first color are not included in the plurality of combinations of processed reconstructed samples. The processing circuit determines an output value of the filtering process based on one of a default value, a predicted output value, and one of the output values indicated in the prediction information. The processing circuit reconstructs the current block based on an output value applied to the reconstructed samples of a second color component of the current block. The second color component is different from the first color component.

[0023] In one embodiment, the processing circuit determines one of a first value and a second value as an output value of the filtering process. The first value is one of the output values indicated in the prediction information, and the second value is the inverse of the first value.

[0024] In one embodiment, each of the processed reconstruction samples of the first color component is the inverse of the corresponding processed reconstruction sample in one of a plurality of combinations of processed reconstruction samples indicated in the prediction information, and one of the output values indicated in the prediction information corresponds to one of the plurality of combinations of processed reconstruction samples indicated in the prediction information.

[0025] In one embodiment, the sum of the index of the processed reconstruction sample of the first color component in the mapping table of the filtering process and the index of one of the output values indicated in the prediction information in the mapping table of the filtering process is equal to the maximum index of the mapping table.

[0026] In one embodiment, the processing circuit determines a default value as the output value of the filtering process based on prediction information indicating that a plurality of combinations of processed reconstruction samples indicated in the prediction information are a subset of the total number of combinations of processed reconstruction samples in the filtering process.

[0027] In one embodiment, the predicted output value is determined based on the output value of the filtering process applied to another coding block.

[0028] In one embodiment, the processed reconstruction sample of the first color component is determined based on the reconstruction sample of the first color component output from the first coding module, and the reconstruction sample of the second color component is output from the first coding module or the second coding module applied after the first coding module.

[0029] Aspects of the present disclosure provide a method for video encoding / decoding. In the method, prediction information of a current block in a current picture that is part of a coded video sequence is decoded. The prediction information indicates a plurality of combinations of processed reconstructed samples and corresponding output values of filtering. The filtering is performed on the processed reconstructed samples of the first color component of the current block. The processed reconstructed samples of the first color are not included in the plurality of combinations of processed reconstructed samples. The output value of the filtering is determined based on one of a default value, a predicted output value, and one of the output values indicated in the prediction information. The current block is reconstructed based on the output value applied to the reconstructed samples of the second color component of the current block. The second color component is different from the first color component.

[0030] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform any one or combination of the methods for video decoding.

[0031] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

Brief Description of the Drawings

[0032]

Figure 1A

Figure 1B

Figure 1C

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8A

Figure 8B

Figure 9A

Figure 9B

Figure 9C

Figure 9D

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14A

Figure 14A

Figure 14B

Figure 14B

Figure 15

Figure 15

Figure 16

Figure 16

Figure 17

Figure 17

Figure 18

Figure 18

Figure 19A

Figure 19A

Figure 19B

Figure 19B

Figure 19C

Figure 19C

Figure 19D

Figure 19D

Figure 19E

Figure 19E

Figure 19F

Figure 19F

Figure 19G

Figure 19H

Figure 20

Figure 21

[0033] I. VIDEO DECODER AND ENCODER SYSTEMS FIG. 2 shows a simplified block diagram of a communication system (200) according to one embodiment of the present disclosure. The communication system (200) includes a plurality of terminal devices that can communicate with each other, for example, via a network (250). For example, the communication system (200) includes a first pair of terminal devices (210) and (220) interconnected via the network (250). In the example of FIG. 2, the first pair of terminal devices (210) and (220) perform unidirectional data transmission. For example, the terminal device (210) can code video data (e.g., a stream of video pictures captured by the terminal device (210)) for transmission to another terminal device (220) via the network (250). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (220) can receive the encoded video data from the network (250), decode the encoded video data to restore the video pictures, and display the video pictures according to the restored video data. Unidirectional data transmission can be common in media serving applications and the like.

[0034] In another example, the communication system (200) includes a second pair of terminal devices (230) and (240) that perform bidirectional transmission of coded video data that may occur, for example, during a video conference. For the bidirectional transmission of data, in one example, each of the terminal devices (230) and (240) may code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (230) and (240) via the network (250). Each of the terminal devices (230) and (240) may also receive the coded video data transmitted by the other of the terminal devices (230) and (240), may decode the coded video data to restore the video picture, and may display the video picture on a display device accessible according to the restored video data.

[0035] In the example of FIG. 2, the terminal devices (210), (220), (230), and (240) may be shown as servers, personal computers, and smartphones, but the principles of the present disclosure may not be so limited. Embodiments of the present disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (250) represents any number of networks that transmit coded video data between the terminal devices (210), (220), (230), and (240), including, for example, wired (wired) and / or wireless communication networks. The communication network (250) may exchange data on circuit-switched channels and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of the network (250) may not be important for the operation of the present disclosure, unless otherwise described herein below.

[0036] FIG. 3 shows the placement of a video encoder and a video decoder in a streaming environment as an example of the disclosed subject matter's use. The disclosed subject matter may be equally applicable to other video-related applications, including, for example, the storage of compressed video on digital media such as video conferencing, digital TV, CDs, DVDs, and memory sticks.

[0037] A streaming system may include a video source (301) that creates a stream (302) of, for example, uncompressed video pictures, and a capture subsystem (313) that may include, for example, a digital camera. In one example, the stream (302) of video pictures includes samples taken by a digital camera. The stream (302) of video pictures, shown as a thick line to emphasize the high data volume compared to the encoded video data (304) (or encoded video bitstream), may be processed by an electronic device (320) that includes a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (304) (or encoded video bitstream (304)) is shown as a thin line to emphasize the lower data volume compared to the stream (302) of video pictures and may be stored in a streaming server (305) for future use. One or more streaming client subsystems, such as client subsystems (306) and (308) of FIG. 3, may access the streaming server (305) to retrieve copies (307) and (309) of the encoded video data (304). The client subsystem (306) may include, for example, a video decoder (310) within an electronic device (330). The video decoder (310) decodes an input copy (307) of the encoded video data and creates an output stream (311) of video pictures that may be rendered on a display (312) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (304), (307), and (309) (e.g., video bitstreams) may be encoded according to a particular video coding / compression standard. Examples of those standards include ITU-T Recommendation H.265.In one example, the video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0038] Note that electronic devices (320) and (330) may include other components (not shown). For example, electronic device (320) may include a video decoder (not shown), and electronic device (330) may also include a video encoder (not shown).

[0039] FIG. 4 shows a block diagram of a video decoder (410) according to an embodiment of the present disclosure. The video decoder (410) may be included in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., a receiving circuit). The video decoder (410) may be used in place of the video decoder (310) of the example of FIG. 3.

[0040] The receiver (431) may receive one or more coded video sequences to be decoded by the video decoder (410), and in the same or another embodiment, may receive one coded video sequence at a time, in which case the decoding of each coded video sequence is independent of other coded video sequences. The coded video sequence may be received from a channel (401), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (431) may receive the encoded video data together with other data, such as a coded audio data stream and / or an auxiliary data stream, which may be transferred to respective using entities (not shown). The receiver (431) may separate the coded video sequence from other data. To counter network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420) (hereinafter “parser (420)”). In certain applications, the buffer memory (415) is part of the video decoder (410). In others, it may be external to the video decoder (410) (not shown). In yet others, for example, there may be a buffer memory (not shown) external to the video decoder (410) to counter network jitter, and in addition, there may be another buffer memory (415) internal to the video decoder (410) to process, for example, playout timing. If the receiver (431) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isochronous network, the buffer memory (415) may not be necessary or may be made small. For use in a best-effort packet network such as the Internet, the buffer memory (415) may be required, may be relatively large, may preferably be adaptively sized, and may be at least partially implemented in an operating system or similar element (not shown) external to the video decoder (410).

[0041] Video decoder (410) may include a parser (420) for reconstructing symbols (421) from the coded video sequence. The categories of these symbols include information used to manage the operation of the video decoder (410), and in some cases, although not an essential part of the electronic device (430), information for controlling rendering devices (e.g., display screens) such as a rendering device (412) that may be coupled to the electronic device (430), as shown in FIG. 4. The control information for the rendering device may be in the form of supplemental enhancement information (SEI (Supplemental Enhancement Information) message) or a video usability information (VUI) parameter set fragment (not shown). The parser (420) may parse / entropy-decode the received coded video sequence. The coding of the coded video sequence may conform to a video coding technology or standard and may follow various principles including variable-length coding, Huffman coding, and arithmetic coding with or without context dependence. The parser (420) may extract a set of subgroup parameters regarding at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to that group. The subgroups may include group of pictures (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), and prediction unit (PU), etc. The parser (420) may also extract from the coded video sequence information such as transform coefficients, quantizer parameter values, and MVs.

[0042] The parser (420) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (415) to create symbols (421).

[0043] The reconstruction of symbol (421) may involve multiple different units depending on the type of the coded video picture or a portion thereof (such as inter-picture and intra-picture, inter-block and intra-block), as well as other factors. How to involve which unit can be controlled by subgroup control information parsed from the video sequence coded by parser (420). The flow of such subgroup control information between parser (420) and the following multiple units is not shown for clarity.

[0044] In addition to the function blocks already mentioned, video decoder (410) may be conceptually subdivided into several functional units as described below. In an actual implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purpose of explaining the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.

[0045] The first unit is a scaler / inverse transform unit (451). The scaler / inverse transform unit (451) receives, as symbol (421), from parser (420) the quantized transform coefficients and control information including the transform to be used, block size, quantization coefficients, quantization scaling matrix, etc. The scaler / inverse transform unit (451) may output a block including sample values that can be input to aggregator (455).

[0046] In some cases, the output samples of the scaler / inverse transform (451) may be related to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but may use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by the intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) uses the surrounding already reconstructed information fetched from the current picture buffer (458) to generate a block of the same size and shape as the block being reconstructed. The current picture buffer (458) buffers, for example, the partially reconstructed current picture and / or the fully reconstructed current picture. The aggregator (455) may, in some cases, add, for each sample, the prediction information generated by the intra prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451).

[0047] In other cases, the output samples of the scaler / inverse transform unit (451) may be related to inter-coded blocks, which may be motion-compensated in some cases. In such cases, the motion-compensation prediction unit (453) may access the reference picture memory (457) to fetch the samples used for prediction. According to the symbol (421) related to the block, after motion-compensating the fetched samples, these samples (in this case, called residual samples or residual signal) may be added by the aggregator (455) to the output of the scaler / inverse transform unit (451) to generate the output sample information. The address in the reference picture memory (457) where the motion-compensation prediction unit (453) fetches the prediction samples may be controlled by the MV available to the motion-compensation prediction unit (453) in the form of a symbol (421) that may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of the sample values fetched from the reference picture memory (457) when an exact MV of sub-samples is used, and an MV prediction mechanism, etc.

[0048] The output samples of the aggregator (455) may be subject to various loop filtering techniques in the loop filter unit (456). Video compression techniques may be included in the coded video sequence (also referred to as the coded video bitstream) and may include in-loop filter techniques controlled by parameters made available to the loop filter unit (456) as symbols (421) from the parser (420), which may also respond to meta information obtained during the decoding of a previous part of the coded picture or coded video sequence (in decoding order) and also to previously reconstructed and loop-filtered sample values.

[0049] The output of the loop filter unit (456) can be not only output to the render device (412), but also a sample stream that can be stored in the reference picture memory (457) for use in future inter-picture prediction.

[0050] When a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, when the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (420)), the current picture buffer (458) can become part of the reference picture memory (457), and a new current picture buffer can be reallocated before starting the reconstruction of subsequent coded pictures.

[0051] The video decoder (410) may perform a decoding operation according to a predetermined video compression technique in a standard such as ITU-T Rec.H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to both the syntax and profile of the video compression technique or standard as documented in the video compression technique or standard. Specifically, the profile may select a particular tool as the only tool available for use under that profile from among all the tools available in the video compression technique or standard. Also, for compliance, it may be necessary that the complexity of the coded video sequence be within the range defined by the level of the video compression technique or standard. In some cases, the level may limit the maximum picture size, maximum frame rate, maximum reconstructed sample rate (measured, for example, in megasamples per second), and maximum reference picture size. The limits set by the level may, in some cases, be further restricted by the specifications of a Hypothetical Reference Decoder (HRD) and the metadata of HRD buffer management signaled in the coded video sequence.

[0052] In one embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (410) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, and forward error correction codes.

[0053] FIG. 5 shows a block diagram of a video encoder (503) according to an embodiment of the present disclosure. The video encoder (503) is included in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmission circuit). The video encoder (503) may be used in place of the video encoder (303) of the example of FIG. 3.

[0054] The video encoder (503) may receive video samples from a video source (501) (not part of the electronic device (520) in the example of FIG. 5) that can capture a video image to be coded by the video encoder (503). In another example, the video source (501) is part of the electronic device (520).

[0055] The video source (501) may provide a source video sequence to be coded by the video encoder (503) in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit,...), any color space (e.g., BT.601 Y CrCB, RGB,...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (501) can be a storage device that stores previously prepared videos. In a video conferencing system, the video source (501) can be a camera that captures local image information as a video sequence. The video data can be provided as a plurality of individual pictures that convey motion when viewed in sequence. The pictures themselves can be organized as a spatial array of pixels, and each pixel can include one or more samples depending on the sampling structure, color space, etc. in use. A person skilled in the art can easily understand the relationship between pixels and samples. In the following description, samples will be mainly described.

[0056] According to one embodiment, the video encoder (503) can code pictures of a source video sequence in real time or under any other temporal constraints according to the requirements of the application and compress them into a coded video sequence (543). Implementing an appropriate coding speed is one function of the controller (550). In some embodiments, the controller (550) controls other functional units and is functionally coupled to other functional units as described below. For clarity, the couplings are not shown. Parameters set by the controller (550) may include rate control related parameters (picture skip, quantizer, lambda value of rate distortion optimization techniques, …), picture size, group of pictures (GOP) layout, and maximum MV allowable reference region, etc. The controller (550) may be configured to have other appropriate functions related to the video encoder (503) optimized for a specific system design.

[0057] In some embodiments, the video encoder (503) is configured to operate in a coding loop. As an overly simplified explanation, in one example, the coding loop may include a source coder (530) (e.g., responsible for creating symbols such as a symbol stream based on an input picture to be coded and reference pictures) and a (local) decoder (533) incorporated in the video encoder (503). The decoder (533) reconstructs symbols to create sample data in the same way as a (remote) decoder does (when the compression between symbols and the coded video bitstream is reversible in the video compression technology considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input into the reference picture memory (534). Since the decoding of the symbol stream results in bit-exact results regardless of the decoder's location (local or remote), the content of the reference picture memory (534) is also bit-exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" the same sample values as the decoder "sees" as reference picture samples when using prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, e.g., due to channel errors) is also used in some related arts.

[0058] The operation of the "local" decoder (533) may be the same as that of a "remote" decoder such as the video decoder (410) already described in detail above in relation to FIG. 4. However, referring briefly to FIG. 4 as well, since the symbols are available and the encoding / decoding of the symbols by the entropy encoder (545) and the parser (420) to / from the coded video sequence can be reversible, the entropy decoding part of the video decoder (410), including the buffer memory (415) and the parser (420), may not be fully implemented in the local decoder (533).

[0059] An observation that can be made in this regard is that decoder technologies other than parsing / entropy decoding existing in the decoder must also necessarily exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter focuses on decoder operations. The description of encoder technology may be omitted because it is the reverse of the decoder technology described comprehensively. More detailed descriptions are necessary only in certain areas and are provided below.

[0060] During operation, in some examples, the source coder (530) may perform motion-compensated predictive coding that predictively codes an input picture by referring to one or more previously coded pictures from a video sequence designated as "reference pictures". In this way, the coding engine (532) codes the difference between a pixel block of the input picture and a pixel block of a reference picture that can be selected as a prediction reference for the input picture.

[0061] The local video decoder (533) may decode the coded video data of a picture that can be designated as a reference picture based on the symbols created by the source coder (530). The operation of the coding engine (532) may preferably be an irreversible process. When the coded video data can be decoded by a video decoder (not shown in FIG. 5), the reconstructed video sequence can usually be a replica of the source video sequence with some errors. The local video decoder (533) may replicate the decoding process that can be performed by the video decoder for the reference picture and store the reconstructed reference picture in the reference picture cache (534). In this way, the video encoder (503) may locally store a copy of the reconstructed reference picture having the same content as the reconstructed reference picture obtained by the remote video decoder (if there are no transmission errors).

[0062] Predictor (535) can perform predictive search for the coding engine (532). That is, regarding a new picture to be coded, predictor (535) can search the reference picture memory (534) for sample data (as candidate reference pixel blocks) that can function as an appropriate predictive reference for the new picture, or for specific metadata such as reference picture MVs and block shapes. Predictor (535) can operate on a per-sample block / pixel block basis to find an appropriate predictive reference. In some cases, the input picture can have a predictive reference drawn from among a plurality of reference pictures stored in the reference picture memory (534), as determined by the search results obtained by predictor (535).

[0063] Controller (550) can manage the coding operations of source coder (530), including, for example, setting parameters and subgroup parameters used to encode video data.

[0064] The outputs of all the foregoing functional units can undergo entropy coding by entropy coder (545). Entropy coder (545) converts the symbols generated by the various functional units into a coded video sequence by reversibly compressing the symbols according to techniques such as Huffman coding, variable length coding, and arithmetic coding.

[0065] Transmitter (540) can buffer the coded video sequence created by entropy coder (545) to prepare for transmission via communication channel (560), which can be a hardware / software link to a storage device that stores the encoded video data. Transmitter (540) can merge the coded video data from video coder (503) with other data to be transmitted, such as a coded audio data stream and / or an auxiliary data stream (source not shown).

[0066] The controller (550) may manage the operation of the video encoder (503). During coding, the controller (550) may assign a type of a particular coded picture to each coded picture, which may affect the coding techniques applicable to each picture. For example, a picture may often be assigned as one of the following picture types.

[0067] An intra picture (I picture) may be coded and decoded without using other pictures in the sequence as a source of prediction. Some video codecs support various types of intra pictures, including for example independent decoder refresh (IDR) pictures. Those skilled in the art are aware of those variants of I pictures as well as their respective uses and characteristics.

[0068] A predicted picture (P picture) may be coded and decoded using intra prediction or inter prediction that uses at most one motion vector (MV) and a reference index to predict the sample values of each block.

[0069] A bi - directionally predicted picture (B picture) may be coded and decoded using intra prediction or inter prediction that uses at most two MVs and reference indices to predict the sample values of each block. Similarly, multiple predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0070] The source picture can generally be spatially subdivided into a plurality of sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and coded block by block. The blocks can be coded predictively by referring to other (already coded) blocks as determined by the coding assignment applied to each picture of the block. For example, the blocks of an I picture can be coded non-predictively or predictively by referring to already coded blocks of the same picture (spatial prediction or intra prediction). The pixel blocks of a P picture can be coded predictively by referring to one previously coded reference picture either by spatial prediction or by temporal prediction. The blocks of a B picture can be coded predictively by referring to one or two previously coded reference pictures either by spatial prediction or by temporal prediction.

[0071] The video encoder (503) can perform coding operations according to a predetermined video coding technology or standard such as ITU-T Rec.H.265. In that operation, the video encoder (503) can perform various compression operations including predictive coding operations that utilize the temporal and spatial redundancy of the input video sequence. Thus, the coded video data can conform to the syntax specified by the video coding technology or standard being used.

[0072] In one embodiment, the transmitter (540) can transmit additional data along with the encoded video. The source coder (530) can include such data as part of the coded video sequence. The additional data can include, for example, temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, and VUI parameter set fragments.

[0073] Video can be captured as a plurality of source pictures (video pictures) in time series. Intra-picture prediction (often abbreviated as intra prediction) utilizes the spatial correlation in a given picture, while inter-picture prediction utilizes the (temporal or other) correlation between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. If a block within the current picture is similar to a reference block within a reference picture that has been previously encoded and is still buffered within the video, the block within the current picture can be encoded by a vector called an MV. The MV points to the reference block within the reference picture and can have a third dimension to identify the reference picture if multiple reference pictures are being used.

[0074] In some embodiments, dual prediction techniques can be used for inter-picture prediction. According to the dual prediction technique, two reference pictures such as a first reference picture and a second reference picture that are both before the current picture in the video in decoding order (but may be past and future respectively in display order) are used. A block within the current picture can be encoded by a first MV that points to a first reference block within the first reference picture and a second MV that points to a second reference block within the second reference picture. The block can be predicted by a combination of the first reference block and the second reference block.

[0075] Furthermore, to improve coding efficiency, merge mode techniques can be used for inter-picture prediction.

[0076] According to some embodiments of the present disclosure, predictions such as inter-image prediction and intra-image prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and CTUs in a picture have the same size such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs) which are one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree divided into one or more coding units (CUs). For example, a 64×64 pixel CTU can be divided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter-prediction type or an intra-prediction type. The CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values (e.g., luma values) such as 8×8 pixels, 16×16 pixels, 8×16 pixels, and 16×8 pixels.

[0077] FIG. 6 shows a diagram of a video encoder (603) according to another embodiment of the present disclosure. The video encoder (603) receives a processing block (e.g., a prediction block) of sample values in a current video picture in a sequence of video pictures and is configured to encode the processing block into a coded picture that is part of a coded video sequence. In one example, the video encoder (603) is used in place of the video encoder (303) in the example of FIG. 3.

[0078] In the example of HEVC, the video encoder (603) receives a matrix of sample values of a processing block, such as a prediction block of 8×8 samples. The video encoder (603) determines whether the processing block would be best coded using an intra mode, an inter mode, or a bi-prediction mode, for example using rate distortion optimization. If the processing block is to be coded in the intra mode, the video encoder (603) may use intra prediction techniques to encode the processing block into the coded picture, and if the processing block is to be coded in the inter mode or the bi-prediction mode, the video encoder (603) may use inter prediction techniques or bi-prediction techniques, respectively, to encode the processing block into the coded picture. In certain video coding techniques, the merge mode may be an inter-picture prediction sub-mode, and the MV is derived from one or more MV predictors without the benefit of the coded MV components outside the predictor. In certain other video coding techniques, there may be MV components applicable to the target block. In one example, the video encoder (603) includes other components such as a mode decision module (not shown) for determining the mode of the processing block.

[0079] In the example of FIG. 6, the video encoder (603) includes an inter encoder (630), an intra encoder (622), a residual calculator (623), a switch (626), a residual encoder (624), a general-purpose controller (621), and an entropy encoder (625) coupled to each other as shown in FIG. 6.

[0080] The inter-encoder (630) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks (e.g., blocks in a previous picture and a subsequent picture) in a reference picture, generate inter-prediction information (e.g., description of redundant information by inter-encoding techniques, MV, merge mode information), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0081] The intra-encoder (622) is configured to receive samples of a current block (e.g., a processing block), optionally compare the block with already encoded blocks in the same picture, generate quantized coefficients after transformation, and optionally also generate intra-prediction information (e.g., intra-prediction direction information by one or more intra-encoding techniques). In one example, the intra-encoder (622) also calculates an intra-prediction result (e.g., a predicted block) based on the intra-prediction information and reference blocks in the same picture.

[0082] The general-purpose controller (621) is configured to determine general-purpose control data and control other components of the video encoder (603) based on the general-purpose control data. In one example, the general-purpose controller (621) determines the mode of a block and provides a control signal to the switch (626) based on the mode. For example, when the mode is the intra mode, the general-purpose controller (621) controls the switch (626) to select the intra mode result for use by the residual calculator (623), selects the intra prediction information, controls the entropy encoder (625) to include the intra prediction information in the bitstream, and when the mode is the inter mode, the general-purpose controller (621) controls the switch (626) to select the inter prediction result for use by the residual calculator (623), selects the inter prediction information, and controls the entropy encoder (625) to include the inter prediction information in the bitstream.

[0083] The residual calculator (623) calculates the difference (residual data) between the received block and the prediction result selected from the intra encoder (622) or the inter encoder (630). The residual encoder (624) is configured to operate based on the residual data to encode the residual data in order to generate transformation coefficients. In one example, the residual encoder (624) is configured to convert the residual data from the spatial domain to the frequency domain and generate transformation coefficients. The transformation coefficients then undergo quantization processing to obtain the quantized transformation coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is configured to perform inverse transformation and generate the decoded residual data. The decoded residual data can be appropriately used by the intra encoder (622) and the inter encoder (630). For example, the inter encoder (630) can generate the decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (622) can generate the decoded block based on the decoded residual data and the intra prediction information. In some examples, the decoded block is appropriately processed to generate the decoded picture, and the decoded picture can be buffered in a memory circuit (not shown) and used as a reference picture.

[0084] The entropy encoder (625) is configured to format the bitstream to include the encoded block. The entropy encoder (625) is configured to include various information according to an appropriate standard such as HEVC. In one example, the entropy encoder (625) is configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other appropriate information in the bitstream. Note that according to the disclosed subject matter, there is no residual information when coding a block in either the merge submode of the inter mode or the bi-prediction mode.

[0085] Figure 7 shows a diagram of a video decoder (710) according to another embodiment of the present disclosure. The video decoder (710) is configured to receive a coded picture that is part of a coded video sequence and decode the coded picture to generate a reconstructed picture. In one example, the video decoder (710) is used in place of the video decoder (310) of the example of FIG. 3.

[0086] In the example of FIG. 7, the video decoder (710) includes an entropy decoder (771), an inter decoder (780), a residual decoder (773), a reconstruction module (774), and an intra decoder (772) coupled to each other as shown in FIG. 7.

[0087] The entropy decoder (771) may be configured to reconstruct from the coded picture specific symbols representing the syntax elements that the coded picture is composed of. Such symbols may include, for example, the mode in which a block is coded (e.g., intra mode, inter mode, bi-prediction mode, etc., the latter two being merge sub-modes or another sub-mode), prediction information (e.g., intra prediction information or inter prediction information, etc.) that can identify specific samples or metadata used for prediction by the intra decoder (772) or the inter decoder (780) respectively, and residual information in the form of, for example, quantized transform coefficients. In one example, when the prediction mode is an inter prediction mode or a bi-prediction mode, the inter prediction information is provided to the inter decoder (780), and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (772). The residual information may undergo inverse quantization and be provided to the residual decoder (773).

[0088] The inter decoder (780) is configured to receive inter prediction information and generate an inter prediction result based on the inter prediction information.

[0089] The intra decoder (772) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0090] The residual decoder (773) is configured to perform inverse quantization to extract inverse quantized transform coefficients, and process the inverse quantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require certain control information (including quantizer parameters (QP)), and that information may be provided by the entropy decoder (771) (data paths not shown as such may be only for low-volume control information).

[0091] The reconstruction module (774) is configured to combine, in the spatial domain, the residual output by the residual decoder (773) and the prediction result (output by the inter prediction module or the intra prediction module in some cases) to form a reconstructed block that can be part of a reconstructed picture that can be part of the ultimately reconstructed video. Note that other appropriate operations, such as deblocking operations, may be performed to improve visual quality.

[0092] Note that the video encoders (303), (503), and (603), and the video decoders (310), (410), and (710) may be implemented using any suitable technology. In one embodiment, the video encoders (303), (503), and (603), and the video decoders (310), (410), and (710) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (303), (503), and (603), and the video decoders (310), (410), and (710) may be implemented using one or more processors that execute software instructions.

[0093] II. Adaptive Loop Filter In some related examples, such as VVC Draft 8, an Adaptive Loop Filter (ALF) with block-based filter adaptation may be applied. For the luma component, based on the directionality and activity of the local gradient of each 4×4 block, one of 25 filters can be selected for each 4×4 block.

[0094] Figures 8A - 8B show exemplary diamond filter shapes of the ALF according to some embodiments of the present disclosure. Figure 8A shows a 5×5 diamond shape applicable to the chroma component, and Figure 8B shows a 7×7 diamond shape applicable to the luma component.

[0095] In some related examples, for the luma component, each 4×4 block can be classified into one of 25 classes. The classification index C of the 4×4 block can be derived as follows based on the directionality D and activity

Number

Number

[0096] D and

Number

Number

Number

Number

Number

[0097] To reduce the complexity of block classification, subsampled 1-D Laplacian calculations can be applied. FIGS. 9A through 9D show exemplary subsample positions used for gradient calculations in the vertical, horizontal, and two diagonal directions, respectively.

[0098] Next, the maximum and minimum values of the horizontal and vertical gradients can be set as follows.

Equation

Equation

[0099] The maximum and minimum values of the gradients in the two diagonal directions can be set as follows.

Equation

Equation

[0100] To derive the value of the directionality D, these values can be compared to two thresholds t1 and t2 with respect to each other.

[0101] Step 1:

Equation

Equation

[0102] Step 2:

Equation

[0103] Step 3:

Number

[0104] Step 4:

Number

[0105] The motion value A is calculated as follows.

Number

[0106] Furthermore, A is quantized in the range of 0 to 4 (including both ends), and the quantization value is

Number

[0107] For the chroma components within the picture, the classification method is not applied. For example, a single set of ALF coefficients can be applied to each chroma component.

[0108] Before filtering each 4×4 luma block, geometric transformations such as rotation, diagonal flip, and vertical flip are applied to the filter coefficients f(k, l) and the corresponding filter clipping values c(k, l) according to the gradient values calculated for each block. This is equivalent to applying these transformations to the samples within the filter support region in order to make the different blocks to which ALF is applied more similar by aligning their directions.

[0109] Three geometric transformations, including diagonal flip, vertical flip, and rotation, can be described as follows. Diagonal: f D (k, l) = f(l, k), c D (k, l) = c(l, k) Equation (11) Vertical flip: f V (k, l) = f(k, K - l - 1), c V (k, l) = c(k, K - l - 1) Equation (12) Rotation: f R (k, l) = f(K - l - 1, k), c R (k, l) = c(K - l - 1, k) Equation (13) Here, K is the filter size, 0 ≤ k, l ≤ K - 1 are the coordinates of the transformation coefficients, the position (0, 0) is the upper left corner, and the position (K - 1, K - 1) is the lower right corner. The transformation is applied to the filter coefficients f(k, l) and the clipping values c(k, l) according to the gradient values calculated in the corresponding blocks. The relationship between the transformation and the gradients in four directions can be summarized in Table 1.

[0110]

Table 1

[0111] In some related examples such as VVC Draft 8, the filter parameters of ALF are signaled with an Adaptive Parameter Set (APS). In one APS, up to 25 sets of luma filter coefficients and clipping value indices, and up to 8 sets of chroma filter coefficients and clipping value indices can be signaled. To reduce the bit overhead, the filter coefficients of different classifications of the luma component can be merged. In the slice header, the index of the APS used for the current slice is signaled. In VVC Draft 8, the signaling of ALF is CTU-based.

[0112] The clipping value index decoded from the APS enables the determination of the clipping values using a table of clipping values for the luma and chroma components. These clipping values depend on the internal bit depth. For example, the table of clip values can be obtained by the following formula. [Number] Here, B is equal to the internal bit depth, α is a predefined constant value equal to 2.35, and N is equal to 4, which is the number of clipping values allowed in VVC Draft 8. An example of the output of Equation (14) is shown in Table 2.

[0113] [Table 2]

[0114] In the slice header, up to seven APS indices can be signaled to specify the luma filter set used for the current slice. The filtering process can be further controlled at the CTB level. A flag can be signaled to indicate whether ALF is applied to the luma CTB. The luma CTB can select a filter set from among 16 fixed filter sets and the filter set from the APS. To indicate which filter set to apply, the filter set index of the luma CTB is signaled. The 16 fixed filter sets are predefined and can be hard-coded in both the encoder and decoder.

[0115] For the chroma component, the APS index can be signaled in the slice header to indicate the chroma filter set used for the current slice. At the CTB level, if there are two or more chroma filter sets in the APS, the filter set index can be signaled for each chroma CTB.

[0116] The filter coefficients can be quantized with a norm equal to 128. To limit the complexity of multiplication, bitstream compatibility can be applied so that the coefficient values at non-central positions are within the range from -27 to 27-1. The central position coefficient is not signaled in the bitstream and is considered to be equal to 128.

[0117] In some related examples such as VVC draft 8, the syntax and semantics of the clipping index and the corresponding values can be defined as follows.

[0118] alf_luma_clip_idx[sfIdx][j] specifies the clipping index of the clipping value used before multiplying the j-th coefficient of the signaled luma filter indicated by sfIdx. The requirement for bitstream compatibility is that the value of alf_luma_clip_idx[sfIdx][j] when sfIdx = 0..alf_luma_num_filters_signalled_minus1 and j = 0..11 is in the range from 0 to 3 (including both ends).

[0119] The luma filter clipping value AlfClipL[adaptation_parameter_set_id][filtIdx] with the element AlfClipL[adaptation_parameter_set_id][filtIdx][j] where filtIdx = 0..NumAlfFilters-1 and j = 0..11 is derived in Table 2 based on the condition that the bit depth set is equal to the bit depth Y and the clipIdx set is equal to alf_luma_clip_idx[alf_luma_coeff_delta_idx[filtIdx]][j].

[0120] alf_chroma_clip_idx[altIdx][j] specifies the clipping index of the clipping value to be used, and then multiplies the j-th coefficient of the alternative chroma filter with index altIdx. It is a requirement for bitstream compliance that the value of alf_chroma_clip_idx[altIdx][j] for altIdx = 0..alf_chroma_num_alt_filters_minus1, j = 0..5 be in the range of 0 to 3 (including both ends).

[0121] The chroma filter clipping value AlfClipC[adaptation_parameter_set_id][altIdx] with element AlfClipC[adaptation_parameter_set_id][altIdx][j] is derived in Table 2 by setting the bit depth set equal to bit depth C and the clipIdx set equal to alf_chroma_clip_idx[altIdx][j] for the elements where altIdx = 0..alf_chroma_num_alt_filters_minus1, j = 0..5.

[0122] On the decoder side, when ALF is enabled for a CTB, each sample R(i, j) within the CU is filtered and becomes the sample value R'(i, j) as shown below.

Number

Number

Number

[0123] FIG. 10 is a diagram showing an exemplary modified block classification applied to the luma component according to an embodiment of the present disclosure. The modified block classification and filtering employed for samples near the horizontal CTU boundary can reduce the line buffer requirements of the ALF. As shown in FIG. 10, by shifting the horizontal CTU boundary by "N" samples, a virtual boundary can be defined as a line, where N is equal to 4 for the luma component and 2 for the chroma component, respectively.

[0124] For the one-dimensional Laplacian gradient calculation of 4x4 blocks above the virtual boundary, only the samples above the virtual boundary are used. Similarly, for the one-dimensional Laplacian gradient calculation of 4x4 blocks below the virtual boundary, only the samples below the virtual boundary are used. Therefore, the quantization of the activity value A is scaled by taking into account the reduction in the number of samples used in the one-dimensional Laplacian gradient calculation.

[0125] FIG. 11 is a diagram showing an exemplary modified ALF of the luma component at the virtual boundary according to some embodiments. For the filtering process, a symmetric padding operation at the virtual boundary can be used for both the luma and chroma components. As shown in FIG. 11, when the sample being filtered is located below the virtual boundary, the adjacent sample located above the virtual boundary is padded. On the other hand, the other corresponding sample is also symmetrically padded.

[0126] To improve coding efficiency, in some related examples, a coding unit synchronous picture quadtree-based ALP is used. The luma picture can be divided into a plurality of multi-level quadtree partitions, and the boundaries of each partition are aligned with the boundaries of the largest coding unit (LCU). Each partition has its own filtering process and may thus be referred to as a filter unit (FU).

[0127] A two-pass encoding flow is described below. In the first pass, a quadtree partitioning pattern and an optimal filter for each FU are determined. The filtering distortion is estimated by fast filtering distortion estimation (FFDE) during the determination process. According to the determined quadtree partitioning pattern and the selected filters for all FUs, the reconstructed picture is filtered. In the second pass, on / off control of CU synchronous ALF is performed. According to the on / off result of ALF, the picture filtered from the first pass is partially restored by the reconstructed picture.

[0128] FIG. 12 shows an exemplary LCU-aligned picture quadtree partitioning according to an embodiment. A top-down partitioning means is adopted to divide the image into multi-level quadtree partitions using a rate-distortion criterion. Each partition is called a filter unit. In the partitioning process, the quadtree partitions are aligned with the boundaries of the LCU. The encoding order of the FUs follows the z-scan order. For example, as shown in FIG. 12, the picture is divided into 10 FUs, and the encoding order is FU0, FU1, FU2, FU3, FU4, FU5, FU6, FU7, FU8, and FU9.

[0129] FIG. 13 shows an exemplary quadtree partitioning pattern corresponding to FIG. 12. To show the picture quadtree partitioning pattern, the partitioning flags can be encoded and transmitted in the z-order.

[0130] The filters for each FU can be selected from two filter sets based on the rate distortion criterion. The first set has 1 / 2 symmetric square and diamond filters newly derived for the current FU. The second set is from a time-delay filter buffer that stores the filters previously derived for the FU of the previous picture. The filter with the minimum rate distortion cost of these two sets can be selected for the current FU. Similarly, if the current FU is not the minimum FU and can be further divided into four child FUs, the rate distortion costs of the four child FUs are calculated. By recursively comparing the rate distortion costs with and without division, the quadtree partition pattern of the picture can be determined.

[0131] In some related examples, the maximum quadtree division level is 2, meaning the maximum number of FUs is 16. During quadtree division determination, the correlation values for deriving the Wiener coefficients of the 16 FUs at the lowest quadtree level (minimum FUs) can be reused. The remaining FUs can derive their Wiener filters from the correlations of the 16 FUs at the lowest quadtree level. Therefore, there is only one frame buffer access for deriving the filter coefficients of all FUs.

[0132] After the quadtree partition pattern is determined, CU-synchronized ALF on / off control can be performed to further reduce the filtering distortion. By comparing the filtering distortion and the non-filtering distortion, the leaf CU can explicitly switch the on / off of ALF in its local region. By redesigning the filter coefficients according to the on / off result of ALF, the coding efficiency can be further improved. However, the redesign process requires access to an additional frame buffer. In some related examples, to minimize the number of frame buffer accesses, there is no redesign process after the on / off determination of CU-synchronized ALF.

[0133] III. Cross-Component Adaptive Loop Filter In some related examples, a cross-component adaptive loop filter (CC-ALF) is employed. The CC-ALF refines each chroma component using luma sample values.

[0134] FIG. 14A shows an exemplary arrangement of the CC-ALF according to an embodiment of the present disclosure. FIG. 14B shows an exemplary linear diamond-shaped filter applied to the luma channel of each chroma component during CC-ALF operation. The filter coefficients are transmitted in APS and can be scaled, for example, by a factor of 2 10 and rounded for fixed-point representation. The application of the filter is controlled with a variable block size and signaled by the context-coded flag received for each block of samples. The block size is received at the slice level of each chroma component together with the CC-ALF enable flag. In one example, the following block sizes (in chroma sample units) are supported: 16×16, 32×32, 64×64.

[0135] Table 3 shows the syntax elements related to the CC-ALF.

[0136]

Table 3

[0137] In Table 3, the syntax element alf_ctb_cross_component_cb_idc[xCtb >> CtbLog2SizeY][yCtb >> CtbLog2SizeY] equal to 0 indicates that the cross-component Cb filter is not applied to the block of samples of the Cb color component at the luma position (xCtb, yCtb). The syntax element alf_ctb_cross_component_cb_idc[xCtb >> CtbLog2SizeY][yCtb >> CtbLog2SizeY] not equal to 0 indicates that the alf_ctb_cross_component_cb_idc[xCtb >> CtbLog2SizeY][yCtb >> CtbLog2SizeY]-th cross-component Cb filter is applied to the block of samples of the Cb color component at the luma position (xCtb, yCtb).

[0138] The syntax element alf_ctb_cross_component_cr_idc[xCtb >> CtbLog2SizeY][yCtb >> CtbLog2SizeY] equal to 0 indicates that the cross-component Cr filter is not applied to the block of samples of the Cr color component at the luma position (xCtb, yCtb). The syntax element alf_ctb_cross_component_cr_idc[xCtb >> CtbLog2SizeY][yCtb >> CtbLog2SizeY] not equal to 0 indicates that the alf_ctb_cross_component_cr_idc[xCtb >> CtbLog2SizeY][yCtb >> CtbLog2SizeY]-th cross-component Cr filter is applied to the block of samples of the Cr color component at the luma position (xCtb, yCtb).

[0139] IV. Chroma Sampling Format Figure 15 shows an exemplary relative position of the top-left chroma sample when chroma_format_idc is equal to 1 (4:2:0 chroma format) and chroma_sample_loc_type_top_field or chroma_sample_loc_type_bottom_field is equal to the value of the variable ChromaLocType. The area represented by the top-left 4:2:0 chroma sample is shown relative to the area represented by the top-left luma sample.

[0140] V. Constrained Directional Enhancement Filter One purpose of the in-loop Constrained Directional Enhancement Filter (CDEF) is to remove coding artifacts while preserving the details of the image. In some related examples such as HEVC, the Sample Adaptive Offset (SAO) algorithm can achieve a similar purpose by defining signaling offsets for different classes of pixels. Different from SAO, CDEF is a non-linear spatial filter. The filter design is constrained to be easily vectorizable, such as being implementable with single instruction multiple data (SIMD) operations, which does not apply to other non-linear filters such as median filters and bilateral filters.

[0141] The design of CDEF is based on the following observations. The amount of ringing artifacts in the coded image tends to be approximately proportional to the quantization step size. Although the level of detail is a property of the input image, the minimum level of detail retained in the quantized image also tends to be proportional to the quantization step size. At a given quantization step size, generally the amplitude of ringing is smaller than the amplitude of the detail.

[0142] CDEF identifies the direction of each block and adaptively filters at a smaller angle along the identified direction and along the direction rotated 45 degrees from the identified direction. Since the filter strength is explicitly signaled, advanced control over blurring is possible. An efficient encoder search for the filter strength has been designed. CDEF is based on two previously proposed in-loop filters, and the combined filter has been adopted in the new AOMedia Video 1 (AV1) codec.

[0143] Figure 16 shows an exemplary direction search for an 8×8 block according to an embodiment of the present disclosure. The direction search is performed immediately after the deblocking filter on the reconstructed pixels. Since these pixels are available at the decoder, the direction does not require signaling. The search operates on 8×8 blocks, which are small enough to handle non-linear edges appropriately and large enough to reliably estimate the direction when applied to the quantized image. By giving a certain directionality to the 8×8 region, vectorization of the filter becomes easier. For each block, the direction that best matches the pattern within each block is determined by minimizing the sum of squared differences (SSD) between the quantized block and the closest fully directional block. A fully directional block is a block in which all pixels along a line in a certain direction have the same value.

[0144] One reason for identifying the direction is to align the filter taps along that direction while maintaining the directional edges or patterns and reduce ringing. However, directional filtering alone may not be sufficient to reduce ringing significantly. It is also desirable to use filter taps for pixels not along the main direction. To reduce the risk of blurring, these extra taps are handled more conservatively. Therefore, CDEF defines primary taps and secondary taps. The complete two-dimensional CDEF filter is expressed as follows.

Equation

[0145] Loop Restoration in VI.AV1 In general, to remove noise and improve the quality of edges, in addition to the conventional deblocking operation, a set of in-loop restoration schemes can be used in video coding after deblocking. These schemes are switchable within the frame for each appropriate size tile. The specific schemes described are based on a separable symmetric Wiener filter and a dual self-guided filter using subspace projection. Since the content statistics can vary substantially within the frame, these tools are integrated into a switchable framework that can activate different tools in different regions of the frame.

[0146] In the case of the Wiener filter, all pixels in the degraded frame can be reconstructed as non-causal filtered versions of the pixels within a w×w window around each pixel, where w = 2r + 1 is odd for an integer r. When the 2D filter taps are represented by a w 2 ×1 element vector F in column vectorized form, the filter parameters given by F = H -1 M are derived by direct linear minimum mean square error (LMMSE) optimization. Here, H = E[XX Tis the autocovariance of x, and is the vectorized version of the w samples in the w×w window around the pixel in the column direction, and M = E[YX 2 is the cross-correlation between x and the scalar source sample y to be estimated. The encoder can estimate H and M from the realization of the deblocked frame and the source and send the resulting filter F to the decoder. However, doing so not only incurs a significant bitrate cost when transmitting the w T taps, but also makes decoding very complex due to non-separable filtering. Therefore, some additional constraints are imposed on the properties of F. First, F is constrained to be separable, and the filtering can be performed as separable horizontal and vertical w-tap convolutions. Second, each of the horizontal filter and the vertical filter is constrained to be symmetric. Third, it is assumed that the sum of both the horizontal filter coefficients and the vertical filter coefficients is 1. 2 In the case of guided filtering processing, the local linear model can be expressed as follows.

[0147] y = Fx + G Equation (17) y = Fx + G Equation (17)

[0148] The local linear model is used to calculate the filtered output y from the unfiltered sample x, and F and G are determined based on the statistics of the degraded image and the guidance image in the vicinity of the filtered pixel. When the guidance image is the same as the degraded image, the resulting so-called self-guided filtering has the effect of edge-preserving smoothing. A particular form of self-guided filtering depends on two parameters, the radius r and the noise parameter e, and is listed as follows.

[0149] (1) Obtain the mean μ and variance σ of the pixels in the (2r + 1)×(2r + 1) window around each pixel. This can be efficiently implemented by box filtering based on integral imaging. 2 This can be efficiently implemented by box filtering based on integral imaging.

[0150] (2) Calculate for all pixels, i.e., f = σ 2 / (σ 2 + e); g = (1 - f)μ.

[0151] (3) Calculate F and G for all pixels as the average of the values of f and g within a 3×3 window around the pixels to be used.

[0152] Filtering is controlled by r and e. The larger r is, the greater the spatial dispersion becomes, and the larger e is, the greater the range dispersion becomes.

[0153] FIG. 17 shows an exemplary subspace projection according to an embodiment. Even if neither of the inexpensive restorations X1 and X2 is close to the source Y, appropriate multipliers {α, β} can bring them quite close to the source as long as they are moving in somewhat correct directions. An inexpensive restoration can represent a restoration using relatively low-complexity calculations.

[0154] VII. Cross-Component Sample Offset Filter The cross-component sample offset (CCSO) filter is a loop filtering technique for reducing the distortion of reconstructed samples. In some embodiments, the reconstructed samples can be the samples output from an aggregator (455). The reconstructed samples can be input to a loop filter (456). In the CCSO filter, when a processed input reconstructed sample of a first color component is given, an output offset can be derived using a non-linear mapping, and the output offset can be added to the reconstructed samples of a second color component in the filtering process of the CCSO filter. The input reconstructed samples of the first color component are arranged in a filter support region.

[0155] FIG. 18 shows an exemplary filter support region according to an embodiment of the present disclosure. The filter support region includes four reconstructed samples, namely, p0, p1, p2, and p3. The four input reconstructed samples follow a cross shape in the vertical and horizontal directions. The central sample of the first color component (denoted by c) and the filtered sample of the second color component are arranged at the same location. When processing the input reconstructed samples, the following steps are applied.

[0156] Step 1: The delta values between p0 to p3 and c are first calculated and denoted as m0, m1, m2, and m3.

[0157] Step 2: The delta values m0 to m3 are further quantized. The quantized values are denoted as d0, d1, d2, and d3. The quantized values can be -1, 0, and 1 based on the following quantization process, that is, when m < -N, (a) d = -1, (b) when -N <= m <= N, d = 0, and (c) when m > N, d = 1, where N is called the quantization step size, and examples of the value of N include 4, 8, 12, and 16.

[0158] The quantized values d0 to d3 can be used to identify one combination of non - linear mappings. For example, the CCSO filter in FIG. 8 has four filter taps d0 to d3, and each filter tap can have one of three quantized values (e.g., -1, 0, and 1), so there can be a total of 3 4 = 81 combinations.

[0159] FIGS. 19A - 19B show a table presenting 81 exemplary combinations of non - linear mappings in a CCSO filter according to an embodiment in tabular form. The last column of the table represents the output offset value for each combination. Exemplary offset values are integers such as 0, 1, -1, 3, -3, 5, -5, and -7.

[0160] The final filtering process of the CCSO filter can be applied as follows. f’ = clip(f + s) Equation (18) Here, f is the reconstructed sample to be filtered, and s is the output offset value retrieved from the tables of FIGS. 19A - 19B. The filtered sample value f’ is further clipped to a range associated with the bit depth.

[0161] VIII. Improved Cross - Component Sample Offset Filter The present disclosure includes methods for improving the performance of CCSO filters, such as improved cross - component sample offset methods.

[0162] According to aspects of the present disclosure, the non - linear mapping of the CCSO filter may include specific characteristics that enable the signaling overhead indicating the mapping relationship to be compressed. Reducing the size of the mapping relationship may reduce the signaling overhead. Thus, an improvement in coding efficiency can be achieved.

[0163] In the present disclosure, the CCSO filter can be defined as a filtering process that uses the reconstructed sample of the first color component (e.g., Y, Cb, or Cr) as an input and applies the output to a second color component different from the first color component.

[0164] In the CCSO filter, a non - linear mapping can be derived on the encoder side. The non - linear mapping describes the relationship between the input reconstructed sample of one color component and the offset added to the other color component. The region of the non - linear mapping can be determined by different combinations of the processed input reconstructed samples, such as the input reconstructed samples on which linear processing has been performed.

[0165] According to aspects of the present disclosure, the non - linear mapping of the CCSO filter can be compressed to reduce the signaling overhead without significantly affecting the performance of the CCSO filter with respect to distortion reduction. As a result, the overall coding efficiency can be improved.

[0166] In some embodiments, when using non - linear mapping, an output offset can be derived when an input reconstruction sample processed when a CCSO filter is applied is given. The overhead bits for signaling the non - linear mapping can be reduced. For example, instead of explicitly signaling each entry of the non - linear mapping, only a subset of the non - linear mapping can be signaled.

[0167] According to an aspect of the present disclosure, the output offset value of input combination A can be excluded from being signaled. Instead, the output offset value of input combination A can be derived from another input combination B. This derivation process can be called mirroring derivation. One combination of inputs refers to a combination of processed input reconstruction samples as described above (e.g., d0 - d3).

[0168] In one embodiment, the output offset value of input combination A can be derived as the inverse of the output offset value of input combination B (e.g., a number with the same absolute value and a different sign), and A and B have opposite sign values for the input combinations d0 - d3. For example, if the output offset value of input combination B (d0 = a, d1 = b, d2 = c, d3 = d) is s, then the output offset value of input combination A (d0=-a, d1=-b, d2=-c, d3=-d) is - s.

[0169] In one embodiment, the output offset value of input combination A can be derived as the same number as the output offset value of input combination B, and A and B1 have opposite sign values for the processed input reconstruction samples d0 - d3. For example, if the output offset value of input combination B (d0 = a, d1 = b, d2 = c, d3 = d) is s, then the output offset value of input combination A (d0=-a, d1=-b, d2=-c, d3=-d) is s.

[0170] In one embodiment, the output offset value of input combination B can be signaled. The signaling can be provided in high-level syntax elements such as APS, slice header, frame header, PPS, SPS, video parameter set (VPS).

[0171] According to an aspect of the present disclosure, a lookup table can be used to describe a compressed non-linear mapping. Each row of the lookup table corresponds to input combinations d0~d3 and the associated output offset value. In some embodiments, a portion of the tables shown in FIGS. 19A-19B can be used to describe the compressed non-linear mapping. In one embodiment, the output offset values of the first 41 rows of the table shown in FIG. 19A can be signaled, and the output offset values of the remaining 40 rows can be derived based on the signaled output offset values of the first 41 rows. FIG. 19C shows an example in which the output offset value of the Nth row of the remaining 40 rows can be derived as the same number as the output offset value of the (80-N)th row signaled in the first 41 rows. For example, as shown in FIG. 19C, the output offset value of the 41st row is s39, which is the output offset value of the 39th row in FIG. 19A. FIG. 19D shows another example in which the output offset value of the Nth row of the remaining 40 rows can be derived as the inverse of the output offset value of the (80-N)th row signaled in the first 41 rows. For example, as shown in FIG. 19D, the output offset value of the 41st row is -s39, which is the inverse of the output offset value of the 39th row in FIG. 19A.

[0172] In one embodiment, a parameter indicating the activation of mirroring derivation can be signaled. The signaling can be provided in high-level syntax elements such as APS, slice header, frame header, PPS, SPS, VPS.

[0173] According to an aspect of the present disclosure, only a subset of the non-linear mapping is signaled. For input combinations that are not signaled, an output offset value can be derived as a default value. Examples of default values are integers such as 0, 1, -1, 3, -3, 5, -5, -7, etc.

[0174] In some embodiments, different subsets can be used and the signaled subset can be switched between different subsets. The selection of the signaled subset of the non-linear mapping can be signaled by block-level or high-level syntax elements such as, for example, APS, slice header, frame header, PPS, SPS, VPS. In one example, two subsets with 16 and 81 combinations of the non-linear mapping of the tables shown in FIGS. 19A-19B are supported and the selection is signaled as a flag. In another example, two subsets with 17 and 81 combinations of the non-linear mapping of the tables shown in FIGS. 19A-19B are supported and the selection is signaled as a flag. In another example, two subsets with 33 and 81 combinations of the non-linear mapping of the tables shown in FIGS. 19A-19B are supported and the selection is signaled as a flag. In another example, three subsets with 16, 32, and 81 combinations of the non-linear mapping of the tables shown in FIGS. 19A-19B are supported and the selection is signaled by an index. In another example, three subsets with 17, 33, and 81 combinations of the non-linear mapping of the tables shown in FIGS. 19A-19B are supported and the selection is signaled by an index. In another example, five subsets with 16, 17, 32, 33, and 81 combinations of the non-linear mapping of the tables shown in FIGS. 19A-19B are supported and the selection is signaled by an index.

[0175] FIG. 19E shows an example in which only 17 combinations of the non - linear mapping of the table shown in FIGS. 19A - 19B are signaled, and the output offset values of the remaining combinations can be derived as default values (for example, 0).

[0176] FIG. 19F shows an example in which only 16 combinations of the non - linear mapping of the table shown in FIGS. 19A - 19B are signaled, and the output offset values of the remaining combinations can be derived as default values (for example, 0).

[0177] FIG. 19G shows an example in which only 33 combinations of the non - linear mapping of the table shown in FIGS. 19A - 19B are signaled, and the output offset values of the remaining combinations can be derived as default values (for example, 0).

[0178] FIG. 19H shows an example in which only 32 combinations of the non - linear mapping of the table shown in FIGS. 19A - 19B are signaled, and the output offset values of the remaining combinations can be derived as default values (for example, 0).

[0179] According to an aspect of the present disclosure, the non - linear mapping can be divided into a plurality of subsets such as three subsets A, B, and C, and only subset B is signaled. The output offset values of the input combinations within subset A can be derived from subset B. The output offset values of the input combinations within subset C can be derived as default values. Examples of default values are integers such as 0, 1, - 1, 3, - 3, 5, - 5, - 7, etc.

[0180] In one embodiment, the output offset value of the input combination A1 in subset A can be derived as the inverse of the output offset value of the input combination B1 in subset B, and A1 and B1 have opposite sign values for the input combinations d0 to d3. The output offset value for the input combinations within subset C can be derived as a default value (for example, 0). For example, if the output offset value of the input combination B1 (d0 = a, d1 = b, d2 = c, d3 = d) within subset B is s, then the output offset value of the input combination A1 (d0 = -a, d1 = -b, d2 = -c, d3 = -d) within subset A is -s.

[0181] In one embodiment, the output offset value of the input combination A1 in subset A can be derived as the same number as the output offset value of the input combination B1 in subset B, and A1 and B1 have opposite sign values for the input combinations d0 to d3. The output offset value for the input combinations within subset C can be derived as a default value (for example, 0). For example, if the output offset value of the input combination B1 (d0 = a, d1 = b, d2 = c, d3 = d) within subset B is s, then the output offset value of the input combination A1 (d0 = -a, d1 = -b, d2 = -c, d3 = -d) within subset A is s.

[0182] In one embodiment, a parameter indicating the enabling of mirroring derivation of subsets of non - linear mapping can be signaled. The signaling can be provided by high - level syntax elements such as APS, slice header, frame header, PPS, SPS, VPS, etc.

[0183] According to an aspect of the present disclosure, only a limited set of selected output offset values is supported in non - linear mapping.

[0184] In one embodiment, the range of supported output offset values is [-4, +3], [-8, +7], [-16, +15], [-2 K , 2K can cover [K, L], predefined integers, 0 and odd integers, as well as 0 and even integers, where K and L are integers.

[0185] In one example, the set of supported output offset values can include 0 and a set of odd integers with different sign values such as 0, 1, -1, 3, -3, 5, -5, and -7. In another example, the set of supported output offset values can include 0 and a set of even integers with different sign values such as 0, 2, -2, 4, -4, 6, -6, and -8.

[0186] In one embodiment, the set of supported output offset values can be signaled. The signaling can be provided by high-level syntax elements such as APS, slice header, frame header, PPS, SPS, VPS.

[0187] In one embodiment, an array of offset values can be supported with a non-linear mapping and the index can be signaled. The signaling can be provided by high-level syntax elements such as APS, slice header, frame header, PPS, SPS, VPS to indicate the output offset value within the array selected for the input combination.

[0188] In one embodiment, different combinations of inputs can have different sets of supported output offset values.

[0189] In one embodiment, the set of supported offset values can depend on quantization parameters.

[0190] In one embodiment, the output offset value or the index of the output offset value can be signaled using fixed-length coding.

[0191] According to an aspect of the present disclosure, the output offset value for an input combination applied to a current coding block within a current slice of a current picture can be predicted from output offset values applied to different pictures, different slices, or different coding blocks.

[0192] In one embodiment, instead of signaling the output offset value, a delta value between the output offset value and a predicted output offset value can be signaled.

[0193] In one embodiment, the output offset value for at least one input combination is not signaled and can be derived from the predicted output offset value.

[0194] In one embodiment, whether the output offset value is predicted can be signaled through a flag. The flag can be provided in high-level syntax elements such as APS, slice header, frame header, PPS, SPS, VPS.

[0195] The prediction of the output offset value may depend on coding information including, but not limited to, frame type, temporal layer, and quantization parameter.

[0196] In some related examples, one or more filtering processes may be applied to the reconstructed samples of the coded block. For example, a deblocking module may be applied to the reconstructed samples of the coded block. Next, the CDEF module can be applied to the filtered reconstructed samples output from the deblocking module, and the loop filter module can be applied to the filtered reconstructed samples output from the CDEF module.

[0197] According to an aspect of the present disclosure, when a CCSO filter is applied to a reconstructed sample of a coded block, the input position and the output position of the CCSO filter are flexible. The input position of the CCSO filter is where the input reconstructed sample of the CCSO filter is selected. The output position of the CCSO filter is where the output offset value of the CCSO filter is applied (e.g., where CCSO filtering is applied).

[0198] In some embodiments, the input and output can share the same position. That is, there is no other coding module disposed between the input position and the output position of the CCSO filter. In one example, both the input position and the output position can be immediately before the deblocking module. In another example, both the input position and the output position can be after the deblocking module and before the CDEF module. In another example, both the input position and the output position can be after the CDEF module and before the loop recovery filter (LR) module. In another example, both the input position and the output position can be after the LR module.

[0199] In some embodiments, the input position and the output position may be different, and the input position is in front of the output position. That is, there is at least one coding module disposed between the input position and the output position of the CCSO filter. In one example, the input position can be immediately before the deblocking module, and the output position can be after the deblocking module and before the CDEF module. In another example, the input position can be after the deblocking module and before the CDEF module, and the output position can be after the CDEF module and before the LR module. In another example, the input position can be after the CDEF module and before the LR module, and the output position can be after the LR module.

[0200] In one embodiment, the CDEF and LR modules of the above embodiments can be replaced with an SAO module and an ALF module, respectively. The same methods as those of the above embodiments can be applied to this embodiment.

[0201] IX. Flowchart FIG. 20 shows a flowchart illustrating an overview of an exemplary process (2000) according to an embodiment of the present disclosure. In various embodiments, the process (2000) is executed by processing circuits such as processing circuits in terminal devices (210), (220), (230), and (240), a processing circuit that executes the functions of video encoder (303), a processing circuit that executes the functions of video decoder (310), a processing circuit that executes the functions of video decoder (410), a processing circuit that executes the functions of intra prediction module (452), a processing circuit that executes the functions of video encoder (503), a processing circuit that executes the functions of predictor (535), a processing circuit that executes the functions of intra encoder (622), and a processing circuit that executes the functions of intra decoder (772). In some embodiments, the process (2000) is implemented within software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit executes the process (2000).

[0202] Process (2000) can generally start at step (S2010). Process (2000) decodes prediction information of a current block in a current picture that is part of a coded video sequence. The prediction information indicates a plurality of combinations of processed reconstructed samples and corresponding output values of filtering. Filtering can be used to reduce compression artifacts and distortion of the reconstructed samples. The filtering can use processed reconstructed samples of a first color component as input (e.g., Y, Cb, or Cr) and apply the output to a second color component different from the first color component. In some embodiments, the filtering can be performed in a loop filter unit (456). Examples of filtering include, but are not limited to, CCSO and CC-ALF. Then, process (2000) proceeds to step (S2020).

[0203] In step (S2020), process (2000) performs filtering on the processed reconstructed samples of the first color component of the current block. The processed reconstructed samples of the first color are not included in the plurality of combinations of processed reconstructed samples. Then, process (2000) proceeds to step (S2030).

[0204] In step (S2030), process (2000) determines the output value of the filtering based on one of the default value, the predicted output value, and one of the output values indicated in the prediction information. Then, process (2000) proceeds to step (S2040).

[0205] In step (S2040), process (2000) reconstructs the current block based on the output value applied to the reconstructed samples of the second color component of the current block. The second color component is different from the first color component. Then, process (2000) ends.

[0206] In one embodiment, the process (2000) determines, as the output value of the filtering process, one of a first value and a second value. The first value is one of the output values indicated in the prediction information, and the second value is the inverse of the first value.

[0207] In one embodiment, each of the processed reconstruction samples of the first color component is the inverse of the corresponding processed reconstruction sample in one of a plurality of combinations of processed reconstruction samples indicated in the prediction information, and one of the output values indicated in the prediction information corresponds to one of the plurality of combinations of processed reconstruction samples indicated in the prediction information.

[0208] In one embodiment, the sum of the index of the processed reconstruction sample of the first color component in the mapping table of the filtering process and the index of one of the output values indicated in the prediction information in the mapping table of the filtering process is equal to the maximum index of the mapping table.

[0209] In one embodiment, the process (2000) determines a default value as the output value of the filtering process based on prediction information indicating that a plurality of combinations of processed reconstruction samples indicated in the prediction information are a subset of the total number of combinations of processed reconstruction samples in the filtering process.

[0210] In one embodiment, the predicted output value is determined based on the output value of the filtering process applied to another coding block.

[0211] In one embodiment, the processed reconstruction sample of the first color component is determined based on the reconstruction sample of the first color component output from the first coding module, and the reconstruction sample of the second color component is output from the first coding module or the second coding module applied after the first coding module.

[0212] X. Computer System The technology described above can be implemented as computer software using computer-readable instructions physically stored on one or more computer-readable media. For example, FIG. 21 shows a computer system (2100) suitable for implementing certain embodiments of the disclosed subject matter.

[0213] The computer software can be coded using any suitable machine code or computer language that can undergo assembly, compilation, linking, or similar mechanisms to create code that includes instructions that can be executed directly by, for example, one or more computer central processing units (CPUs) and graphics processing units (GPUs), or through interpretation and microcode execution.

[0214] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.

[0215] The components shown in FIG. 21 with respect to the computer system (2100) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. Also, the configuration of the components should not be construed as having dependencies or requirements related to any one or combination of the components shown in the exemplary embodiments of the computer system (2100).

[0216] The computer system (2100) may include a specific human interface input device. Such a human interface input device may respond to input by one or more human users via, for example, tactile input (keystrokes, swipes, movement of a data glove, etc.), voice input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not shown). The human interface device may also be used to capture specific media such as audio (voice, music, environmental sounds, etc.), images (scanned images, photographic images obtained from a still image camera, etc.), video (2D video, 3D video including stereoscopic video, etc.) that are not necessarily directly related to conscious input by a human.

[0217] The input human interface device may include one or more (only one of each shown) of a keyboard (2101), a mouse (2102), a trackpad (2103), a touch screen (2110), a data glove (not shown), a joystick (2105), a microphone (2106), a scanner (2107), and a camera (2108).

[0218] The computer system (2100) may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more human users, for example, by tactile output, sound, light, and smell / taste. Such human interface output devices include, for example, tactile output devices (e.g., a touch screen (2110), a data glove (not shown), or a joystick (2105) with tactile feedback, although there may also be tactile feedback devices that do not function as input devices), audio output devices (such as speakers (2109), headphones (not shown), etc.), visual output devices (such as screens (2110) including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have a touch screen input function, and each of which may or may not have a tactile feedback function, and some of which may be capable of two-dimensional visual output or three-dimensional or higher output through means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and may include a printer (not shown). These visual output devices (such as screens (2110)) can be connected to the system bus (2148) via a graphics adapter (2150).

[0219] The computer system (2100) may also include optical media such as a CD / DVD ROM / RW (2120) having a medium (2121) such as a CD / DVD, a thumb drive (2122), a removable hard drive or a solid state drive (2123), legacy magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown), etc., and human-accessible storage devices and their associated media.

[0220] One of ordinary skill in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not include a transmission medium, a carrier wave, or other transient signals.

[0221] The computer system (2100) can also include a network interface (2154) to one or more communication networks (2155). The one or more communication networks (2155) can be, for example, wireless, wired, or optical. Further, the one or more communication networks (2155) can be local, wide area, metropolitan, vehicle and industrial, real-time, delay tolerant, etc. Examples of the one or more communication networks (2155) include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide area digital networks including cable TV, satellite TV, terrestrial TV, etc., vehicle or industrial networks including CANBus, etc. Certain networks generally require an external network interface adapter (such as a USB port of the computer system (2100)) attached to a specific general-purpose data port or peripheral bus (2149), and others are generally integrated into the core of the computer system (2100) by attachment to a system bus as described later (such as an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2100) can communicate with other entities. Such communication can be, for example, unidirectional receive only (such as broadcast TV), unidirectional transmit only (such as CANbus to a specific CANbus device), or bidirectional to other computer systems using a local or wide area digital network. Specific protocols and protocol stacks can be used for each of those networks and network interfaces as described above.

[0222] The foregoing human interface devices, human-accessible storage devices, and network interfaces can be attached to the core (2140) of the computer system (2100).

[0223] The core (2140) can include one or more central processing units (CPUs) (2141), a graphics processing unit (GPU) (2142), a dedicated programmable processing device in the form of a field programmable gate array (FPGA) (2143), a hardware accelerator (2144) for specific tasks, a graphics adapter (2150), etc. These devices may be connected via a system bus (2148) together with a read-only memory (ROM) (2145), a random access memory (2146), an internal large-capacity storage such as an internal hard drive or SSD that is not accessible to the user (2147). In some computer systems, access to the system bus (2148) can be in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus (2148) or via a peripheral bus (2149). In one example, a screen (2110) can be connected to the graphics adapter (2150). Peripheral bus architectures include PCI and USB, etc.

[0224] The CPU (2141), GPU (2142), FPGA (2143), and accelerator (2144) can execute specific instructions that can, in combination, constitute the aforementioned computer code. That computer code can be stored in the ROM (2145) or RAM (2146). Migration data can also be stored in the RAM (2146), while persistent data can be stored, for example, in the internal large-capacity storage (2147). Fast storage and retrieval to / from any of the memory devices can be enabled by the use of cache memory that can be closely associated with one or more CPUs (2141), GPUs (2142), large-capacity storage (2147), ROM (2145), RAM (2146), etc.

[0225] A computer-readable medium can have computer code for performing various computer-implemented operations. The medium and the computer code can be specially designed and constructed for the purposes of this disclosure, or they can be of the kinds that are available to and well known to those of ordinary skill in the computer software arts.

[0226] As an example, without limitation, a computer system having an architecture (2100), specifically a core (2140), can provide functionality as a result of software executed by one or more tangible computer-readable media by a processor(s) (including a CPU, GPU, FPGA, accelerator, etc.). Such computer-readable media can be associated with user-accessible mass storage as described above, as well as media associated with specific storage of the core (2140) of a non-transitory nature such as core internal mass storage (2147) or ROM (2145). The software implementing various embodiments of the present disclosure can be stored on such a device and executed by the core (2140). The computer-readable media can include one or more memory devices or chips depending on specific requirements. The software can cause the core (2140), specifically the processor(s) (including a CPU, GPU, FPGA, etc.) therein, to execute a specific process or a specific part of a specific process described herein, which includes defining a data structure stored in the RAM (2146) and modifying such a data structure according to a process defined by the software. Additionally, or alternatively, the computer system can provide functionality as a result of logic embodied in circuitry (e.g., an accelerator (2144)) that is hardwired or otherwise, and can execute a specific process or a specific part of a specific process described herein instead of or in conjunction with the software. References to software can include logic, and vice versa where appropriate. Where appropriate, references to computer-readable media can include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0227] Although the present disclosure has described several exemplary embodiments, there are modifications, substitutions, and various alternative equivalents within the scope of the present disclosure. Therefore, it will be understood by those skilled in the art that many systems and methods can be devised that embody the principles of the present disclosure and are thus within its spirit and scope, even though not explicitly shown or described herein. Appendix A: Acronyms ALF: Adaptive Loop Filter AMVP: Advanced Motion Vector Prediction APS: Adaptation Parameter Set ASIC: Application-Specific Integrated Circuit ATMVP: Alternative / Advanced Temporal Motion Vector Prediction AV1: AOMedia Video 1 AV2: AOMedia Video 2 BMS: Benchmark Set BV: Block Vector CANBus: Controller Area Network Bus CB: Coding Block CC-ALF: Cross-Component Adaptive Loop Filter CD: Compact Disc CDEF: Constrained Directional Enhancement Filter CPR: Current Picture Referencing CPU: Central Processing Unit CRT: Cathode Ray Tube CTB: Coding Tree Block CTU: Coding Tree Unit CU: Coding Unit DPB: Decoder Picture Buffer DPCM: Differential Pulse-Code Modulation DPS: Decoding Parameter Set DVD: Digital Video Disc FPGA: Field Programmable Gate Area JCCR: Joint CbCr Residual Coding JVET: Joint Video Exploration Team GOP: Groups of Pictures GPU: Graphics Processing Unit GSM: Global System for Mobile communication HDR: High Dynamic Range HEVC: High Efficiency Video Coding HRD: Hypothetical Reference Decoder IBC: Intra Block Copy IC: Integrated Circuit ISP: Intra Sub-Partition JEM: Joint Exploration Model LAN: Local Area Network LCD: Liquid-Crystal Display LR: Loop Restoration Filter LRU: Loop Restoration Unit LTE: Long-Term Evolution MPM: Most Probable Mode MV: Motion Vector OLED: Organic Light-Emitting Diode PB: Prediction Block PCI: Peripheral Component Interconnect PDPC: Position Dependent Prediction Combination PLD: Programmable Logic Device PPS: Picture Parameter Set PU: Prediction Unit RAM: Random Access Memory ROM: Read-Only Memory SAO: Sample Adaptive Offset SCC: Screen Content Coding SDR: Standard Dynamic Range SEI: Supplementary Enhancement Information SNR: Signal Noise Ratio SPS: Sequence Parameter Set SSD: Solid-state Drive TU: Transform Unit USB: Universal Serial Bus VPS: Video Parameter Set VUI: Video Usability Information VVC: Versatile Video Coding WAIP: Wide-Angle Intra Prediction

Explanation of Symbols

[0228] 101 points 102, 103 Arrows 104 Square Block 105 Schematic Diagram 111 Current Block 112, 113, 114, 115, 116 Surrounding Samples 200 Communication System 210, 220, 230, 240 Terminal Devices 250 Network 301 Video Source Stream of uncompressed video pictures 303, 503, 603 Video encoders 304 Encoded video data 305 Streaming server 306, 308 Client subsystems 307, 309 Copies of encoded video data 310, 410, 710 Video decoders 311 Output stream 312 Display 313 Capture subsystem 320, 330, 430, 520 Electronic devices 401 Channel 412 Render device 415 Buffer memory 420 Parser 421 Symbol 431 Receiver 451 Scaler / inverse transform unit 452 Intra-picture prediction unit 453 Motion compensation prediction unit 455 Aggregator 456 Loop filter unit 457 Reference picture memory 458 Current picture buffer 501 Video source 530 Source coder 532 Coding engine 533 Decoder 534 Reference picture memory 535 Predictor 540 Transmitter 543 Encoded video sequence 545 Entropy coder 550 Controller 560 Communication channel 621 General-purpose controller 622 Intra encoder 623 Residual calculator 624 Residual Encoder 625 Entropy Encoder 626 Switch 628 Residual Decoder 630 Inter Encoder 771 Entropy Decoder 772 Intra Decoder 773 Residual Decoder 774 Reconstruction Module 780 Inter Decoder 2100 Computer System 2101 Keyboard 2102 Mouse 2103 Track Pad 2105 Joystick 2106 Microphone 2107 Scanner 2108 Camera 2109 Speaker 2110 Touch Screen 2120 Optical Medium 2121 Medium 2122 Thumb Drive 2123 Removable Hard Drive or Solid State Drive 2140 Core 2141 Processing Device (CPU) 2142 Graphics Processing Unit (GPU) 2143 Field Programmable Gate Array (FPGA) 2144 Accelerator 2145 Read Only Memory (ROM) 2146 Random Access Memory (RAM) 2147 Mass Storage 2148 System Bus 2149 Peripheral Bus 2150 Graphics Adapter 2154 Network Interface 2155 Communication Network

Claims

1. A method of video encoding performed by an encoder, comprising: determining prediction information indicative of a plurality of combinations of the processed reconstructed samples and corresponding output values ​​of the filtering operation; performing the filtering operation on processed reconstructed samples of a first color component of a current block in a current picture, wherein the processed reconstructed samples of the first color component are not included in the plurality of combinations of the processed reconstructed samples indicated in the prediction information; determining an output value of the filtering process based on one of a default value, a predicted output value, and one of the output values ​​indicated in the prediction information; determining prediction information indicative of a plurality of combinations of the reconstructed samples to be processed and the corresponding output values ​​of the filtering operation, by coding prediction information of a current block in the current picture into a video sequence; the output values ​​applied to reconstructed samples of a second color component of the current block are used to reconstruct the current block, the second color component being different from the first color component; method. determining an output value of the filtering process; determining one of a first value and a second value as the output value of the filtering process, the first value being one of the output values ​​indicated in the prediction information, and the second value being the reciprocal of the first value; The method of claim 1, comprising:

3. A method as described in claim 1 or 2, wherein each of the processed reconstructed samples of the first color component is the inverse of the corresponding processed reconstructed sample in one of the multiple combinations of processed reconstructed samples indicated in the prediction information, and one of the output values ​​indicated in the prediction information corresponds to one of the multiple combinations of processed reconstructed samples indicated in the prediction information.

4. A method described in any one of claims 1 to 3, wherein the sum of the index of the processed reconstructed sample of the first color component in the mapping table of the filtering process and the index of one of the output values ​​indicated in the prediction information in the mapping table of the filtering process is equal to the maximum index of the mapping table.

5. Determining the output value of the filtering process includes determining the default value as the output value of the filtering process based on the prediction information indicating that the plurality of combinations of processed reconstructed samples indicated in the prediction information are a subset of a total number of combinations of processed reconstructed samples in the filtering process.

2. The method of claim 1, comprising:

6. A method described in any one of claims 1 to 5, wherein the predicted output value is determined based on the output value of the filtering process applied to another coding block.

7. A method described in any one of claims 1 to 6, wherein the processed reconstructed sample of the first color component is determined based on the reconstructed sample of the first color component output from a first coding module, and the reconstructed sample of the second color component is output from the first coding module or a second coding module applied after the first coding module.

8. An apparatus configured to perform a method according to any one of claims 1 to 7.

9. A computer program product for causing a processor to carry out the method of any one of claims 1 to 7.

10. A method for transmitting a bitstream, comprising: generating a bitstream, determining prediction information indicative of a plurality of combinations of the processed reconstructed samples and corresponding output values ​​of the filtering operation; performing the filtering operation on processed reconstructed samples of a first color component of a current block in a current picture, wherein the processed reconstructed samples of the first color component are not included in the plurality of combinations of the processed reconstructed samples indicated in the prediction information; determining an output value of the filtering process based on one of a default value, a predicted output value, and one of the output values ​​indicated in the prediction information; determining prediction information indicative of a plurality of combinations of the reconstructed samples to be processed and the corresponding output values ​​of the filtering operation, by coding prediction information of a current block in the current picture into a video sequence; the output values ​​applied to reconstructed samples of a second color component of the current block are used to reconstruct the current block, the second color component being different from the first color component; generating a bitstream; transmitting the bitstream; A method comprising:

Images