Method and apparatus for video decoding

Abstract

Aspects of the present disclosure provide methods and apparatuses for video encoding / decoding. In some examples, a method for video decoding includes: generating first reconstructed samples of a current block in a current frame; obtaining reconstructed samples in a temporal source frame different from the current frame; applying a filter to at least the reconstructed samples in the temporal source frame to generate an offset; and combining the first reconstructed samples of the current block with the offset to generate second reconstructed samples of the current block.

Description

[0001] Incorporated herein by reference

[0002] This disclosure claims priority to U.S. Provisional Application No. 62 / 975,516, filed on February 12, 2020, with the title "Time Interleaved Component Filter", and U.S. Application No. 17 / 091,605, filed on November 6, 2020, with the title "Methods and Apparatus for Interleaved Component Filtering", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to video coding and decoding technologies. Specifically, it relates to methods and apparatuses, devices, and computer-readable media for video decoding. Background Art

[0004] Through the inter-picture prediction technology with motion compensation, video encoding and decoding can be performed. Uncompressed digital video may include a series of pictures, each picture having a spatial dimension such as 1920×1080 luminance samples and associated chrominance samples. The series of pictures has a fixed or variable picture rate (also informally referred to as the frame rate), such as 60 pictures per second or 60 Hz. Uncompressed video has specific bitrate requirements. For example, a 1080p60 4:2:0 video with 8 bits per sample (1920x1080 luminance sample resolution, 60 Hz frame rate) requires a bandwidth of nearly 1.5 Gbit / s. An hour of such video would require more than 600 GB of storage space.

[0005] One purpose of video encoding and decoding is to reduce redundant information in the input video signal through compression. Video compression can help reduce the requirements for the above-mentioned bandwidth and / or storage space, and in some cases can reduce by two or more orders of magnitude. Lossless compression, lossy compression, and combinations of both can be employed. Lossless compression refers to a technique for reconstructing an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be exactly the same as the original signal, but the distortion between the original signal and the reconstructed signal is small enough such that the reconstructed signal can be used for the intended application. Lossy compression is widely used in video. The amount of allowable distortion depends on the application. For example, users of certain consumer streaming applications can tolerate higher distortion compared to users of television applications. The achievable compression ratio reflects that higher allowed / tolerable distortion can result in a higher compression ratio.

[0006] Video encoders and decoders can utilize several major categories of technologies, such as including: motion compensation, transformation, quantization, and entropy coding.

[0007] Video codec technology may include known intra - coding techniques. In intra - coding, sample values are represented without reference to samples or other data of previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into sample blocks. When all sample blocks are encoded in the intra - mode, the picture can be an intra - picture. Intra - pictures and their derivatives (e.g., independent decoder refresh pictures) can be used to reset the decoder state and can thus be used as the first picture in an encoded video bitstream and a video session, or as a still image. Samples of an intra - block can be used for transformation, and the transform coefficients can be quantized before entropy coding. Intra - prediction can be a technique that minimizes the sample values in the pre - transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficients, the fewer bits are needed to represent the block after entropy coding for a given quantization step size.

[0008] As is known from, for example, MPEG - 2 generation coding techniques, traditional intra - coding does not use intra - prediction. However, some newer video compression techniques include techniques that attempt to obtain data blocks from, for example, surrounding sample data and / or metadata, where the surrounding sample data and / or metadata are obtained during spatially adjacent encoding / decoding and before the decoding order. Such techniques have later been referred to as "intra - prediction" techniques. Note that, at least in some cases, intra - prediction uses only the reference data of the currently being reconstructed current picture and not the reference data of the reference pictures.

[0009] There can be many different forms of intra - prediction. When more than one such technique can be used in a given video coding technique, the techniques used can be encoded in an intra - prediction mode. In some cases, a mode can have sub - modes and / or parameters, and these modes can be encoded separately or included in a mode codeword. Which codeword is used for a given mode / sub - mode / parameter combination affects the coding efficiency gain through intra - prediction, and thus also for the entropy coding technique used to convert the codewords into a bitstream. SUMMARY OF THE INVENTION

[0010] Aspects of the present disclosure provide methods and apparatuses, devices, and computer - readable media for video coding / decoding, aiming to solve the problem that the source of cross - component filtering (CCF) is limited to the reconstructed samples of the current block in the current frame, thus restricting the CCF performance.

[0011] In some examples, a method of video decoding includes: generating first reconstructed samples of a current block in a current frame; obtaining reconstructed samples in a temporal source frame different from the current frame;

[0012] applying a filter to at least the reconstructed samples in the temporal source frame to generate an offset; and combining the first reconstructed samples of the current block with the offset to generate second reconstructed samples of the current block.

[0013] In some examples, the apparatus for video decoding includes: a generation module configured to generate first reconstructed samples of a current block in a current frame; an acquisition module configured to acquire reconstructed samples in a temporal source frame different from the current frame; an application module configured to apply a filter to at least the reconstructed samples in the temporal source frame to generate an offset; and a combination module configured to combine the first reconstructed samples of the current block with the offset to generate second reconstructed samples of the current block.

[0014] Embodiments of the present disclosure also provide a video decoding device, including: one or more processors and one or more memories, where at least one program instruction is stored in the one or more memories, and the at least one program instruction is loaded and executed by the one or more processors to implement the video decoding method.

[0015] Embodiments of the present disclosure also provide a non-volatile computer-readable medium for storing program instructions, which, when executed by a computer for video encoding / decoding, cause the computer to execute the video decoding method.

[0016] In the embodiments of the present disclosure, samples of a frame different from the current frame are used as the source of the CCF, so that the source of the CCF is no longer limited to the reconstructed samples of the current block in the current frame, thereby improving the performance of the CCF. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Other features, properties, and various advantages of the disclosed subject matter will become further apparent from the following detailed description and the accompanying drawings.

[0018] Where:

[0019] Figure 1A A schematic diagram showing an exemplary subset of intra prediction modes;

[0020] Figure 1B A schematic diagram showing exemplary intra prediction directions;

[0021] Figure 2 A schematic diagram showing a current block and its surrounding spatial merge candidates in an example;

[0022] Figure 3 A schematic diagram showing a simplified block diagram of a communication system according to an embodiment;

[0023] Figure 4 A schematic diagram showing a simplified block diagram of a communication system according to an embodiment;

[0024] Figure 5 A schematic diagram showing a simplified block diagram of a decoder according to an embodiment;

[0025] Figure 6 A schematic diagram showing a simplified block diagram of an encoder according to an embodiment;

[0026] Figure 7 A block diagram of an encoder according to another embodiment;

[0027] Figure 8 A block diagram of a decoder according to another embodiment;

[0028] Figure 9 An example of a filter shape according to an embodiment of the present disclosure;

[0029] Figures 10A - 10D An example of a subsampling position for calculating a gradient according to an embodiment of the present disclosure;

[0030] Figures 11A - 11B An example of a virtual boundary filtering process according to an embodiment of the present disclosure;

[0031] Figures 12A - 12F An example of a symmetric padding operation at a virtual boundary according to an embodiment of the present disclosure;

[0032] Figure 13 An example of partitioning of a picture according to some embodiments of the present disclosure;

[0033] Figure 14 An example of a quadtree segmentation pattern in some examples;

[0034] Figure 15 A cross-component filter according to an embodiment of the present disclosure;

[0035] Figure 16 An example of a filter shape according to an embodiment of the present disclosure;

[0036] Figure 17 A syntax example for a cross-component filter according to some embodiments of the present disclosure;

[0037] Figures 18A - 18B An exemplary position of chrominance samples relative to luminance samples according to an embodiment of the present disclosure;

[0038] Figure 19 An example of a direction search according to an embodiment of the present disclosure;

[0039] Figure 20 An example of subspace projection in some examples.

[0040] Figure 21 A flowchart showing an overview process according to an embodiment of the present disclosure.

[0041] Figure 22 is a schematic diagram of a computer system according to an embodiment. Detailed implementation

[0042] H.264 introduced an intra prediction mode, which was improved in H.265 and further improved in updated coding technologies such as the Joint Exploration Model (JEM) / Versatile Video Coding (VVC) / Benchmark Model Set (BMS). A prediction block can be formed by using the adjacent sample values belonging to the already available samples. In some examples, the sample values of the adjacent samples are copied into the prediction block in a certain direction. The reference to the direction used can be encoded in the bitstream or can itself be predicted.

[0043] Referring to Figure 1A , a subset of nine known prediction directions out of 33 possible prediction directions from H.265 (corresponding to 33 angular modes of 35 intra modes) is depicted in the lower right. The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which the sample is being predicted. For example, arrow (102) represents predicting sample (101) based on one or more samples at a 45-degree angle to the horizontal in the upper right. Similarly, arrow (103) represents predicting sample (101) based on one or more samples at a 22.5-degree angle to the horizontal in the lower left.

[0044] Still referring to Figure 1A , a square block (104) including 4×4 samples (represented by a thick dashed line) is shown in the upper left. The square block (104) includes 16 samples, each sample being labeled with "S", as well as its position in the Y dimension (e.g., row index) and in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (starting from the top) and the first sample in the X dimension (starting from the left). Similarly, sample S44 is the fourth sample in both the Y dimension and the X dimension within block (104). Since the block is of 4×4 sample size, S44 is located in the lower right corner. Reference samples following a similar numbering scheme are also shown. The reference samples are labeled with "R", as well as their Y position (e.g., row index) and X position (e.g., column index) relative to block (104). In H.264 and H.265, the predicted sample is adjacent to the block being reconstructed, so negative values are not needed.

[0045] Intra picture prediction can be performed by copying reference sample values from neighboring samples occupied by the predicted direction of signaling. For example, assume that the coded video bitstream includes signaling that, for a block, indicates a prediction direction consistent with arrow (102), i.e., samples are predicted based on one or more prediction samples at a 45-degree angle from the upper right to the horizontal direction. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Then, sample S44 is predicted based on reference sample R08.

[0046] In some cases, such as by interpolation, the values of multiple reference samples can be combined to calculate a reference sample, especially when the direction is not divisible by 45 degrees.

[0047] With the development of video coding technology, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. In H.265 (2013) and JEM / VVC / BMS, it increased to 33, and at the time of this application, up to 65 directions can be supported. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding are used to represent those likely directions using fewer bits, accepting certain costs for the less likely directions. Additionally, sometimes the direction itself can be predicted based on neighboring directions used in neighboring, already decoded blocks.

[0048] Figure 1B A schematic diagram (180) is shown that depicts 65 intra prediction directions according to JEM to illustrate the increase in the number of prediction directions over time.

[0049] The mapping of intra prediction direction bits in the coded video bitstream representing the direction can vary depending on the video coding technology and, for example, can range from a simple direct mapping of the intra prediction mode to the prediction direction of the codeword to a complex adaptive scheme including the most likely modes and similar techniques. However, in all cases, there may be certain directions in the video content that are statistically less likely to occur than other directions. Since the purpose of video compression is to reduce redundancy, in well-performing video coding techniques, those less likely directions will use a greater number of bits to represent compared to the more likely directions.

[0050] Motion compensation can be a lossy compression technique and can involve techniques where a block of sample data from a previously reconstructed picture or a portion of a reconstructed picture (reference picture) is spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV) and used for prediction of a newly reconstructed picture or picture portion. In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV can have two dimensions X and Y, or three dimensions, where the third dimension represents the reference picture in use (which can indirectly be a temporal dimension).

[0051] In some video compression techniques, the MV applied to a certain sample data region can be predicted based on other MVs, for example, based on those MVs that are related to another sample data region that is spatially adjacent to the region being reconstructed and that are in front of the MV in decoding order. This can greatly reduce the amount of data required to encode the MV, thereby eliminating redundant information and increasing the amount of compression. MV prediction can be carried out effectively. For example, when encoding an input video signal (referred to as natural video) derived from a camera, there is a statistical likelihood that regions larger than the area applicable to a single MV will move in a similar direction. Therefore, in some cases, a similar motion vector derived from the MVs of adjacent regions can be used for prediction. This results in the MV found for a given region being similar or identical to the MV predicted based on the surrounding MVs, and after entropy coding, it can be represented with fewer bits than the number of bits used when directly encoding the MV. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from an original signal (i.e., sample stream). In other cases, MV prediction itself may be lossy, for example, due to rounding errors generated when calculating the predicted value based on several surrounding MVs.

[0052] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding", December 2016). Among the various MV prediction mechanisms provided by H.265, the technique hereinafter referred to as "spatial merge" is described herein.

[0053] Please refer to Figure 2 , the current block (201) includes samples that have been discovered by the encoder during the motion search process, and the samples can be predicted based on a previous block of the same size that has been spatially offset. Additionally, the MV can be derived from the metadata associated with one or more reference pictures instead of directly encoding the MV. For example, the MV is derived from the metadata of the nearest reference picture using the MV associated with any one of five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 202 to 206 respectively) (in decoding order). In H.265, MV prediction can use the predicted value of the same reference picture that the adjacent blocks are also using.

[0054] Figure 3 is a simplified block diagram of a communication system (300) according to an embodiment disclosed in the present application. The communication system (300) includes a plurality of terminal devices, and the terminal devices can communicate with each other through, for example, a network (350). For example, the communication system (300) includes a first terminal device (310) and a second terminal device (320) interconnected through a network (350). In Figure 3 the embodiment of, the first terminal device (310) and the second terminal device (320) perform unidirectional data transmission. For example, the first terminal device (310) can encode video data (such as a video picture stream collected by the terminal device (310)) for transmission to the second terminal device (320) through the network (350). The encoded video data is transmitted in the form of one or more encoded video bitstreams. The second terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to recover the video data, and display video pictures according to the recovered video data. Unidirectional data transmission is more common in applications such as media services.

[0055] In another embodiment, the communication system (300) includes a third terminal device (330) and a fourth terminal device (340) that perform bidirectional transmission of encoded video data, and the bidirectional transmission can occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (330) and the fourth terminal device (340) can encode video data (such as a video picture stream collected by the terminal device) for transmission to the other terminal device among the third terminal device (330) and the fourth terminal device (340) through the network (350). Each of the third terminal device (330) and the fourth terminal device (340) can also receive the encoded video data transmitted by the other terminal device among the third terminal device (330) and the fourth terminal device (340), and can decode the encoded video data to recover the video data, and can display video pictures on an accessible display device according to the recovered video data.

[0056] In Figure 3In an embodiment, the first terminal device (310), the second terminal device (320), the third terminal device (330), and the fourth terminal device (340) may be servers, personal computers, and smart phones, but the principles disclosed in this application are not limited thereto. The embodiments disclosed in this application are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing devices. The network (350) represents any number of networks that transmit the encoded video data between the first terminal device (310), the second terminal device (320), the third terminal device (330), and the fourth terminal device (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) may exchange data in a circuit-switched and / or packet-switched channel. The network may include a telecommunications network, a local area network, a wide area network, and / or the Internet. For the purposes of this application, unless otherwise explained below, the architecture and topology of the network (350) may be irrelevant to the operations disclosed in this application.

[0057] As an example, Figure 4 illustrates the placement of the video encoder and the video decoder in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, and so on.

[0058] The streaming system may include an acquisition subsystem (413), which may include a video source (401) such as a digital camera that creates an uncompressed video picture stream (402). In an embodiment, the video picture stream (402) includes samples taken by the digital camera. Compared to the encoded video data (404) (or the encoded video bitstream), the video picture stream (402) is depicted as a thick line to emphasize the high data volume of the video picture stream. The video picture stream (402) may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or carry out aspects of the disclosed subject matter described in more detail below. Compared to the video picture stream (402), the encoded video data (404) (or the encoded video bitstream (404)) is depicted as a thin line to emphasize the lower data volume of the encoded video data (404) (or the encoded video bitstream (404)), which may be stored on the streaming server (405) for future use. One or more streaming client subsystems, such as Figure 4The client subsystems (406) and client subsystems (408) therein can access the streaming server (405) to retrieve copies (407) and copies (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and generates an output video picture stream (411) that can be presented on a display (412) (such as a display screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (404), video data (407), and video data (409) (such as a video bitstream) can be encoded according to certain video coding / compression standards. Embodiments of such standards include ITU-T H.265. In an embodiment, a video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application can be used in the context of the VVC standard.

[0059] It should be noted that the electronic device (420) and the electronic device (430) can include other components (not shown). For example, the electronic device (420) can include a video decoder (not shown), and the electronic device (430) can further include a video encoder (not shown).

[0060] Figure 5 is a block diagram of a video decoder (510) according to an embodiment disclosed in the present application. The video decoder (510) can be provided in an electronic device (530). The electronic device (530) can include a receiver (531) (such as a receiving circuit). The video decoder (510) can be used to replace Figure 4 the video decoder (410) in the embodiment.

[0061] A receiver (531) may receive one or more encoded video sequences to be decoded by a video decoder (510); in the same or another embodiment, one encoded video sequence is received at a time, where the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data as well as other data, e.g., encoded audio data and / or auxiliary data streams that may be forwarded to their respective using entities (not shown). The receiver (531) may separate the encoded video sequences from the other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and an entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, the buffer memory (515) may be provided external to the video decoder (510) (not shown). In other cases, a buffer memory is provided external to the video decoder (510) (not shown) to, for example, prevent network jitter, and another buffer memory (515) may be configured inside the video decoder (510) to, for example, handle playout timing. And when the receiver (531) receives data from a storage / forward device with sufficient bandwidth and controllability or from an isochronous network, it may also be possible not to configure the buffer memory (515), or the buffer memory may be made smaller. Of course, for use on a service packet network such as the Internet, a buffer memory (515) may also be required, which may be relatively large and may have an adaptive size, and may be implemented at least partially in an operating system or a similar element (not shown) external to the video decoder (510).

[0062] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequences. The categories of these symbols include information for managing the operation of the video decoder (510), and potential information for controlling a display device (512) (e.g., a display screen), such as a display device that is not part of the electronic device (530), but may be coupled to the electronic device (530), such as Figure 5As shown in. The control information for the display device may be a parameter set segment (not labeled) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be performed according to video coding techniques or standards and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so on. The parser (520) may extract subgroup parameter sets for at least one subgroup of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the group. The subgroups may include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, Coding Unit (CU), blocks, Transform Unit (TU), Prediction Unit (PU), and so on. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0063] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0064] Depending on the type of the encoded video picture or a part of the encoded video picture (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of the symbols (521) may involve multiple different units. Which units are involved and the way they are involved may be controlled by the subgroup control information parsed by the parser (520) from the encoded video sequence. For the sake of brevity, such subgroup control information flows between the parser (520) and multiple units below are not described.

[0065] In addition to the functional blocks already mentioned, the video decoder (510) may be conceptually divided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and may be integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually divide into the functional units below.

[0066] The first unit is a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives the quantized transform coefficients as symbols (521) and control information from the parser (520), including which transform mode to use, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (551) can output a block including sample values, and the sample values can be input into the aggregator (555).

[0067] In some cases, the output samples of the scaler / inverse transform unit (551) can belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed pictures but can use predictive information from previously reconstructed parts of the current picture. Such predictive information can be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates surrounding blocks of the same size and shape as the block being reconstructed using the reconstructed information extracted from the current picture buffer (558). For example, the current picture buffer (558) buffers the partially reconstructed current picture and / or the fully reconstructed current picture. In some cases, the aggregator (555) adds the predictive information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) based on each sample.

[0068] In other cases, the output samples of the scaler / inverse transform unit (551) can belong to inter-coded and potentially motion-compensated blocks. In this case, the motion compensation prediction unit (553) can access the reference picture memory (557) to extract samples for prediction. After motion compensation of the extracted samples according to the symbol (521), these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (which is called the residual sample or residual signal in this case), thereby generating output sample information. The motion compensation prediction unit (553) obtaining the prediction samples from the address in the reference picture memory (557) can be controlled by a motion vector, and the motion vector is in the form of the symbol (521) for use by the motion compensation prediction unit (553), and the symbol (521) includes, for example, X, Y, and reference picture components. Motion compensation can also include interpolation of the sample values extracted from the reference picture memory (557) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, and so on.

[0069] The output samples of the aggregator (555) can be employed by various loop filtering techniques in the loop filter unit (556). Video compression techniques can include in-loop filter techniques that are controlled by parameters included in an encoded video sequence (also referred to as an encoded video bitstream), and the parameters can be used by the loop filter unit (556) as symbols (521) from the parser (520). However, in other embodiments, video compression techniques can also respond to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or an encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0070] The output of the loop filter unit (556) can be a sample stream that can be output to the display device (512) and stored in the reference picture memory (557) for subsequent inter-picture prediction.

[0071] Once fully reconstructed, certain encoded pictures can be used as reference pictures for future prediction. For example, once the encoded picture corresponding to the current picture is fully reconstructed and the encoded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before starting to reconstruct subsequent encoded pictures.

[0072] The video decoder (510) can perform decoding operations according to, for example, a predetermined video compression technique in the ITU-T H.265 standard. In the sense that an encoded video sequence conforms to the syntax specified by the video compression technique or standard used, the encoded video sequence can comply with the syntax of the video compression technique or standard. Specifically, a profile can select certain tools from all the tools available in the video compression technique or standard as the only tools available under that profile. For compliance, it is also required that the complexity of the encoded video sequence be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level can be further defined by the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the encoded video sequence.

[0073] In an embodiment, a receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by a video decoder (510) to decode the data appropriately and / or reconstruct the original video data more accurately. The additional data may be in the form of, for example, a temporal, spatial, or signal noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, forward error correction codes, and the like.

[0074] Figure 6 is a block diagram of a video encoder (603) according to an embodiment disclosed in the present application. The video encoder (603) is provided in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace Figure 4 the video encoder (403) in the embodiment.

[0075] The video encoder (603) may receive video samples from a video source (601) (which is not Figure 6 part of the electronic device (620) in the embodiment), and the video source may capture video images to be encoded by the video encoder (603). In another embodiment, the video source (601) is part of the electronic device (620).

[0076] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (603), and the digital video sample stream may have any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits...), any color space (e.g., BT.601 Y CrCB, RGB...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (601) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures, which are given motion when viewed in sequence. The pictures themselves may be constructed as a spatial pixel array, where each pixel may include one or more samples depending on the sampling structure, color space, etc. used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0077] According to an embodiment, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by an application. Implementing an appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to these units. For the sake of brevity, the couplings are not labeled in the figures. Parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be used for other suitable functions that relate to optimizing the video encoder (603) for a certain system design.

[0078] In some embodiments, the video encoder (603) operates in an encoding loop. As a simple description, in an embodiment, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols in a manner similar to how a (remote) decoder creates sample data to create sample data (since in the video compression techniques contemplated in this application, any compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input into the reference picture memory (634). Since the decoding of the symbol stream produces a bit-exact result independent of the decoder location (local or remote), the content in the reference picture memory (634) is also bit-exact corresponding between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same as the sample values that the decoder will "see" when using the prediction during decoding. This reference picture synchronization principle (and the drift that occurs, for example, when synchrony cannot be maintained due to channel errors) is also used in some related technologies.

[0079] The operation of the "local" decoder (633) may be the same as that of the "remote" decoder that has been described in detail above in connection with Figure 5 the video decoder (510). However, briefly referring additionally to Figure 5 , when symbols are available and the entropy encoder (645) and the parser (520) can encode / decode the symbols losslessly into an encoded video sequence, the entropy decoding part of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0080] At this point, it can be observed that any decoder technology other than parsing / entropy decoding existing in the decoder must also exist in the corresponding encoder in substantially the same functional form. For this reason, the present application focuses on decoder operations. The description of encoder technology can be simplified because the encoder technology is reciprocal to the decoder technology described comprehensively. A more detailed description is only required in certain areas and is provided below.

[0081] During operation, in some embodiments, the source encoder (630) may perform motion compensation predictive coding. With reference to one or more previously encoded pictures designated as "reference pictures" in the video sequence, the motion compensation predictive coding performs predictive coding on the input picture. In this way, the coding engine (632) encodes the difference between the pixel blocks of the input picture and the pixel blocks of the reference picture, and the reference picture can be selected as the prediction reference for the input picture.

[0082] The local video decoder (633) may decode the encoded video data that can be designated as a reference picture based on the symbols created by the source encoder (630). The operation of the coding engine (632) may be a lossy process. When the encoded video data can be decoded at the video decoder ( Figure 6 not shown), the reconstructed video sequence is generally a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that can be performed by the video decoder on the reference picture and may store the reconstructed reference picture in the reference picture cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture, which has the same content (in the absence of transmission errors) as the reconstructed reference picture to be obtained by the remote video decoder.

[0083] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be encoded, the predictor (635) may search in the reference picture memory (634) for sample data (as candidate reference pixel blocks) or some metadata, such as reference picture motion vectors, block shapes, etc., that can be used as an appropriate prediction reference for the new picture. The predictor (635) may operate on a per-pixel block basis of the sample blocks to find a suitable prediction reference. In some cases, according to the search results obtained by the predictor (635), it can be determined that the input picture may have a prediction reference obtained from multiple reference pictures stored in the reference picture memory (634).

[0084] The controller (650) may manage the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding the video data.

[0085] The outputs of all the above functional units can be entropy encoded in an entropy encoder (645). The entropy encoder (645) losslessly compresses the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into an encoded video sequence.

[0086] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) to prepare for transmission over a communication channel (660), which can be a hardware / software link to a storage device that will store the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0087] The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a certain encoded picture type to each encoded picture, but this may affect the encoding techniques applicable to the corresponding picture. For example, a picture can typically be assigned to any of the following picture types:

[0088] An intra picture (I picture), which can be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of the variants of I pictures and their corresponding applications and characteristics.

[0089] A predictive picture (P picture), which can be a picture that can be encoded and decoded using intra prediction or inter prediction, where the intra prediction or inter prediction uses at most one motion vector and a reference index to predict the sample values of each block.

[0090] A bi - predictive picture (B picture), which can be a picture that can be encoded and decoded using intra prediction or inter prediction, where the intra prediction or inter prediction uses at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstructing a single block.

[0091] Source pictures can typically be spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and encoded block by block. These blocks can be prediction-encoded with reference to other (already encoded) blocks, and the other blocks are determined according to the encoding assignment of the corresponding picture applied to the block. For example, blocks of an I picture can be non-prediction-encoded, or the blocks can be prediction-encoded with reference to already encoded blocks of the same picture (spatial prediction or intra-frame prediction). Pixel blocks of a P picture can be prediction-encoded by spatial prediction or by temporal prediction with reference to a previously encoded reference picture. Blocks of a B picture can be prediction-encoded by spatial prediction or by temporal prediction with reference to one or two previously encoded reference pictures.

[0092] The video encoder (603) can perform encoding operations according to a predetermined video encoding technique or standard such as the ITU-T H.265 recommendation. In operation, the video encoder (603) can perform various compression operations, including prediction encoding operations that utilize the temporal and spatial redundancies in the input video sequence. Thus, the encoded video data can conform to the syntax specified by the video encoding technique or standard used.

[0093] In an embodiment, the transmitter (640) can transmit additional data when transmitting the encoded video. The source encoder (630) can include such data as part of the encoded video sequence. The additional data can include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0094] The captured video can be a plurality of source pictures (video pictures) in a time series. Intra-picture prediction (often simplified to intra-frame prediction) utilizes the spatial correlation in a given picture, while inter-picture prediction utilizes the (temporal or other) correlation between pictures. In an embodiment, the particular picture being encoded / decoded is segmented into blocks, and the particular picture being encoded / decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a reference picture that has been previously encoded and is still buffered in the video, the block in the current picture can be encoded by a vector called a motion vector. The motion vector points to the reference block in the reference picture, and in the case of using multiple reference pictures, the motion vector can have a third dimension identifying the reference picture.

[0095] In some embodiments, bidirectional prediction techniques can be used in inter - picture prediction. According to the bidirectional prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture that are both before the current picture in the decoded order (but may be past and future respectively in the display order). A block in the current picture can be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Specifically, the block can be predicted by a combination of the first reference block and the second reference block.

[0096] In addition, merge mode techniques can be used in inter - picture prediction to improve coding efficiency.

[0097] According to some embodiments disclosed in the present application, predictions such as inter - picture prediction and intra - picture prediction are performed on a block - by - block basis. For example, according to the HEVC standard, a picture in a video picture sequence is segmented into coding tree units (CTUs) for compression. CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), namely one luminance CTB and two chrominance CTBs. Further, each CTU can be split into one or more coding units (CUs) in a quadtree manner. For example, a 64×64 - pixel CTU can be split into a 64×64 - pixel CU, or 4 32×32 - pixel CUs, or 16 16×16 - pixel CUs. In an embodiment, each CU is analyzed to determine the prediction type for the CU, such as an inter - frame prediction type or an intra - frame prediction type. In addition, depending on temporal and / or spatial predictability, the CU is split into one or more prediction units (PUs). Generally, each PU includes a luminance prediction block (PB) and two chrominance PBs. In an embodiment, prediction operations in encoding (encoding / decoding) are performed on a prediction - block basis. Taking the luminance prediction block as an example of the prediction block, the prediction block includes a matrix of pixel values (such as luminance values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and so on.

[0098] Figure 7 is a diagram of a video encoder (703) according to another embodiment disclosed in the present application. The video encoder (703) is configured to receive sample values in a processing block (such as a prediction block) within a current video picture in a video picture sequence and encode the processing block into an encoded picture that is part of an encoded video sequence. In this embodiment, the video encoder (703) is used to replace Figure 4The video encoder (403) in the embodiment.

[0099] In HEVC embodiments, the video encoder (703) receives a matrix of sample values for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) uses, for example, rate-distortion (RD) optimization to determine whether to use an intra mode, an inter mode, or a bi-prediction mode to encode the processing block. When encoding a processing block in the intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded picture; and when encoding a processing block in the inter mode or the bi-prediction mode, the video encoder (703) may use inter prediction or bi-prediction techniques respectively to encode the processing block into an encoded picture. In some video coding techniques, the merge mode may be an inter-picture prediction sub-mode, in which a motion vector is derived from one or more motion vector prediction values without resorting to encoded motion vector components external to the prediction values. In some other video coding techniques, there may be motion vector components applicable to the subject block. In an embodiment, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0100] In Figure 7 the embodiments of Figure 7 shown, the video encoder (703) includes an inter encoder (730), an intra encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as

[0101] The inter encoder (730) is configured to receive samples of a current block (such as a processing block), compare the block with one or more reference blocks in a reference picture (such as blocks in a previous picture and a subsequent picture), generate inter prediction information (such as a redundancy information description according to inter coding techniques, a motion vector, merge mode information), and calculate an inter prediction result (such as a predicted block) based on the inter prediction information using any suitable technique. In some embodiments, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0102] The intra encoder (722) is configured to receive samples of a current block (such as a processing block), compare the block with encoded blocks in the same picture in some cases, generate quantization coefficients after transformation, and also generate intra prediction information in some cases (such as intra prediction direction information according to one or more intra coding techniques). In an embodiment, the intra encoder (722) also calculates an intra prediction result (such as a predicted block) based on the intra prediction information and reference blocks in the same picture.

[0103] The general controller (721) is used to determine general control data and control other components of the video encoder (703) based on the general control data. In an embodiment, the general controller (721) determines the mode of a block and provides a control signal to the switch (726) based on the mode. For example, when the mode is the intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select the intra prediction information and add the intra prediction information to the bitstream; and when the mode is the inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select the inter prediction information and add the inter prediction information to the bitstream.

[0104] The residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) is used to operate based on the residual data to encode the residual data to generate transform coefficients. In an embodiment, the residual encoder (724) is used to convert the residual data from the time domain to the frequency domain and generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) further includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and the intra prediction information. The decoded block is appropriately processed to generate a decoded picture, and in some embodiments, the decoded picture can be buffered in a memory circuit (not shown) and used as a reference picture.

[0105] The entropy encoder (725) is used to format the bitstream to produce an encoded block. The entropy encoder (725) generates various information according to a suitable standard such as the HEVC standard. In an embodiment, the entropy encoder (725) is used to obtain general control data, the selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other suitable information in the bitstream. It should be noted that according to the disclosed subject matter, there is no residual information when encoding a block in the merge submode of the inter mode or the bi - directional prediction mode.

[0106] Figure 8FIG. is a diagram of a video decoder (810) according to another embodiment disclosed in the present application. The video decoder (810) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (810) is configured to replace Figure 4 the video decoder (410) in the embodiment.

[0107] In Figure 8 the embodiment, the video decoder (810) includes an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together as shown in Figure 8 .

[0108] The entropy decoder (871) can be used to reconstruct certain symbols based on the encoded picture, and these symbols represent the syntax elements that make up the encoded picture. Such symbols may include, for example, the mode used to encode the block (e.g., intra-frame mode, inter-frame mode, bi-prediction mode, merge sub-mode of the latter two, or another sub-mode), prediction information that can respectively identify certain samples or metadata for the intra-frame decoder (872) or the inter-frame decoder (880) to perform prediction (e.g., intra-frame prediction information or inter-frame prediction information), residual information in the form of, for example, quantized transform coefficients, and so on. In an embodiment, when the prediction mode is an inter-frame or bi-prediction mode, the inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is an intra-frame prediction type, the intra-frame prediction information is provided to the intra-frame decoder (872). The residual information can be inverse quantized and provided to the residual decoder (873).

[0109] The inter-frame decoder (880) is configured to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0110] The intra-frame decoder (872) is configured to receive the intra-frame prediction information and generate a prediction result based on the intra-frame prediction information.

[0111] The residual decoder (873) is configured to perform inverse quantization to extract the dequantized transform coefficients and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to obtain the quantizer parameter QP), and this information can be provided by the entropy decoder (871) (the data path is not labeled because this is only low-volume control information).

[0112] The reconstruction module (874) is used to combine, in the spatial domain, the residual output by the residual decoder (873) with the prediction result (which can be output by an inter-frame prediction module or an intra-frame prediction module) to form a reconstructed block, and the reconstructed block can be part of a reconstructed picture, and the reconstructed picture can in turn be part of a reconstructed video. It should be noted that other suitable operations such as deblocking operations can be performed to improve the visual quality.

[0113] It should be noted that any suitable technology can be used to implement the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810). In an embodiment, one or more integrated circuits can be used to implement the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810). In another embodiment, one or more processors executing software instructions can be used to implement the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810).

[0114] Aspects of the present disclosure provide temporal filtering techniques that can be applied to various filters, such as an adaptive loop filter (ALF), a cross component filter (CCF), a loop restoration filter, an in loop constrained directional enhanced filter (CDEF), etc.

[0115] An adaptive loop filter (ALF) with block-based filter adaptation can be applied to an encoder / decoder to reduce artifacts. For the luminance component, for example, one of a plurality of filters (e.g., 25 filters) can be selected for a 4×4 luminance block based on the direction and activity of the local gradient.

[0116] The ALF can have any suitable shape and size. Refer to Figure 9, ALF(910) and ALE(911) have a rhombus shape. For example, ALF(910) is a 5×5 rhombus, and ALF(911) is a 7×7 rhombus. In ALF(910), elements (920) to (932) form this rhombus shape and can be used for the filtering process. Seven values (e.g., C0 - C6) can be used for elements (920)-(932). In ALF(911), elements (940)-(964) form this rhombus shape and can be used for the filtering process. Thirteen values (e.g., C0 - C12) can be used for elements (940)-(964).

[0117] Reference Figure 9 , in some examples, these two ALF(910) and ALE(911) with rhombus filter shapes are used. The 5×5 rhombus filter (910) can be applied to the chrominance component (e.g., chrominance block, chrominance CB), and the 7×7 rhombus filter (911) can be applied to the luminance component (e.g., luminance block, luminance CB). Other suitable shapes and sizes can be used in the ALF. For example, a 9×9 rhombus filter can be used.

[0118] The filter coefficients at the positions indicated by these values (e.g., C0 - C6 in (910) or C0 - C12 in (911)) can be non - zero. Additionally, when the ALF includes a clipping function, the clipping values at these positions can be non - zero.

[0119] For the block classification of the luminance component, a 4×4 block (or luminance block, luminance CB) can be classified or grouped into one of multiple (e.g., 25) categories. The classification index C can be derived using Equation (1) based on the quantization values of the directionality parameter D and the activity value A to be derived.

[0120]

[0121] To calculate the directionality parameter D and the quantization value the gradients g v , g h ,, g d1 and g d2 in the vertical, horizontal, and two diagonal directions (e.g., d1 and d2) can be calculated respectively using a 1 - D Laplacian operator as follows:

[0122]

[0123]

[0124]

[0125]

[0126] Where the indices i and j represent the coordinates of the upper left sample within a 4×4 block, and R(k, l) represents the reconstructed sample at the coordinates (k, l). Directions (e.g., d1 and d2) can represent two diagonal directions.

[0127] To reduce the complexity of the above block classification, a subsampled 1-D Laplacian operator can be applied. Figures 10A - 10D Examples of subsampling positions for calculating the gradients g Figure 10A in the vertical direction ( Figure 10B ), horizontal direction ( Figure 10C ), and two diagonal directions d1 ( Figure 10D ) and d2 ( v ), g h ), g d1 ), and g d2 ) are shown. The same subsampling positions can be used for gradient calculations in different directions. In Figure 10A , the label "V" shows the subsampling positions for calculating the vertical gradient g v . In Figure 10B , the label "H" shows the subsampling positions for calculating the horizontal gradient g h . In Figure 10C , the label "D1" shows the subsampling positions for calculating the d1 diagonal gradient g d1 . In Figure 10D , the label "D2" shows the subsampling positions for calculating the d2 diagonal gradient g d2 .

[0128] The maximum v g and minimum h g of the horizontal and vertical gradients and can be set to:

[0129]

[0130] The maximum d1 g and minimum d2 Xg of the two diagonal gradients and can be set to:

[0131]

[0132] The directionality parameter D can be derived based on the above values and two thresholds t1 and t2 as follows:.

[0133] Step 1: If (1) and (2) If it is true, then D is set to 0.

[0134] Step 2: If then proceed to Step 3; otherwise proceed to Step 4.

[0135] Step 3: If then D is set to 2; otherwise D is set to 1.

[0136] Step 4: If then D is set to 4; otherwise D is set to 3.

[0137] The activity value A can be calculated as:

[0138]

[0139] A can be further quantized to the range from 0 to 4 (including 0 and 4), and the quantized value is represented as

[0140] Block classification is not applied to the chrominance components in the picture, so a single set of ALF coefficients can be applied to each chrominance component.

[0141] Geometric transformations can be applied to the filter coefficients and the corresponding filter clipping values (also known as clipping values). Before filtering a block (e.g., a 4×4 luma block), geometric transformations (such as rotation, or diagonal and vertical flipping) can be applied to the filter coefficients f(k, l) and the corresponding filter clipping values c(k, l) according to the gradient values calculated for the block (e.g., g v , g h , g d1 and / or g d2 ). The geometric transformation applied to the filter coefficients f(k, l) and the corresponding filter clipping values c(k, l) can be equivalent to applying the geometric transformation to the samples in the region supported by the filter. By aligning the corresponding directions, the geometric transformation can make different blocks to which ALF is applied more similar.

[0142] As described in equations (9)-(11), three geometric transformations can be performed separately, including diagonal flipping, vertical flipping, and rotation.

[0143] f D (k, l) = f(l, k),, c D (k, l) = c(l, k), Equation (9)

[0144] f V (k, l) = f(k, K - l - 1), c V (k, l) = c(k, K - l - 1) Equation (10)

[0145] f R(k, l) = f(K - l - 1, k), c R (k, l) = c(K - l - 1, k) Equation (11)

[0146] Where K is the size of the ALF or filter, and 0 ≤ k, l ≤ K - 1 are the coordinates of the coefficients. For example, the position (0, 0) is at the upper left corner of the filter f or the clipping value matrix (or clipping matrix) c, and the position (K - l, K - 1) is at the lower right corner. Based on the gradient values calculated for the block, the transformation can be applied to the filter coefficient f(k, l) and the clipping value c(k, l). An example of the relationship between the transformation and the four gradients is summarized in Table 1.

[0147] Table 1: Mapping of Gradients Calculated for Blocks and Transformations

[0148] Gradient value Transformation <![CDATA[g d2 <g d1 and g h <g v > No transformation <![CDATA[g d2 <g d1 and g v <g h > Diagonal flip <![CDATA[g d1 <g d2 and g h <g v > Vertical flip <![CDATA[g d1 <g d2 and g v <g h > Rotation

[0149] In some embodiments, the ALF filter parameters are signaled in the Adaptive Parameter Set (APS) of the picture. In the APS, one or more sets (e.g., up to 25 sets) of luminance filter coefficients and clipping value indices can be signaled. In an example, one of the one or more sets can include luminance filter coefficients and one or more clipping value indices. One or more sets (e.g., up to 8 sets) of chrominance filter coefficients and clipping value indices can be signaled. To reduce signaling overhead, the filter coefficients for different classifications (e.g., with different classification indices) used for the luminance component can be merged. In the slice header, the index of the APS for the current slice can be signaled.

[0150] In an embodiment, the clipping value index (also referred to as the clipping index) can be decoded from the APS. The clipping value index can be used, for example, to determine the corresponding clipping value based on the relationship between the clipping value index and the corresponding clipping value. This relationship can be predefined and stored in the decoder. In an example, this relationship is described by a table, such as a luminance table of the clipping value index and the corresponding clipping value (e.g., for luminance CB), a chrominance table of the clipping value index and the corresponding clipping value (e.g., for chrominance CB). The clipping value can depend on the bit depth B. The bit depth B can refer to the internal bit depth, the bit depth of the reconstructed samples in the CB to be filtered, etc. In some examples, Equation (12) is used to obtain the table (e.g., luminance table, chrominance table).

[0151]

[0152] Among them, AlfClip is the clipping value, B is the bit depth (e.g., bitDepth), N (e.g., N = 4) is the number of allowed clipping values, and (n - 1) is the clipping value index (also referred to as the clipping index or clipIdx). Table 2 shows an example of a table obtained using equation (12) in the case of N = 4. The clipping index (n - 1) can be 0, 1, 2, and 3 in Table 2, and n can be 1, 2, 3, and 4 respectively. Table 2 can be used for luminance blocks or chrominance blocks.

[0153] Table 2: AlfClip can depend on the bit depth B and clipIdx

[0154]

[0155] In the slice header of the current slice, one or more APS indices (e.g., up to 7 APS indices) can be signaled to specify the luminance filter bank available for the current slice. The filtering process can be controlled at one or more appropriate levels (e.g., picture level, slice level, CTB level, etc.). In an embodiment, the filtering process can be further controlled at the CTB level. A flag can be signaled to indicate whether ALF is applied to the luminance CTB. The luminance CTB can select a filter bank from multiple fixed filter banks (e.g., 16 fixed filter banks) and the filter bank signaled in the APS (also referred to as the signaled filter bank). A filter bank index for the luminance CTB can be signaled to indicate the filter bank to be applied (e.g., the filter bank among multiple fixed filter banks and the signaled filter bank). The multiple fixed filter banks can be predefined and hard-coded in the encoder and decoder, and can be referred to as predefined filter banks.

[0156] For the chrominance component, an APS index can be signaled in the slice header to indicate the chrominance filter bank to be used for the current slice. At the CTB level, if there are multiple chrominance filter banks in the APS, a filter bank index can be signaled for each chrominance CTB.

[0157] The filter coefficients can be quantized using a norm equal to 128. To reduce the multiplication complexity, bitstream consistency can be applied such that the coefficient values at non-central positions can be in the range of -2^7 to 2^7 - 1 (including -2^7 and 2^7 - 1). In an example, the coefficient at the central position is not signaled in the bitstream but is considered to be equal to 128.

[0158] In some embodiments, the syntax and semantics of the clipping index and clipping value are defined as follows:

[0159] alf_luma_clip_idx[sfIdx][j] is used to specify the clipping index of the clipping value to be used before multiplying the j-th coefficient of the luminance filter signaled by sfIdx. The requirements for bitstream consistency may include: the value of alf_luma_clip_idx[sfIdx][j] shall be in the range of 0 to 3 (including 0 and 3), where sfIdx = 0 to alf_luma_num_filters_signalled_minus1 and j = 0 to 11.

[0160] As specified in Table 2, the luminance filter clipping value AlfClipL[adaptation_parameter_set_id][filtIdx][j] with element AlfClipL[adaptation_parameter_set_id] can be derived according to the bitDepth set equal to BitDepthY and the clipIdx set equal to alf_luma_clip_idx[alf_luma_coeff_delta_idx[filtIdx]][j], where filtIdx = 0 to NumAlfFilters - 1 and j = 0 to 11.

[0161] alf_chroma_clip_idx[altIdx][j] is used to specify the clipping index of the clipping value to be used before multiplying the j-th coefficient of the alternative chrominance filter indexed by altIdx. The requirements for bitstream consistency may include: the value of alf_chroma_clip_idx[altIdx][j] shall be in the range of 0 to 3 (including 0 and 3), where altIdx = 0 to alf_chroma_num_alt_filters_minus1 and j = 0 to 5.

[0162] As specified in Table 2, the chrominance filter clipping value AlfClipC[adaptation_parameter_set_id][altIdx][j] with element AlfClipC[adaptation_parameter_set_id][altIdx] can be derived according to the bitDepth set equal to BitDepthC and the clipIdx set equal to alf_chroma_clip_idx[altIdx][j], where altIdx = 0 to alf_chroma_num_alt_filters_minus1 and j = 0 to 5.

[0163] In an embodiment, the filtering process can be described as follows. On the decoder side, when ALF is enabled for a CTB, the samples R(i, j) in a CU (or CB) can be filtered to generate a filtered sample value R'(i, j), which is shown using Equation (13) as follows. In an example, each sample in the CU is filtered.

[0164] R′(i,j) = R(i,j)+((∑ k≠0 ∑ l≠0 f(k,l)×K(R(i + k,j + l)-R(i,j),c(k,l)) + 64) >> 7) Equation (13)

[0165] Where f(k, l) represents the decoded filter coefficient, K(x, y) is a clipping function, and c(k, l) represents the decoded clipping parameter (or clipping value). The variables k and l can vary between -L / 2 and L / 2, where L represents the filter length. The clipping function K(x, y) = min(y, max(-y, x)) corresponds to the clipping function Clip3(-y, y, x). By incorporating the clipping function K(x, y), the loop filtering method (e.g., ALF) becomes a non - linear process and can be referred to as non - linear ALF.

[0166] In non - linear ALF, multiple sets of clipping values can be provided in Table 3. In an example, the luminance set includes four clipping values {1024, 181, 32, 6}, and the chrominance set includes 4 clipping values {1024, 161, 25, 4}. The four clipping values in the luminance set can be selected by roughly equally dividing the entire range (e.g., 1024) of the sample values (encoded in 10 bits) of the luminance block in the logarithmic domain. The range of the chrominance set can range from 4 to 1024.

[0167] Table 3: Examples of Clipping Values

[0168]

[0169]

[0170] The selected clipping values can be encoded in the "alf_data" syntax element as follows: The clipping index corresponding to the selected clipping values (as shown in Table 3) can be encoded using an appropriate coding scheme (e.g., Golomb coding scheme). The coding scheme can be the same as the coding scheme used for encoding the filter bank index.

[0171] In an embodiment, a virtual boundary filtering process may be used to reduce the line buffer requirements of the ALF. Accordingly, modified block classification and filtering may be used for samples near a CTU boundary (e.g., a horizontal CTU boundary). By shifting the horizontal CTU boundary (1120) by “N samples ” samples, a virtual boundary (1130) may be defined as a line, as Figure 11A shown, where N samples may be a positive integer. In an example, for the luminance component, N samples equals 4, and for the chrominance component, N samples equals 2.

[0172] Referring to Figure 11A , the modified block classification may be applied to the luminance component. In an example, for the 1-D Laplacian gradient calculation of a 4×4 block (1110) above the virtual boundary (1130), only the samples above the virtual boundary (1130) are used. Similarly, referring to Figure 11B , for the 1-D Laplacian gradient calculation of a 4×4 block (1111) below the virtual boundary (1131) shifted from the CTU boundary (1121), only the samples below the virtual boundary (1131) are used. By considering that a reduced number of samples are used in the 1-D Laplacian gradient calculation, the quantization of the activity value A may be scaled accordingly.

[0173] For the filtering process, a symmetric padding operation at the virtual boundary may be used for both the luminance and chrominance components. Figures 12A - 12F An example of such modified ALF filtering at the virtual boundary for the luminance component is shown. When the sample to be filtered is below the virtual boundary, adjacent samples above the virtual boundary may be padded. When the sample to be filtered is above the virtual boundary, adjacent samples below the virtual boundary may be padded. Referring to Figure 12A , the adjacent sample C0 may be padded with the sample C2 below the virtual boundary (1210). Referring to Figure 12B , the adjacent sample C0 may be padded with the sample C2 above the virtual boundary (1220). Referring to Figure 12C , the adjacent samples C1-C3 may be padded with the samples C5-C7 below the virtual boundary (1230), respectively. Referring to Figure 12D , the adjacent samples C1-C3 may be padded with the samples C5-C7 above the virtual boundary (1240), respectively. Referring to Figure 12E , the adjacent samples C4-C8 may be padded with the samples C10, C11, C12, C11, and C10 below the virtual boundary (1250), respectively. Referring to Figure 12F, the adjacent samples C4 - C8 can be filled with samples C10, C11, C12, C11, and C10 respectively, which are located above the virtual boundary (1260).

[0174] In some examples, when the sample and the adjacent sample are on the left (or right) and right (or left) sides of the virtual boundary, the above description can be appropriately modified.

[0175] According to aspects of the present disclosure, to improve the encoding and decoding efficiency, the picture can be partitioned based on the filtering process. In some examples, the CTU is also referred to as the largest coding unit (LCU). In an example, the CTU or LCU can have a size of 64×64 pixels. In some embodiments, LCU - aligned picture quadtree segmentation can be used for filtering - based partitioning. In some examples, an adaptive loop filter based on the coding unit - synchronized picture quadtree can be used. For example, the luminance picture can be segmented into several multi - level quadtree partitions, and each partition boundary is aligned with the boundary of the LCU. Each partition has its own filtering process and is thus referred to as a filter unit (FU).

[0176] In some examples, a 2 - pass encoding stream can be used. At the first pass of the 2 - pass encoding stream, the quadtree segmentation pattern of the picture and the optimal filter for each FU can be determined. In some embodiments, the determination of the quadtree segmentation pattern of the picture and the determination of the optimal filter for the FU are based on the filtering distortion. The filtering distortion can be estimated by the fast filtering distortion estimation (FFDE) technique during the determination process. The picture is partitioned using the quadtree partition. According to the determined quadtree segmentation pattern and the filters of all the selected FUs, the reconstructed picture can be filtered.

[0177] At the second pass of the 2 - pass encoding stream, CU - synchronized ALF on / off control is performed. According to the ALF on / off result, the first filtered picture is partially restored from the reconstructed picture.

[0178] Specifically, in some examples, a top - down segmentation strategy is adopted to divide the picture into multi - level quadtree partitions by using the rate - distortion criterion. Each partition is called a filter unit (FU). The segmentation process aligns the quadtree partition with the LCU boundary. The encoding order of the FUs follows the z - scan order.

[0179] Figure 13 Illustrates a partitioning example according to some embodiments of the present disclosure. In Figure 13 the example, the picture (1300) is segmented into 10 FUs, and the encoding order is FU0, FU1, FU2, FU,3, FU4, FU5, FU6, FU7, FU8, and FU9.

[0180] Figure 14Shows a quadtree segmentation pattern (1400) for a picture (1300). In Figure 14 the example of Figure 14 , segmentation flags are used to indicate the picture partitioning pattern. For example, "1" indicates that quadtree partitioning is performed on the block, and "0" indicates that the block is not further partitioned. In some examples, the minimum size FU has the LCU size, and the minimum size FU does not require a segmentation flag. As Figure 14 shown, the segmentation flags are encoded and sent in z-order.

[0181] In some examples, filters for each FU are selected from two filter banks based on a rate-distortion criterion. The first group has 1 / 2 symmetric square filters and diamond filters derived for the current FU. The second group comes from a delay filter buffer that stores filters previously derived for FUs of existing pictures. The filter with the minimum rate-distortion cost in these two groups can be selected for the current FU. Similarly, if the current FU is not the minimum FU and can be further divided into 4 sub-FUs, the rate-distortion costs of these 4 sub-FUs are calculated. By recursively comparing the rate-distortion costs in the divided and non-divided cases, the picture quadtree segmentation pattern can be determined.

[0182] In some examples, the maximum quadtree segmentation level can be used to limit the maximum number of FUs. In the example, when the maximum quadtree segmentation level is 2, the maximum number of FUs is 16. Additionally, during the process of quadtree segmentation determination, relevant values of Wiener coefficients used to derive 16 FUs at the bottom quadtree level (minimum FUs) can be reused. The remaining FUs can derive their Wiener filters from the correlations of these 16 FUs at the bottom quadtree level. Therefore, in this example, only one frame buffer access is performed to derive the filter coefficients of all FUs.

[0183] After determining the quadtree segmentation pattern, to further reduce filtering distortion, CU synchronous ALF on / off control can be performed. By comparing the filtering distortion and non-filtering distortion at each leaf CU, the leaf CU can explicitly switch the ALF on / off in its local area. In some examples, by redesigning the filter coefficients according to the ALF on / off result, the codec efficiency can be further improved.

[0184] The cross-component filtering process can apply a cross-component filter, such as a cross-component adaptive loop filter (CC-ALF). The cross-component filter can use the luminance sample values of the luminance component (e.g., luminance CB) to correct the chrominance component (e.g., chrominance CB corresponding to luminance CB). In the example, luminance CB and chrominance CB are included in the CU.

[0185] Figure 15Shows a cross-component filter (e.g., CC-ALF) for generating chrominance components according to an embodiment of the present disclosure. In some examples, Figure 15 Shows a filtering process for a first chrominance component (e.g., first chrominance CB), a second chrominance component (e.g., second chrominance CB), and a luminance component (e.g., luminance CB). The luminance component can be filtered by a sample adaptive offset (SAO) filter (1510) to generate a SAO-filtered luminance component (1541). The SAO-filtered luminance component (1541) can be further filtered by an ALF luminance filter (1516) to become a filtered luminance CB (1561) (e.g., “Y”).

[0186] The first chrominance component can be filtered by a SAO filter (1512) and an ALF chrominance filter (1518) to generate a first intermediate component (1552). In addition, the SAO-filtered luminance component (1541) can be filtered by a cross-component filter (1521) (e.g., CC-ALF) for the first chrominance component to generate a second intermediate component (1542). Subsequently, a filtered first chrominance component (1562) (e.g., “Cb”) can be generated based on at least one of the second intermediate component (1542) and the first intermediate component (1552). In an example, the filtered first chrominance component (1562) (e.g., “Cb”) can be generated by combining the second intermediate component (1542) and the first intermediate component (1552) using an adder (1522). The cross-component adaptive loop filtering process for the first chrominance component can include steps performed by the CC-ALF (1521) and steps performed by, for example, the adder (1522).

[0187] The above description can be applied to the second chrominance component. The second chrominance component can be filtered by a SAO filter (1514) and an ALF chrominance filter (1518) to generate a third intermediate component (1553). In addition, the SAO-filtered luminance component (1541) can be filtered by a cross-component filter (1531) (e.g., CC-ALF) for the second chrominance component to generate a fourth intermediate component (1543). Subsequently, a filtered second chrominance component (1563) (e.g., “Cr”) can be generated based on at least one of the fourth intermediate component (1543) and the third intermediate component (1553). In an example, the filtered second chrominance component (1563) (e.g., “Cr”) can be generated by combining the fourth intermediate component (1543) and the third intermediate component (1553) using an adder (1532). In an example, the cross-component adaptive loop filtering process for the second chrominance component can include steps performed by the CC-ALF (1531) and steps performed by, for example, the adder (1532).

[0188] Cross-component filters (e.g., CC-ALF(1521), CC-ALF(1531)) can operate by applying a linear filter with any suitable filter shape to the luminance component (or luminance channel) to correct each chrominance component (e.g., the first chrominance component, the second chrominance component).

[0189] Figure 16 An example of a filter (1600) according to an embodiment of the present disclosure is shown. The filter (1600) can include non-zero filter coefficients and zero filter coefficients. The filter (1600) has a diamond shape (1620) formed by filter coefficients (1610) (represented by circles filled with black). In the example, the non-zero filter coefficients in the filter (1600) are included in the filter coefficients (1610), and the filter coefficients not included in the filter coefficients (1610) are zero. Thus, the non-zero filter coefficients in the filter (1600) are included in the diamond shape (1620), and the filter coefficients not included in the diamond shape (1620) are zero. In the example, the number of filter coefficients of the filter (1600) is equal to the number of filter coefficients (1610), which is Figure 16 18 in the example shown.

[0190] CC-ALF can include any suitable filter coefficients (also referred to as CC-ALF filter coefficients). Referring again to Figure 15 , CC-ALF(1521) and CC-ALF(1531) can have the same filter shape, such as Figure 16 the diamond shape (1620) shown in

[0191] and have the same number of filter coefficients. In the example, the values of the filter coefficients in CC-ALF(1521) are different from the values of the filter coefficients in CC-ALF(1531). 10 Typically, the filter coefficients (e.g., non-zero filter coefficients) in CC-ALF can be sent in the APS, for example. In the example, the filter coefficients can be scaled by a factor (e.g., 2 10 ) and rounded to a fixed-point representation. The application of CC-ALF can be controlled on a variable block size, and the application of CC-ALF is signaled by a context coding flag (e.g., CC-ALF enable flag) received for each sample block. The context coding flag (e.g., CC-ALF enable flag) can be signaled at any suitable level such as the block level. For each chrominance component, the block size and the CC-ALF enable flag can be received at the slice level. In some examples, block sizes (in chrominance samples) of 16×16, 32×32, and 64×64 can be supported.

[0192] Figure 17 shows a syntax example for CC - ALF according to some embodiments of the present disclosure. In Figure 17 the example, alf_ctb_cross_component_cb_idc[xCtb >> CtbLog2SizeY][yCtb >> CtbLog2SizeY] is an index indicating whether to use the cross - component Cb filter, and if so, it is the index of the cross - component Cb filter. For example, when alf_ctb_cross_component_cb_idc[xCtb >> CtbLog2SizeY][yCtb >> CtbLog2SizeY] is equal to 0, the cross - component Cb filter is not applied to the Cb color - component sample block at the luma position (xCtb, yCtb); when alf_ctb_cross_component_cb_idc[xCtb >> CtbLog2SizeY][yCtb >> CtbLog2SizeY] is not equal to 0, alf_ctb_cross_component_cb_idc[xCtb >> CtbLog2SizeY][yCtb >> CtbLog2SizeY] is the index of the filter to be applied. For example,

[0193] the alf_ctb_cross_component_cb_idc[xCtb >> CtbLog2SizeY][yCtb >> CtbLog2SizeY]-th cross - component Cb filter is applied to the Cb color - component sample block at the luma position (xCtb, yCtb).

[0194] In addition, in Figure 17 the example,

[0195] alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is used to indicate whether to use the cross-component Cr filter and the index of whether to use the cross-component Cr filter. For example, when alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is equal to 0, the cross-component Cr filter is not applied to the Cr color component sample block at the luminance position (xCtb, yCtb); when alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is not equal to 0, alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is the index of the cross-component Cr filter. For example,

[0196] the alf_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]-th cross-component Cr filter can be applied to the Cr color component sample block at the luminance position (xCtb, yCtb).

[0197] In some examples, chroma subsampling techniques are used, so the number of samples in each chroma block can be less than the number of samples in the luminance block. The chroma subsampling format (also known as the chroma subsampling format specified by, for example, chroma_format_idc) can indicate the chroma horizontal subsampling factor (e.g., SubWidthC) and the chroma vertical subsampling factor (e.g., SubHeightC) between each chroma block and the corresponding luminance block. In an example, the chroma subsampling format is 4:2:0, so the chroma horizontal subsampling factor (e.g., SubWidthC) and the chroma vertical subsampling factor (e.g., SubHeightC) are 2, as Figures 18A - 18B shown. In an example, the chroma subsampling format is 4:2:2, so the chroma horizontal subsampling factor (e.g., SubWidthC) is 2 and the chroma vertical subsampling factor (e.g., SubHeightC) is 1. In an example, the chroma subsampling format is 4:4:4, so the chroma horizontal subsampling factor (e.g., SubWidthC) and the chroma vertical subsampling factor (e.g., SubHeightC) are both 1. The chroma sample type (also known as the chroma sample position) can indicate the relative position of the chroma samples in the chroma block with respect to at least one corresponding luminance sample in the luminance block.

[0198] Figures 18A - 18B Shows an exemplary position of a chrominance sample relative to a luminance sample according to an embodiment of the present disclosure. Referring to Figure 18A , the luminance samples (1801) are located in rows (1811)-(1818). Figure 18A The luminance samples (1801) shown in may represent a part of a picture. In an example, a luminance block (e.g., a luminance CB) includes the luminance samples (1801). The luminance block may correspond to two chrominance blocks having a chrominance subsampling format of 4:2:0. In an example, each chrominance block includes chrominance samples (1803). Each chrominance sample (e.g., chrominance sample (1803(1)) corresponds to four luminance samples (e.g., luminance samples (1801(1))-(1801(4))). In an example, these four luminance samples are the top-left sample (1801(1)), the top-right sample (1801(2)), the bottom-left sample (1801(3)), and the bottom-right sample (1801(4)). The chrominance sample (e.g., (1803(1))) is located at a left-center position between the top-left sample (1801(1)) and the bottom-left sample (1801(3)), and the chrominance sample type of the chrominance block having the chrominance sample (1803) may be referred to as chrominance sample type 0. Chrominance sample type 0 indicates a relative position 0 corresponding to the left-center position between the top-left sample (1801(1)) and the bottom-left sample (1801(3)). These four luminance samples (e.g., (1801(1))-(1801(4))) may be referred to as the adjacent luminance samples of the chrominance sample (1803(1)).

[0199] In an example, each chrominance block includes chrominance samples (1804). The above description of the chrominance samples (1803) may be applicable to the chrominance samples (1804), and thus the detailed description may be omitted for the sake of brevity. Each of the chrominance samples (1804) may be located at the center position of four corresponding luminance samples, and the chrominance sample type of the chrominance block having the chrominance samples (1804) may be referred to as chrominance sample type 1. Chrominance sample type 1 indicates a relative position 1 corresponding to the center position of the four luminance samples (e.g., (1801(1))-(1801(4))). For example, one of the chrominance samples (1804) may be located in the central portion of the luminance samples (1801(1))-(1801(4)).

[0200] In the example, each chrominance block includes chrominance samples (1805). Each of the chrominance samples (1805) can be located at the upper-left position that is co-located with the upper-left sample position of four corresponding luma samples (1801), and the chrominance sample type of the chrominance block having the chrominance samples (1805) can be referred to as chrominance sample type 2. Thus, each of the chrominance samples (1805) has the same upper-left sample position as the upper-left sample positions of the four luma samples (1801) corresponding to each chrominance sample. Chrominance sample type 2 indicates the relative position 2 corresponding to the upper-left position of the four luma samples (1801). For example, one of the chrominance samples (1805) can be located at the upper-left position of the luma samples (1801(1))-(1801(4)).

[0201] In the example, each chrominance block includes chrominance samples (1806). Each of the chrominance samples (1806) can be located at the upper-center position between the corresponding upper-left sample and the corresponding upper-right sample, and the chrominance sample type of the chrominance block having the chrominance samples (1806) can be referred to as chrominance sample type 3. Chrominance sample type 3 indicates the relative position 3 corresponding to the upper-center position between the upper-left sample and the upper-right sample. For example, one of the chrominance samples (1806) can be located at the upper-center position of the luma samples (1801(1))-(1801(4)).

[0202] In the example, each chrominance block includes chrominance samples (1807). Each of the chrominance samples (1807) can be located at the lower-left position that is co-located with the lower-left sample position of four corresponding luma samples (1801), and the chrominance sample type of the chrominance block having the chrominance samples (1807) can be referred to as chrominance sample type 4. Thus, each of the chrominance samples (1807) has the same lower-left sample position as the lower-left sample positions of the four luma samples (1801) corresponding to each chrominance sample. Chrominance sample type 4 indicates the relative position 4 corresponding to the lower-left position of the four luma samples (180(1)). For example, one of the chrominance samples (1807) can be located at the lower-left position of the luma samples (1801(1))-(1801(4)).

[0203] In the example, each chrominance block includes chrominance samples (1808). Each of the chrominance samples (1808) is located at the lower-center position between the lower-left sample and the lower-right sample, and the chrominance sample type of the chrominance block having the chrominance samples (1808) can be referred to as chrominance sample type 5. Chrominance sample type 5 indicates the relative position 5 corresponding to the lower-center position between the lower-left sample and the lower-right sample of the four luma samples (1801). For example, one of the chrominance samples (1808) can be located between the lower-left sample and the lower-right sample of the luma samples (1801(1))-(1801(4)).

[0204] In general, any suitable chrominance sample type can be used for a chrominance subsampling format. Chrominance sample types 0-5 are exemplary chrominance sample types described with chrominance subsampling format 4:2:0. Additional chrominance sample types can be used for chrominance subsampling format 4:2:0. Furthermore, other chrominance sample types and / or variations of chrominance sample types 0-5 can be used for other chrominance subsampling formats, such as 4:2:2, 4:4:4, etc. In an example, a chrominance sample type that combines chrominance samples (1805) and (1807) is used for chrominance subsampling format 4:2:2.

[0205] In the example, a luma block is considered to have alternating rows, such as rows (1811)-(1812), where these two rows respectively include the upper two samples (e.g., (1801(1))-(1801(2))) of four luma samples (e.g., (1801(1))-(1801(4))) and the lower two samples (e.g., (1801(3))-(1801(4))) of four luma samples (e.g., (1801(1)-(1801(4)))). Thus, rows (1811), (1813), (1815), and (1817) can be referred to as the current rows (also referred to as the top field), and rows (1812), (1814), (1816), and (1818) can be referred to as the next rows (also referred to as the bottom field). Four luma samples (e.g., (1801(1))-(1801(4))) are located in the current row (e.g., (1811)) and the next row (e.g., (1812)). Relative positions 2-3 are in the current row, relative positions 0-1 are between each current row and the corresponding next row, and relative positions 4-5 are in the next row.

[0206] Chrominance samples (1803), (1804), (1805), (1806), (1807), or (1808) are located in rows (1851)-(1854) of each chrominance block. The specific positions of rows (1851)-(1854) can depend on the chrominance sample type of the chrominance samples. For example, for chrominance samples (1803)-(1804) with corresponding chrominance sample types 0-1, row (1851) is between rows (1811)-(1812). For chrominance samples (1805)-(1806) with corresponding chrominance sample types 2-3, row (1851) is in the same position as the current row (1811). For chrominance samples (1807)-(1808) with corresponding chrominance sample types 4-5, row (1851) is in the same position as the next row (1812). The above description can be appropriately applied to rows (1852)-(1854), and the detailed description is omitted for brevity.

[0207] Any suitable scanning method can be used to display, store, and / or transmit the luminance blocks and corresponding chrominance blocks described above in Figure 18A . In an example, progressive scanning is used.

[0208] Interlaced scanning can be used, as shown in Figure 18B . As described above, the chroma subsampling format is 4:2:0 (e.g., chroma_format_idc equals 1). In an example, the variable chroma position type (e.g., ChromaLocType) indicates the current line (e.g., ChromaLocType is chroma_sample_loc_type_top_field) or the next line (e.g., ChromaLocType is chroma_sample_loc_type_bottom_field). The current lines (1811), (1813), (1815), and (1817) and the next lines (1812), (1814), (1816), and (1818) can be scanned separately. For example, the current lines (1811), (1813), (1815), and (1817) can be scanned first, and then the next lines (1812), (1814), (1816), and (1818) can be scanned. The current line can include luminance samples (1801), and the next line can include luminance samples (1802).

[0209] Similarly, the corresponding chrominance blocks can be scanned interlaced. The lines (1851) and (1853) that include chrominance samples (1803), (1804), (1805), (1806), (1807), or (1808) without padding can be referred to as the current lines (or current chrominance lines), and the lines (1852) and (1854) that include chrominance samples (1803), (1804), (1805), (1806), (1807), or (1808) with gray padding can be referred to as the next lines (or next chrominance lines). In an example, during interlaced scanning, the lines (1851) and (1853) are scanned first, and then the lines (1852) and (1854) are scanned.

[0210] In some examples, constrained directional enhancement filtering techniques can be used. Using an in-loop constrained directional enhancement filter (CDEF) can filter out codec artifacts while preserving the details of the image. In an example (e.g., HEVC), the sample adaptive offset (SAO) algorithm can achieve a similar purpose by defining signal offsets for different classes of pixels. Different from SAO, CDEF is a non-linear spatial filter. In some examples, CDEF can be constrained to be easily vectorized (i.e., can be implemented with single-instruction multiple-data (SIMD) operations). Note that other non-linear filters, such as median filters and bilateral filters, cannot be processed in the same way.

[0211] In some cases, the number of ringing artifacts in the encoded image tends to be roughly proportional to the quantization step. The amount of detail is a property of the input image, but the minimum detail retained in the quantized image also tends to be proportional to the quantization step. For a given quantization step, the amplitude of ringing is typically less than the amplitude of the detail.

[0212] CDEF can be used to identify the direction of each block, then filter adaptively along the identified direction, and filter to a lesser extent along a direction rotated 45 degrees from the identified direction. In some examples, the encoder can search for filter strength and can explicitly signal the filter strength, which allows for a high degree of control over blurring.

[0213] Specifically, in some examples, after the deblocking filter, a direction search is performed on the reconstructed pixels. Since these pixels are available to the decoder, the decoder can search for the direction, so in the examples these directions do not need to be signaled. In some examples, the direction search can operate on certain block sizes (e.g., 8×8 blocks), which are small enough to handle non-linear edges and large enough to reliably estimate the direction when applied to the quantized image. Also, having a constant direction over an 8×8 region makes vectorization of the filter easier. In some examples, each block (e.g., 8×8) can be compared to a fully oriented block to determine the difference. A fully oriented block is a block in which all pixels along a line in one direction have the same value. In the example, a difference measure between the block and each fully oriented block can be calculated, such as the sum of squared differences (SSD), root mean square (RMS) error. Then, the fully oriented block with the minimum difference (e.g., minimum SSD, minimum RMS, etc.) can be determined, and the direction of the determined fully oriented block can be the direction that best matches the pattern in the block.

[0214] Figure 19 An example of a direction search according to an embodiment of the present disclosure is shown. In the example, block (1910) is an 8×8 block that is reconstructed and output from the deblocking filter. In Figure 19 the example, the direction search can determine the direction for block (1910) from the 8 directions shown in (1920). Eight fully oriented blocks (1930) are formed corresponding to the 8 directions (1920) respectively. The fully oriented block corresponding to a direction is a block in which the pixels along the line in that direction have the same value. Additionally, a difference measure between block (1910) and each fully oriented block (1930) can be calculated, such as SSD, RMS error, etc. In Figure 19In the example of, the RMS error is shown by (1940). As shown by (1943), the RMS errors of block (1910) and the fully oriented block (1933) are the smallest, so the direction (1923) is the direction that best matches the pattern in block (1910).

[0215] After identifying the direction of the block, a non-linear low-pass directional filter can be determined. For example, the filter taps of the non-linear low-pass directional filter can be aligned along the identified direction to reduce ringing while preserving the directional edges or patterns. However, in some examples, directional filtering alone sometimes cannot sufficiently reduce ringing. In the example, additional filter taps are also used for pixels that do not lie along the identified direction. To reduce the risk of blurring, these additional filter taps are treated more conservatively. For this purpose, the CDEF includes primary filter taps and secondary filter taps. In the example, the complete 2-D CDEF filter can be expressed as equation (14):

[0216]

[0217] where D represents the damping parameter, S (p) represents the strength of the primary filter taps, S (s) represents the strength of the secondary filter taps, round(·) represents a rounding operation that avoids zero, w represents the filter weight, and f(d, S, D) is a constraint function that operates on the difference between the filtered pixel and each adjacent pixel. In the example, for a small difference, the function f(d, S, D) is equal to D, which can make the filter behave like a linear filter; when the difference is large, the function f(d, S, D) is equal to 0, which can effectively ignore the filter taps.

[0218] In some examples, an in-loop restoration scheme is used in deblocking after video coding to generally reduce noise and improve edge quality in addition to the deblocking operation. In the example, the in-loop restoration scheme is switchable within each appropriately sized tile in the frame. The in-loop restoration scheme is based on a separable symmetric Wiener filter, a dual self-guided filter with subspace projection, and a domain transform recursive filter. Because the content statistics can vary greatly within a frame, the in-loop restoration scheme is integrated into a switchable architecture in which different schemes can be triggered in different regions of the frame.

[0219] The separable symmetric Wiener filter can be one of the in-loop recovery schemes. In some examples, each pixel in a degraded frame can be reconstructed as a non-causal filtered version of the pixels within a w×w window around it, where w = 2r + 1 is an odd number for an integer r. If the 2D filter taps are represented by a w 2 ×1 element vector F in column vector quantization form, then the direct LMMSE optimization will result in the filter parameters being given by F = H -1 M, where H = E[XX T is the autocovariance of x, which is the column vector quantization version of the w 2 samples in the w×w window around the pixel, and M = E[YX T is the cross-correlation between x and the scalar source sample y to be estimated. In an example, the encoder can estimate H and M based on the implementation in the deblocked frame and the source, and can send the resulting filter F to the decoder. However, this will not only incur a large bitrate cost when sending w 2 taps, but also the non-separable filtering will make the decoding very complex. In some embodiments, several additional constraints are imposed on the characteristics of F. For the first constraint, F is constrained to be separable so that the filtering can be implemented as separable horizontal and vertical w-tap convolutions. For the second constraint, each of the horizontal filter and the vertical filter is constrained to be symmetric. For the third constraint, it is assumed that the sum of the horizontal filter coefficients and the vertical filter coefficients is 1.

[0220] Dual self-guided filtering with subspace projection can be one of the in-loop recovery schemes. Guided filtering is an image filtering technique where the local linear model is shown by (Equation 15):

[0221] y = Fx + G (Equation 15).

[0222] (Equation 15) The local linear model shown is used to calculate the filtered output y based on the unfiltered samples x, where F and G are determined based on the statistics of the degraded image and the guidance image near the filtered pixels. If the guidance image is the same as the degraded image, the resulting so-called self-guided filtering will have the effect of keeping the edges smooth. In an example, a specific form of self-guided filtering can be used. The specific form of self-guided filtering depends on two parameters: the radius r and the noise parameter e, and the following steps are listed:

[0223] 1. Obtain the mean μ and variance σ of the pixels in the (2r + 1)×(2r + 1) window around each pixel 2 . This step can be effectively implemented using box filtering based on integral imaging.

[0224] 2. Calculate for each pixel: f = σ 2 / (σ 2 + e); g = (1 - f)μ.

[0225] 3. Calculate F and G for each pixel as the average of the f and g values in the 3×3 window around that pixel for use.

[0226] The specific form of the self - guided filter is controlled by r and e, where a higher r means a higher spatial variance and a higher e means a higher range variance.

[0227] Figure 20 Examples of subspace projections in some examples are shown. As Figure 20 shown, even if neither X1 nor X2 is close to the source Y in the recovery, as long as they are slightly shifted in the correct direction, appropriate multipliers {α, β} can bring them closer to the source Y.

[0228] The domain - transform recursive filter can be one of the in - loop recovery schemes. Domain - transform is a method for edge - preserving image filtering using 1 - D operations, which can be much faster than other edge - aware processing methods. Using a recursive filtering form, where the processing steps include recursively filtering of order 1 horizontally from left to right and from right to left, followed by vertically from top to bottom and from bottom to top filtering, and carried out over several (usually 3) iterations. The filter taps are obtained from the local horizontal and vertical gradients of the pixels and the iteration index.

[0229] Some aspects of the present disclosure provide temporal cross - component filtering techniques to use samples from frames different from the current frame as the source for cross - component filtering (CCF), so that the source of cross - component filtering (CCF) is not limited to the reconstructed samples of the current block in the current frame, and the performance of CCF is improved.

[0230] Generally, cross - component filtering techniques are defined as a filtering process that uses the reconstructed samples of the first color component as input (e.g., Y or Cb or Cr) and applies the output to a second color component different from the first color component.

[0231] According to the present disclosure, temporal cross - component filtering techniques can use the reconstructed samples from a temporal source frame as the source for CCF. The temporal source frame is a frame different from the current frame. The source of CCF can be used to derive the CCF coefficients and can be used as the input samples for the filtering process. In some embodiments, on the encoder side, the encoder can derive the CCF coefficients based on the source of CCF. The CCF coefficients can be appropriately provided to the decoder. In addition, on both the encoder side and the decoder side, the source of CCF can provide input samples for the cross - component filtering process.

[0232] Note that in the following description, the source is used to refer to the source of the CCF. The following description includes various temporal cross-component filtering techniques. The temporal cross-component filtering techniques can be used alone or in any combination in any order. Additionally, each of the methods, temporal cross-component filtering techniques (or embodiments), encoders, and decoders can be implemented by processing circuitry (e.g., one or more processors, or one or more integrated circuits). In an example of some embodiments, one or more processors may execute a program stored in a non-volatile computer-readable medium and perform the temporal cross-component filtering techniques.

[0233] According to an aspect of the present disclosure, the source of the CCF can be the luminance or chrominance component of a temporal source frame different from the current frame. The temporal source frame may have a larger picture order count (POC) than the current frame or a smaller picture order count than the current frame.

[0234] In some embodiments, an intra-coded picture (I-frame) may be used as the temporal source frame. The I-frame can be selected based on the quantization parameter (QP). The QP value can indicate how much spatial detail is retained in the encoded video bitstream. For example, when the QP value is very small, almost all details are retained. When the QP value increases, some spatial details can be aggregated to reduce the bitrate, but at the cost of increased distortion and quality loss. In an embodiment, an I-frame with a QP value lower than that of the current frame can be used as the temporal source frame. In an example, the luminance component of an I-frame with a QP value lower than that of the current frame is used as the temporal source frame. In another example, the chrominance component of an I-frame with a QP value lower than that of the current frame is used as the temporal source frame. In another example, a weighted sum of the luminance and chrominance components of an I-frame with a QP value lower than that of the current frame can be used as the temporal source frame.

[0235] In a certain embodiment, an inter-coded picture (e.g., an inter-predicted picture (P-frame), a bi-predicted picture (B-frame), etc.) that meets certain characteristic requirements (e.g., QP requirements, temporal distance requirements, reference picture requirements, etc.) can be used as the temporal source frame. In an example, it is required to encode the inter-frame with a QP value lower than that of the current frame. In another example, it is required to encode the inter-frame within a certain temporal distance from the current frame. For example, if the difference between the candidate inter-frame and the current frame in the display order (or POC) is less than a threshold d, the candidate inter-frame can be selected as the temporal source frame. In another example, if the encoded inter-frame is a reference picture used for motion compensation during the decoding of the current frame, that inter-frame can be selected as the temporal source frame.

[0236] According to an aspect of the present disclosure, reconstructed samples from different pictures and / or different color components can be used to filter the reconstructed samples of the current picture.

[0237] In some embodiments, the source of the CCF can be the reconstructed samples of the blocks in the temporal source frame. In some examples, motion information (e.g., motion vectors, motion estimation, modified motion vectors, etc.) can be identified for different blocks of the current frame. Then, for each block of the current frame, the corresponding block in the temporal source frame can be identified based on the motion information. Then, the reconstructed samples of the corresponding block can be used as the source of the CCF for that block.

[0238] In an embodiment, the motion information can be exported as a motion vector for motion compensation of the current sample / block.

[0239] In another embodiment, the motion information can be exported through a separate motion estimation (ME) process. The motion estimation process can determine the transformation between the current block and the temporal source frame. Before applying the CCF, the motion estimation process is performed on the encoder and decoder using the reconstructed temporal source frame and the current block. In an example, only the luminance component of the current block and the luminance component of the temporal source frame are used to export the motion information during the motion estimation process. In another example, only the current color component of the current block and the current color component of the temporal source frame are used to export the motion information during the motion estimation process. In another example, all color components of the current block and all color components of the temporal source frame are used to export the motion information during the motion estimation process.

[0240] In another embodiment, the motion information can be exported through a combination of a motion estimation process and a motion vector for motion compensation, such as using a motion vector refinement technique. In an example, the starting point of the motion estimation process is the motion vector for motion compensation of the current sample / block. Based on this starting point, before applying the CCF to the exported motion information, the motion estimation process is performed on the encoder and decoder using the reconstructed temporal source frame and the current block.

[0241] In some examples, the accuracy of the motion information exported using a motion estimation process or a combination of a motion estimation process and a motion vector as the starting point can be higher than the accuracy of the motion vector for motion compensation.

[0242] In an example, the motion information exported using a motion estimation process or a combination of a motion estimation process and a motion vector as the starting point has integer sample accuracy. In another example, the motion information exported using a motion estimation process or a combination of a motion estimation process and a motion vector as the starting point has half-pixel accuracy. In another example, the motion information exported using a motion estimation process or a combination of a motion estimation process and a motion vector as the starting point has quarter-pixel accuracy.

[0243] In some embodiments, the source of the CCF can be a weighted sum of reconstructed samples in the current block and the corresponding block in the temporal source frame. In some examples, motion information (e.g., motion vectors, motion estimation, modified motion vectors, etc.) can be identified for different blocks of the current frame. Then, for each block of the current frame, the corresponding block in the temporal source frame can be identified based on the motion information. Then, the weighted sum of the reconstructed samples in the block and the reconstructed samples in the corresponding block can be used as the source of the CCF for that block. The motion information can be derived by any suitable technique. In an example, the motion information is derived as a motion vector for motion compensation of the current sample / block. In another example, the motion information is derived through a motion estimation process. In another example, the motion information is derived by a combination of a motion vector for motion compensation and a motion estimation process starting from the motion vector.

[0244] According to aspects of the present disclosure, information about the source of the CCF can be signaled. For example, it can be signaled in any parameter set (e.g., high-level syntax (e.g., DPS, VPS, SPS, PPS, APS), slice header, tile / tile group header, SEI message, etc.) whether the source of the CCF is from the current block of the current frame or the corresponding block in the temporal source frame, or both the current frame and the temporal source frame.

[0245] In some embodiments, a flag can be signaled to indicate whether the source of the CCF is the current block of the current frame or the corresponding block in the temporal source frame. For example, a flag represented by blockIdc can be signaled for each block for which the CCF is performed. When blockIdc is equal to 1, the corresponding block in the temporal source frame is used as the source of the CCF; when blockIdc is equal to 0, the current block in the current frame is used as the source of the CCF.

[0246] In some embodiments, when the source of the CCF is a weighted sum of the current block and the corresponding block in the temporal source frame, a flag can be signaled to indicate the weighting between the current block and the corresponding block. For example, a weight index represented by block_weight can be signaled for each block for which the CCF is performed. When block_weight is equal to 0, the weighting between the current block and the corresponding block in the temporal source frame is 0:1. When block_weight is equal to 1, the weighting between the current block and the corresponding block in the temporal source frame is 3:7. When block_weight is equal to 2, the weighting between the current block and the corresponding block in the temporal source frame is 5:5. When block_weight is equal to 3, the weighting between the current block and the corresponding block in the temporal source frame is 7:3. When block_weight is equal to 4, the weighting between the current block and the corresponding block in the temporal source frame is 1:0.

[0247] In some embodiments, the weighting factors for the current block and the corresponding block in the temporal source frame are not signaled directly. In an example, the weighting factors for the current block and the corresponding block in the temporal source frame depend on the QP values of the current block and the corresponding block. Accordingly, the weighting factors can be determined based on the QP values of the current block and the corresponding block.

[0248] Note that although CCF is used in the above description to illustrate the filtering technique using a temporal source frame, the above description can be modified to apply the filtering technique using a temporal source frame to other loop filtering tools, such as ALF, loop restoration, CDEF, etc.

[0249] Figure 21 A flowchart showing an overview process (2100) according to an embodiment of the present disclosure is shown. The process (2100) can be used to reconstruct a block in a picture of an encoded video sequence. The term "block" can be interpreted as a prediction block, a coding unit, a luminance block, a chrominance block, etc. In various embodiments, the process (2100) is executed by a processing circuit, such as the processing circuit in the terminal devices (310), (320), (330), and (340), the processing circuit that performs the functions of the video encoder (403), the processing circuit that performs the functions of the video decoder (410), the processing circuit that performs the functions of the video decoder (510), the processing circuit that performs the functions of the video encoder (603), and so on. In some embodiments, the process (2100) is implemented by software instructions, so when the processing circuit executes the software instructions, the processing circuit executes the process (2100). The process (2100) starts at step (S2101) and proceeds to step (S2110).

[0250] Step (S2110), generating a first reconstructed sample of the current block in the current frame.

[0251] Step (S2120), obtaining a reconstructed sample in a temporal source frame different from the current frame. The temporal source frame can be any suitable frame, such as an intra-coded frame (I-frame), an inter-coded frame (P-frame, B-frame), etc. In an embodiment, the temporal source frame is an I-frame having a lower quantization parameter (QP) value than the current frame.

[0252] In some embodiments, the temporal source frame is an inter-coded frame that meets certain requirements. In an embodiment, the temporal source frame is an inter-coded frame that meets a QP requirement, where the QP requirement indicates that the inter-coded frame has a lower QP value than the current frame. In another embodiment, the temporal source frame is an inter-coded frame that meets a temporal distance requirement, where the temporal distance requirement indicates that the temporal distance (e.g., the absolute difference between POCs) between the inter-coded frame and the current frame is less than a threshold. In another embodiment, the temporal source frame is an inter-coded frame that meets a reference picture requirement, where the reference picture requirement indicates that the inter-coded frame is a reference picture used for motion compensation during the reconstruction of the current frame.

[0253] In some embodiments, a corresponding block in the temporal source frame that corresponds to the current block is determined. For example, the motion information of the current block can be identified. Then, based on the motion information, the corresponding block of the current block is determined in the temporal source frame. In an embodiment, the motion information is derived as a motion vector for motion compensation of the current block. In another embodiment, a motion estimation process is performed to identify the motion information. The motion estimation process can determine the transformation between the current block and the temporal source frame. In an example, the motion estimation process is performed separately from the motion vector. In another example, the motion estimation process is performed starting from the motion vector for motion compensation of the current block, so as to identify the motion information in a manner similar to the motion vector refinement process.

[0254] Note that the motion estimation process can be performed on any suitable color component. In an example, the motion estimation process is performed using the luminance component of the current block and the luminance component of the temporal source frame to identify the motion information. In another example, the motion estimation process is performed using the current color component of the current block and the current color component of the temporal source frame to identify the motion information. In another example, the motion estimation process is performed using all color components of the current block and all color components of the temporal source frame to identify the motion information.

[0255] Step (S2130), applying a filter to at least the reconstructed samples in the temporal source frame to generate an offset.

[0256] In an example, the filter is applied to the luminance component of the temporal source frame to generate an offset. In another example, the filter is applied to the chrominance component of the temporal source frame to generate an offset. In another example, the filter is applied to a weighted sum of the luminance component and one or more chrominance components of the temporal source frame to generate an offset.

[0257] The filter can be any suitable filter, such as a cross-component filter (CCF), an adaptive loop filter (ALF), a loop restoration filter, an in-loop constrained directional enhancement filter (CDEF), etc. In an example, the filter is a cross-component filter, and the filter is applied to the first color component of the time source frame to generate an offset, and the offset is combined with the second color component of the first reconstructed sample.

[0258] In some embodiments, the filter is applied at least to the reconstructed samples of a corresponding block to generate an offset. In an embodiment, the filter is applied to the reconstructed samples of a corresponding block to generate an offset. In another embodiment, the filter is applied to a weighted sum of the reconstructed samples of a corresponding block and the first reconstructed sample of the current block to generate an offset.

[0259] Step (S2140), combining the first reconstructed sample with the offset to generate a second reconstructed sample of the current block. Process (2100) proceeds to the end of step (S2199).

[0260] Note that process (2100) can be executed on the encoder side and the decoder side. In some embodiments, process (2100) is executed on the decoder side. On the decoder side, the prediction information of the current block can be decoded from the encoded video bitstream. The prediction information indicates the characteristics of the filter. In an example, the decoder can determine the input of the filter (e.g., the reconstructed samples of the current frame and / or the reconstructed samples of the time source frame) based on a flag (e.g., blockIdc) decoded from the encoded video bitstream. In another example, the decoder can determine a weighting factor based on a weight index (e.g., block_weight) decoded from the encoded video bitstream. In another example, the decoder can determine a weighting factor based on the quantization parameter (QP) value of the current block in the current frame and the corresponding block in the time source frame.

[0261] In some embodiments, process (2100) is executed on the encoder side. On the encoder side, in an embodiment, the encoder can determine the filter coefficients based on the reconstructed samples of the time source frame, and encode the information of the filter coefficients in the encoded video bitstream sent to the decoder side. In another example, on the encoder side, the encoder can determine the filter coefficients based on the reconstructed samples of the time source frame and the reconstructed samples of the current frame, and encode the information of the filter coefficients in the encoded video bitstream sent to the decoder side.

[0262] Process (2100) can be appropriately modified. One or more steps in process (2100) can be modified and / or omitted. Other steps can be added. Any suitable order of implementation can be used.

[0263] Embodiments of the present disclosure provide an apparatus for video decoding, including: a generation module configured to generate first reconstructed samples of a current block in a current frame; an acquisition module configured to acquire reconstructed samples in a temporal source frame different from the current frame; an application module configured to apply a filter to at least the reconstructed samples in the temporal source frame to generate an offset; and a combination module configured to combine the first reconstructed samples of the current block with the offset to generate second reconstructed samples of the current block.

[0264] Embodiments of the present disclosure also provide a device for video decoding, including: one or more processors and one or more memories, wherein at least one program instruction is stored in the one or more memories and is loaded and executed by the one or more processors to implement the method for video decoding.

[0265] Embodiments in the present disclosure can be used alone or in any combination. In addition, each of the method (or embodiment), encoder, and decoder can be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-volatile computer-readable medium.

[0266] The above technology can be implemented as computer software by computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 22 FIG. shows a computer system (2200) adapted to implement certain embodiments of the disclosed subject matter.

[0267] The computer software can be encoded by any suitable machine code or computer language, and code including instructions is created through mechanisms such as assembly, compilation, and linking. The instructions can be directly executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed through methods such as decoding and microcode.

[0268] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0269] Figure 22 The components shown for the computer system (2200) are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of the computer software implementing the embodiments of the present application. Nor should the configuration of the components be construed as having any dependence on or requirement for any one component or combination thereof shown in the exemplary embodiments of the computer system (2200).

[0270] A computer system (2200) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to inputs from one or more human users through tactile inputs (such as keyboard inputs, swipes, data glove movements), audio inputs (such as sounds, applause), visual inputs (such as gestures), and olfactory inputs (not shown). The human-machine interface device may also be used to capture certain media, which need not be directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0271] The human-machine interface input device may include one or more of the following (only one is shown): keyboard (2201), mouse (2202), touchpad (2203), touch screen (2210), data glove (not shown), joystick (2205), microphone (2206), scanner (2207), camera (2208).

[0272] The computer system (2200) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (such as tactile feedback through the touch screen (2210), data glove (not shown), or joystick (2205), but there may also be tactile feedback devices that do not serve as input devices), audio output devices (such as speakers (2209), headphones (not shown)), visual output devices (such as screens (2210) including cathode ray tube screens, liquid crystal screens, plasma screens, organic light-emitting diode screens, each of which may or may not have touch screen input functionality and each of which may or may not have tactile feedback functionality - some of which may output two-dimensional visual output or output above three dimensions through means such as stereoscopic picture output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

[0273] The computer system (2200) may also include human-accessible storage devices and their associated media, such as optical media including high-density read-only / rewritable optical discs with CD / DVD (CD / DVD ROM / RW) (2220) or similar media (2221), thumb drives (2222), removable hard disk drives or solid state drives (2223), traditional magnetic media such as tapes and floppy disks (not shown), dedicated devices based on ROM / ASIC / PLD such as security software protectors (not shown), and so on.

[0274] Those skilled in the art should also understand that the term "computer-readable medium" used in connection with the disclosed subject matter does not include a transmission medium, a carrier wave, or other transient signals.

[0275] The computer system (2200) may also include a network interface (2254) to one or more communication networks (2255). For example, the one or more communication networks (2255) may be wireless, wired, optical. The one or more communication networks (2255) may also be a local area network, a wide area network, a metropolitan area network, a vehicular network, and an industrial network, a real-time network, a delay-tolerant network, etc. The one or more communication networks (2255) also include local area networks such as Ethernet, wireless local area network, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), wired or wireless wide area digital television networks (including cable television, satellite television, and terrestrial broadcast television), vehicular and industrial networks (including CANBus), etc. Some networks typically require an external network interface adapter for connection to certain common data ports or peripheral buses (2249) (e.g., the USB port of the computer system (2200)); other systems are typically integrated into the core of the computer system (2200) by connecting to the system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smart phone computer system). By using any of these networks, the computer system (2200) can communicate with other entities. The communication can be unidirectional, only for receiving (e.g., wireless television), unidirectional only for sending (e.g., CAN bus to certain CAN bus devices), or bidirectional, e.g., to other computer systems via a local or wide area digital network. Each of the above networks and network interfaces may use certain protocols and protocol stacks.

[0276] The above-described human-machine interface device, human-accessible storage device, and network interface can be connected to the core (2240) of the computer system (2200).

[0277] The core (2240) may include one or more central processing units (CPUs) (2241), a graphics processing unit (GPU) (2242), a dedicated programmable processing unit in the form of a field-programmable gate array (FPGA) (2243), a hardware accelerator for specific tasks (2244), a graphics adapter (2250), etc. These devices, as well as a read-only memory (ROM) (2245), a random access memory (2246), an internal mass storage (such as an internal non-user-accessible hard disk drive, a solid-state drive, etc.) (2247), etc., can be connected via a system bus (2248). In some computer systems, the system bus (2248) can be accessed in the form of one or more physical plugs so as to be expandable via additional central processing units, graphics processing units, etc. Peripheral devices can be directly attached to the system bus (2248) of the core or connected via a peripheral bus (2249). In an example, a touch screen (2210) can be connected to the graphics adapter (2250). The architecture of the peripheral bus includes an external controller interface PCI, a universal serial bus USB, etc.

[0278] The CPU (2241), GPU (2242), FPGA (2243), and accelerator (2244) can execute certain instructions, and these instructions combined can constitute the above-mentioned computer code. The computer code can be stored in the ROM (2245) or the RAM (2246). Transitional data can also be stored in the RAM (2246), while permanent data can be stored in, for example, the internal mass storage (2247). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (2241), GPUs (2242), mass storage (2247), ROM (2245), RAM (2246), etc.

[0279] The computer-readable medium may have computer code for performing various computer-implemented operations. The medium and the computer code can be specially designed and constructed for the purposes of this application or be well-known and available to those skilled in the field of computer software.

[0280] By way of example and not limitation, a computer system having an architecture (2200), particularly a core (2240), can function as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with the above-mentioned user-accessible mass storage, as well as specific memories of the non-volatile core (2240), such as the core internal mass storage (2247) or ROM (2245). The software implementing various embodiments of the present application can be stored in such devices and executed by the core (2240). Depending on specific needs, the computer-readable media can include one or more storage devices or chips. The software can cause the core (2240), particularly the processors therein (including CPU, GPU, FPGA, etc.), to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (2246) and modifying such data structures according to software-defined processes. Additionally or alternatively, the computer system can provide functions that are logically hardwired or otherwise included in a circuit (e.g., accelerator (2244)), which can operate instead of or in conjunction with the software to execute specific processes or specific parts of specific processes described herein. In appropriate cases, references to software can include logic and vice versa. In appropriate cases, references to computer-readable media can include circuits (such as integrated circuits (ICs)) that store and execute software, circuits that contain execution logic, or both. The present application encompasses any suitable combination of hardware and software.

[0281] Appendix A: Acronyms

[0282] JEM: Joint Exploration Model

[0283] VVC: Versatile Video Coding

[0284] BMS: Benchmark Set

[0285] MV: Motion Vector

[0286] HEVC: High Efficiency Video Coding

[0287] MPM: Most Probable Mode

[0288] WAIP: Wide-Angle Intra Prediction

[0289] SEI: Supplementary Enhancement Information

[0290] VUI: Video Usability Information

[0291] GOPs: Groups of Pictures

[0292] TUs: Transform Units

[0293] PUs: Prediction Units

[0294] CTUs: Coding Tree Units

[0295] CTBs: Coding Tree Blocks

[0296] PBs: Prediction Blocks

[0297] HRD: Hypothetical Reference Decoder

[0298] SDR: Standard Dynamic Range

[0299] SNR: Signal Noise Ratio

[0300] CPUs: Central Processing Units

[0301] GPUs: Graphics Processing Units

[0302] CRT: Cathode Ray Tube

[0303] LCD: Liquid-Crystal Display

[0304] OLED: Organic Light-Emitting Diode

[0305] CD: Compact Disc

[0306] DVD: Digital Video Disc

[0307] ROM: Read-Only Memory

[0308] RAM: Random Access Memory

[0309] ASIC: Application-Specific Integrated Circuit

[0310] PLD: Programmable Logic Device

[0311] LAN: Local Area Network

[0312] GSM: Global System for Mobile communications

[0313] LTE: Long-Term Evolution

[0314] CANBus: Controller Area Network Bus

[0315] USB: Universal Serial Bus

[0316] PCI: Peripheral Component Interconnect

[0317] FPGA: Field Programmable Gate Areas

[0318] SSD: Solid-state Drive

[0319] IC: Integrated Circuit

[0320] CU: Coding Unit

[0321] PDPC: Position Dependent Prediction Combination

[0322] ISP: Intra Sub-Partitions

[0323] SPS: Sequence Parameter Setting

[0324] Although the present application has described multiple exemplary embodiments, various changes, permutations, and various equivalent replacements of the embodiments are within the scope of the present application. Therefore, it should be understood that those skilled in the art can design various systems and methods, which, although not explicitly shown or described herein, embody the principles of the present application and thus fall within the spirit and scope of the present application.

Claims

1. A method for video decoding, characterized in that, Comprising: Generating a first reconstructed sample of a current block in a current frame; Obtaining a reconstructed sample in a temporal source frame different from the current frame; Applying a filter to at least the reconstructed sample in the temporal source frame to generate an offset; And Combining the first reconstructed sample of the current block with the offset to generate a second reconstructed sample of the current block; Wherein the temporal source frame is an inter-coded frame satisfying at least one of the following: A quantization parameter requirement, which indicates that the inter-coded frame has a lower QP value than the current frame; A temporal distance requirement, which indicates that the temporal distance between the inter-coded frame and the current frame is less than a threshold; and A reference picture requirement, which indicates that the inter-coded frame is a reference picture used for motion compensation in the reconstruction process of the current frame.

2. The method according to claim 1, characterized in that, The applying a filter to at least the reconstructed sample in the temporal source frame to generate an offset includes at least one of the following: Applying the filter to a luminance component of the temporal source frame to generate the offset; Applying the filter to a chrominance component of the temporal source frame to generate the offset.

3. The method according to claim 1, characterized in that, The applying a filter to at least the reconstructed sample in the temporal source frame to generate an offset includes: Applying the filter to a first color component of the temporal source frame to generate the offset; The combining the first reconstructed sample of the current block with the offset includes: Combining a second color component of the first reconstructed sample of the current block with the offset.

4. The method according to claim 1, characterized in that, Further comprising: Identifying motion information of the current block; And Determining a corresponding block of the current block in the temporal source frame based on the motion information; The applying a filter to at least the reconstructed sample in the temporal source frame to generate an offset includes: Applying the filter to at least the reconstructed sample of the corresponding block to generate the offset.

5. The method according to claim 4, characterized in that Further comprising: Exporting the motion information as a motion vector for motion compensation of the current block.

6. The method according to claim 4, wherein Further comprising: Performing a motion estimation process to identify the motion information.

7. The method according to claim 4, wherein Further comprising: Performing a motion estimation process starting from a motion vector for motion compensation of the current block to identify the motion information.

8. The method according to claim 4, wherein Further comprising at least one of the following: Performing a motion estimation process using a luminance component of the current block and a luminance component of the temporal source frame to identify the motion information; Performing a motion estimation process using a current color component of the current block and a current color component of the temporal source frame to identify the motion information.

9. The method according to claim 8, characterized in that, The applying the filter to at least the reconstructed sample of the corresponding block to generate the offset includes: Applying the filter to a weighted sum of the reconstructed sample of the corresponding block and the first reconstructed sample of the current block to generate the offset.

10. The method according to claim 1, wherein Further comprising: Decoding prediction information of the current block from an encoded video bitstream, the prediction information indicating characteristics of the filter.

11. The method according to claim 10, wherein Further comprising: Determining an input of the filter based on a flag decoded from the encoded video bitstream.

12. The method according to claim 10, wherein Further comprising: Determine a weighting factor based on a weight index decoded from the encoded video bitstream.

13. The method according to claim 10, characterized in that Further comprising: Determine a weighting factor based on quantization parameter values of the current block in the current frame and the corresponding block in the temporal source frame.

14. The method according to any one of claims 1-13, characterized in that, The filter includes at least one of a cross-component filter (CCF), an adaptive loop filter (ALF), a loop restoration filter, and a constrained directional enhancement filter within the loop (CDEF).

15. A video encoding method, applied to an encoder including a local decoder, characterized in that, The local decoder performs the following steps: Generate first reconstructed samples of a current block in a current frame; Obtain reconstructed samples in a temporal source frame different from the current frame; Apply a filter to at least the reconstructed samples in the temporal source frame to generate an offset; And Combine the first reconstructed samples of the current block with the offset to generate second reconstructed samples of the current block; Wherein the temporal source frame is an inter-coded frame that satisfies at least one of the following: A quantization parameter requirement, which indicates that the inter-coded frame has a lower QP value than the current frame; A temporal distance requirement, which indicates that the temporal distance between the inter-coded frame and the current frame is less than a threshold; and A reference picture requirement, which indicates that the inter-coded frame is a reference picture used for motion compensation during the reconstruction of the current frame.

16. A video decoding device, characterized in that, Comprising: A generation module for generating first reconstructed samples of a current block in a current frame; An acquisition module for obtaining reconstructed samples in a temporal source frame different from the current frame; An application module for applying a filter to at least the reconstructed samples in the temporal source frame to generate an offset; And A combination module for combining the first reconstructed samples of the current block with the offset to generate second reconstructed samples of the current block; Wherein the temporal source frame is an inter-coded frame that satisfies at least one of the following: A quantization parameter requirement, which indicates that the inter-coded frame has a lower QP value than the current frame; A temporal distance requirement, which indicates that the temporal distance between the inter-coded frame and the current frame is less than a threshold; and A reference picture requirement, which indicates that the inter-coded frame is a reference picture used for motion compensation during the reconstruction of the current frame.

17. A video decoding device, characterized in that, Comprising: One or more processors and one or more memories, wherein at least one program instruction is stored in the one or more memories and is loaded and executed by the one or more processors to implement the method for video decoding according to any one of claims 1-14.

18. A non-volatile computer-readable medium for storing program instructions, which when executed by a computer, cause the computer to execute the method for video decoding according to any one of claims 1-14 or the method for video encoding according to claim 15.

19. A method for storing a video bitstream, characterized in that, The video bitstream is decoded based on the method for video decoding according to any one of claims 1-14, or the video bitstream is generated according to the method for video encoding according to claim 15.

Images