Method and apparatus for video coding, and program

Abstract

To provide a method and apparatus for video coding, and a program.SOLUTION: Aspects of the disclosure include methods, apparatuses, and non-transitory computer-readable storage mediums for video encoding / decoding. An apparatus includes processing circuitry that determines directionality information of a restoration filter unit included in a video frame based on at least one of a constrained directional enhancement filter (CDEF) process and an intra prediction mode. The processing circuitry determines one of a plurality of sets of filter parameters of a restoration filter process based on the directionality information of the restoration filter unit. The processing circuitry performs the restoration filter process on the restoration filter unit based on the one of the plurality of sets of filter parameters. The processing circuitry reconstructs the video frame based on the filtered restoration filter unit.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] [Incorporating Related Applications] This application claims the benefit of priority to U.S. Patent Application No. 17 / 362,048, entitled "METHOD AND APPARATUS FOR VIDEO CODING," filed June 29, 2021, which claims the benefit of priority to U.S. Provisional Application No. 63 / 091,707, entitled "FEATURE INFORMATION REUSE FOR ENHANCED RESTORATION FILTERING," filed October 14, 2020. The entire disclosure of the prior application is incorporated by reference in its entirety.

[0002] [Technical field] This disclosure describes embodiments generally related to video coding. [Background technology]

[0003] The background discussion provided herein is intended to generally present the background of the present disclosure. The work of the inventors of the present application is not admitted explicitly or implicitly as prior art to the present disclosure to the extent that such work is described in this background section, nor are any aspects of the description that do not specifically qualify as prior art at the time of filing.

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a sequence of pictures, each with spatial dimensions of, for example, 1920x1080 luma samples and associated chroma samples. The sequence of pictures can have a fixed or variable picture rate (also informally known as frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luma sample resolution with a frame rate of 60 Hz) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0005] One goal of video encoding and decoding is to be able to reduce redundancy in the input video signal through compression. Compression can help reduce the bandwidth or storage space requirements, in some cases by more than one order of magnitude. Both lossless and lossy compression, as well as combinations of these, can be used. Lossless compression refers to techniques where an exact copy of the original signal can be restored from the compressed original signal. When lossy compression is used, the restored signal may not be identical to the original signal, but the distortion between the original and restored signals is small enough to make the restored signal useful for the intended application. For video, lossy compression is widely used. The amount of acceptable distortion depends on the application. For example, users of a particular consumer streaming application may tolerate higher distortion than users of a television distribution application. The achievable compression ratio can reflect higher acceptable distortion / acceptable distortion can result in higher compression ratios.

[0006] Video encoders and decoders may utilize techniques from a number of broad categories, including, for example, motion compensation, transformation, quantization, and entropy coding.

[0007] Video codec techniques can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. If all blocks of samples are coded in intra mode, the picture can be an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and therefore can be used as the first picture in a coded video bitstream and video session or as a still image. Samples of an intra block can be subjected to a transform, and the transform coefficients can be quantized before entropy coding. Intra prediction can be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and the smaller the AC coefficients after the transform, the fewer bits are needed at a given quantization step size to represent the block after entropy coding.

[0008] Conventional intra-coding, for example as known from MPEG-2 generation encoding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to predict intra-prediction from surrounding sample data and / or metadata, for example obtained during encoding and / or decoding of a previous block of data that is spatially adjacent and in decoding order. Such techniques are hereinafter referred to as "intra prediction" techniques. It should be noted that at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and does not use reference data from a reference picture.

[0009] There may be many forms of intra prediction. If one or more of such techniques are available in a given video coding technique, the technique used may be coded in intra prediction mode. In some cases, the mode may have sub-modes and / or parameters, which may be coded separately or may be included in the mode codeword. The codeword used for a given mode, sub-mode and / or parameter combination may affect the coding efficiency gain through intra prediction, which may in turn affect the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain intra prediction modes were introduced in H.264, improved in H.265, and further improved in newer coding techniques such as joint exploration model (JEM), versatile video coding (VVC) and benchmark set (BMS). A prediction block can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of the neighboring samples are copied to the prediction block according to the direction. The reference to the direction in use can be coded in the bitstream or it may be predicted itself.

[0011] Referring to FIG. 1A, shown at the bottom right is a subset of 9 known predictor directions from the 33 possible predictor directions (corresponding to the 33 angle modes of the 35 intra modes) of H.265. The point (101) where the arrows converge represents the sample to be predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] Still referring to FIG. 1A, at the top left is shown a square block (104) of 4×4 samples (indicated by a dashed bold line). The square block (104) contains 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. Since the size of the block is 4×4 samples, S44 is at the bottom right. Additionally, reference samples are shown following a similar numbering scheme. The reference samples are labeled with R and their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, the prediction samples are adjacent to the block being reconstructed, so there is no need for negative values ​​to be used.

[0013] Intra-picture prediction can work by copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling for this block indicating a prediction direction consistent with the arrow (102). That is, assume that the samples are predicted from one or more prediction samples to the upper right at an angle of 45 degrees from the horizontal. In this case, samples S41, S32, S23 and S14 are predicted from the same reference sample R05. Then sample S44 is predicted from reference sample R08.

[0014] In some cases, particularly when the orientation is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example by interpolation, to calculate the reference sample.

[0015] As video coding techniques develop, the number of possible directions increases. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), and at the time of disclosure, JEM / VVC / BMS can support up to 65 directions. Experiments have been carried out to identify the most likely directions, and certain techniques in entropy coding have been used to represent these more likely directions with a small number of bits, accepting a certain penalty for the less likely directions. Furthermore, in some cases, the direction itself can be predicted from the neighboring directions used in adjacent already decoded blocks.

[0016] FIG. 1B shows a schematic diagram (105) illustrating 65 intra prediction directions according to JEM, showing the increasing number of prediction directions over time.

[0017] The mapping of intra-prediction direction bits in a coded video bitstream to represent directions can vary across video coding techniques, ranging from simple direct mapping of prediction directions to complex adaptive schemes involving intra-prediction modes, codewords, most probable modes, and similar techniques. In all cases, however, there may be certain directions that are statistically less likely to occur in the video content than certain other directions. Since a goal of video compression is to reduce redundancy, in well-performing video coding techniques, these less likely directions are represented by a greater number of bits than the more likely directions.

[0018] Motion compensation is a lossy compression technique and can be associated with a technique in which blocks of sample data from a previously reconstructed picture or part thereof (reference picture) are used to predict a newly reconstructed picture or part thereof after being spatially shifted in a direction indicated by a motion vector (hereinafter referred to as MV). In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or it may have three dimensions, with the third dimension indicating the reference picture in use (the latter can indirectly be the temporal dimension).

[0019] In some video compression techniques, the MV applicable to a particular region of sample data can be predicted from other MVs, e.g., from MVs associated with other regions of sample data that are spatially adjacent to the region being restored and that precede that MV in decoding order. This can significantly reduce the amount of data required to encode the MV, thereby removing redundancy and increasing compression. For example, when encoding an input video signal derived from a camera (known as natural video), MV prediction can work effectively because there is a statistical possibility that regions larger than the region to which a single MV is applicable move in a similar direction and therefore can potentially be predicted using similar MVs derived from MVs of neighboring regions. As a result, the detected MV for a given region will be similar or identical to the MV predicted from the surrounding MVs, which, after entropy coding, can be represented with fewer bits than would be used to directly encode the MV. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, MV prediction itself can be lossy, e.g., due to rounding errors when computing a predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, “High Efficiency Video Coding”, December 2016). Among the many MV prediction mechanisms provided by H.265, a technique called “spatial merge” will be described in this specification.

[0021] Referring to Figure 1C, the current block (111) may contain samples that the encoder found during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of directly encoding its MV, the MV can be derived from metadata associated with one or more reference pictures, e.g., from the most recent reference picture (in decoding order) using MVs associated with any one of the five surrounding samples denoted A0, A1 and B0, B1, B2 (112-116, respectively). In H.265, MV prediction can use predictors from the same reference picture as the neighboring blocks use. Summary of the Invention

[0022] An aspect of the present disclosure provides an apparatus for video encoding / decoding. The apparatus includes a processing circuit that determines, based on at least one of a constrained directional enhancement filter (CDEF) process or an intra-prediction mode, directionality information of a reconstruction filter unit included in a video frame. The processing circuit determines one of a plurality of sets of filter parameters of the reconstruction filter process based on the directionality information of the reconstruction filter unit. The processing circuit performs a reconstruction filter process on the reconstruction filter unit based on the one of the plurality of sets of filter parameters. The processing circuit reconstructs the video frame based on the filtered reconstruction filter unit.

[0023] In one embodiment, the reconstruction filter unit includes one or more directional information units, and at least one of the CDEF process or intra prediction modes is performed on one of the one or more directional information units.

[0024] In one embodiment, each of the multiple sets of filter parameters of the reconstruction filter is associated with at least one directionality of the CDEF process.

[0025] In one embodiment, the processing circuit determines one of a plurality of sets of filter parameters for the reconstruction filter process based on the directional information and the block variance information of the reconstruction filter unit.

[0026] In one embodiment, the processing circuit determines one of a plurality of sets of filter parameters of the reconstruction filter process based on the directional information of the reconstruction filter unit and the filter strength of the CDEF process.

[0027] In one embodiment, the processing circuitry determines the directional information of the reconstruction filter units based on at least one of a majority vote or a consistency check of the directionalities in the reconstruction filter units.

[0028] In one embodiment, based on the fact that the reconstruction filter unit is not intra-coded and the neighboring block is intra-coded, the processing circuit determines the directional information of the reconstruction filter unit based on the intra-prediction mode performed on the neighboring block of the reconstruction filter unit.

[0029] In one embodiment, the processing circuitry performs a reconstruction filter process for the reconstruction filter unit based on directional information determined from the CDEF process that is consistent with directional information determined from the intra prediction mode.

[0030] In one embodiment, the restoration filter process is one of a Wiener filter process and a self-guided projection (SGRPRJ) filter process.

[0031] In one embodiment, the processing circuitry determines one of multiple sets of filter parameters for the reconstruction filter process based on a default set of filter parameters, a communicated index indicating a set of filter parameters, or one of the communicated sets of filter parameters.

[0032] An aspect of the present disclosure provides a method for video encoding / decoding, in which directional information of a reconstruction filter unit included in a video frame is determined based on at least one of a CDEF process or an intra-prediction mode. One of a plurality of sets of filter parameters of a reconstruction filter process is determined based on the directional information of the reconstruction filter unit. The reconstruction filter process is performed on the reconstruction filter unit based on the one of the plurality of sets of filter parameters. The video frame is reconstructed based on the filtered reconstruction filter unit.

[0033] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform any one or combination of methods for video decoding. [Brief description of the drawings]

[0034] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1A] FIG. 2 is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B] FIG. 2 is a diagram of an example intra-prediction direction. [Figure 1C]FIG. 2 is a schematic diagram of a current block and its surrounding spatial merge candidates in one example. [Diagram 2] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Diagram 3] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Figure 4] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Diagram 5] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 6] 4 shows a block diagram of an encoder according to another embodiment; [Figure 7] 4 shows a block diagram of a decoder according to another embodiment; [Figure 8] 1 illustrates an exemplary nominal angle according to one embodiment of the present disclosure. [Figure 9] 1 illustrates the locations of the top sample, left sample, and top-left sample for one pixel in a current block, according to one embodiment of the present disclosure. [Figure 10] 1 illustrates an exemplary recursive filter intra mode according to one embodiment of the present disclosure. [Figure 11] 1 illustrates some example directions in a constrained directional enhancement filter (CDEF) process according to some embodiments of the present disclosure. [Figure 12] 4 illustrates some example block partitioning according to some embodiments of the present disclosure. [Figure 13] 1 illustrates an example in which a directional unit block is merged into a filtering unit according to one embodiment of the present disclosure. [Figure 14] 13 illustrates another example in which a directional unit block is merged into a filtering unit according to an embodiment of the present disclosure. [Figure 15] 1 illustrates an example mapping of directional information between directional intra-prediction modes and main directions derived during the CDEF process, according to some embodiments of this disclosure. [Figure 16] 1 illustrates an exemplary flow chart according to one embodiment of the present disclosure. [Figure 17] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] [I. Video Decoder and Encoder Systems] FIG. 2 shows a simplified block diagram of a communication system (200) according to an embodiment of the present disclosure. The communication system (200) includes, for example, a plurality of terminal devices that can communicate with each other via a network (250). For example, the communication system (200) includes a first pair of terminal devices (210) and (220) interconnected via the network (250). In the example of FIG. 2, the first pair of terminal devices (210) and (220) perform a one-way transmission of data. For example, the terminal device (210) may encode video data (e.g., a stream of video pictures captured by the terminal device (210)) for transmission to another terminal device (220) via the network (250). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. The terminal device (220) may receive the encoded video data from the network (250), decode the encoded video data, reconstruct the video pictures, and display the video pictures according to the reconstructed video data. One-way data transmission may be common, such as in media presentation applications.

[0036] In another example, the communication system (200) includes a second pair of terminal devices (230) and (240) performing bidirectional transmission of encoded video data, such as may occur during a video conference. For the bidirectional transmission of data, in one example, each of the terminal devices (230) and (240) may encode video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (230) and (240) over the network (250). Each of the terminal devices (230) and (240) may also receive the encoded video data transmitted by the other of the terminal devices (230) and (240), decode the encoded video data to recover the video pictures, and display the video pictures on an accessible display device according to the recovered video data.

[0037] In the example of FIG. 2, the terminal devices (210), (220), (230), and (240) may be depicted as a server, a personal computer, and a smartphone, although the principles of the present disclosure are not limited thereto. The embodiments of the present disclosure may be applied to laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. The network (250) represents any number of networks that convey encoded video data between the terminal devices (210), (220), (230), and (240), including, for example, wired (hardwired) and / or wireless communication networks. The communication network (250) may exchange data in circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of the network (250) is not important to the operation of the present disclosure, unless otherwise described herein below.

[0038] 3 shows an arrangement of a video encoder and a video decoder in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter is equally applicable to other video-enabled applications including, for example, videoconferencing, digital TV, storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.), etc.

[0039] The streaming system may include a capture subsystem (313), which may include, for example, a video source (301) (e.g., a digital camera) that generates a stream of uncompressed video pictures (302). In one example, the stream of video pictures (302) includes samples taken by a digital camera. The stream of video pictures (302), depicted as a thick line to highlight its higher amount of data when compared to the encoded video data (304) (or encoded video bitstream), may be processed by an electronic device (320) that includes a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (304) (or encoded video bitstream (304)), depicted as a thin line to highlight its lower amount of data when compared to the stream of video pictures (302), may be stored in a streaming server (305) for future use. One or more streaming client subsystems, such as client subsystems (306) and (308) in FIG. 3, may access the streaming server (305) to obtain copies (307) and (309) of the encoded video data (304). The client subsystem (306) may include, for example, a video decoder (310) within an electronic device (330). The video decoder (310) decodes an input copy (307) of the encoded video data and generates an output stream (311) of video pictures that can be rendered on a display (312) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (304), (307), and (309) (e.g., a video bitstream) may be encoded according to a particular video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265.In one example, a developing video coding standard is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0040] It should be noted that the electronic devices (320) and (330) may include other components (not shown). For example, the electronic device (320) may include a video decoder (not shown), and the electronic device (330) may include a video encoder (not shown).

[0041] 4 shows a block diagram of a video decoder (410) according to one embodiment of the present disclosure. The video decoder (410) may be included in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., a receiving circuit). The video decoder (410) may be used in place of the video decoder (310) in the example of FIG. 3.

[0042] The receiver (431) may receive one or more coded video sequences to be decoded by the video decoder (410), and in the same or other embodiments, may receive one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel (401), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (431) may receive the coded video data together with other data (e.g., coded audio data and / or auxiliary data streams), which may be forwarded to respective usage entities (not shown). The receiver (431) may separate the coded video sequences from the other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420), hereinafter referred to as the parser (420). In certain applications, the buffer memory (415) is part of the video decoder (410). In other cases, it may be external to the video decoder (410) (not shown). In still other cases, there may be a buffer memory (not shown) external to the video decoder (410), e.g., to prevent network jitter, plus another buffer memory (415) internal to the video decoder (410), e.g., to handle playback timing. If the receiver (431) is receiving data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isochronous network, the buffer memory (415) may not be needed or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (415) may be needed and may be relatively large, advantageously adaptively sized, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (410).

[0043] The video decoder (410) may include a parser (420) for recovering symbols (421) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (410), potentially including information for controlling a rendering device such as a rendering device (412) (e.g., a display screen), which may not be an integral part of the electronic device (430) as shown in FIG. 4, but may be coupled to the electronic device (430). The rendering device control information may be in the form of Supplemental Enhancement Information (SEI) (SEI message) or Video Usability Information (VUI) parameter set fragments (not shown). The parser (420) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (420) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroups may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transformation unit (TU), a prediction unit (PU), etc. The parser (420) may also extract information such as transform coefficients, quantization parameter values, MVs, etc. from the coded video sequence.

[0044] The parser (420) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (415) to generate symbols (421).

[0045] The recovery of the symbols (421) may involve several different units, depending on the type of coded video picture or portion thereof (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors. Which units are involved and how may be controlled by subgroup control information parsed from the coded video sequence by the parser (420). The flow of such subgroup control information between the parser (420) and the following units is not shown for clarity.

[0046] In addition to the functional blocks described above, the video decoder (410) may be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0047] The first unit is a scalar / inverse transform unit (451), which receives the quantized transform coefficients as symbols (421) from the parser (420), along with control information (including which transform to use, block size, quantization factor, quantization scaling matrix, etc.). The scalar / inverse transform unit (451) may output blocks containing sample values ​​that can be input to an aggregator (455).

[0048] In some cases, the output samples of the scaler / inverse transform (451) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) generates a block of the same size and shape of the block being reconstructed using surrounding already reconstructed information retrieved from the current picture buffer (458). The current picture buffer (458) buffers, for example, a partially reconstructed and / or a fully reconstructed current picture. In some cases, the aggregator (455) adds, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451).

[0049] In other cases, the output samples of the scaler / inverse transform unit (451) may relate to an inter-coded and potentially motion-compensated block. In such cases, the motion compensation prediction unit (453) may access the reference picture memory (457) to retrieve samples used for prediction. After motion compensating the retrieved samples according to the symbols (421) associated with the block, these samples may be added by the aggregator (455) to the output of the scaler / inverse transform unit (451) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (457) available to the motion compensation prediction unit (453) from which the motion compensation prediction unit (453) retrieves prediction samples may be controlled by the MV, for example in the form of the symbols (421) that may have X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory (457) when sub-sample accurate MVs are used, MV prediction mechanisms, etc.

[0050] The output samples of the aggregator (455) may be subjected to various loop filtering techniques in a loop filter unit (456). The video compression techniques may include in-loop filter techniques, controlled by parameters contained in the coded video sequence (also called coded video bitstream) and made available to the loop filter unit (456) as symbols (421) from the parser (420), which may be responsive to meta-information obtained during decoding of previous portions of the coded picture or coded video sequence (in decoding order), as well as to previously reconstructed loop filtered sample values.

[0051] The output of the loop filter unit (456) may be a sample stream that may be output to a rendering device (412) and also stored in a reference picture memory (457) for use in future inter-picture prediction.

[0052] Once a particular coded picture is fully reconstructed, it may be used as a reference picture for future prediction. For example, once a coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (420)), the current picture buffer (458) may become part of the reference picture memory (457), and a new current picture buffer may be reallocated before beginning reconstruction of the subsequent coded picture.

[0053] The video decoder (410) may perform decoding operations according to a given video compression technique in a standard, such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. In particular, the profile may select certain tools from all tools available in the video compression technique or standard as the only tools available for use in that profile. Also, what is required for compliance is that the complexity of the coded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited through a Hypothetical Reference Decoder (HRD) specification and metadata about HRD buffer management conveyed in the coded video sequence.

[0054] In one embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (410) to properly decode the data and / or to more accurately recover the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0055] 5 shows a block diagram of a video encoder (503) according to one embodiment of the present disclosure. The video encoder (503) is included in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmission circuit). The video encoder (503) may be used in place of the video encoder (303) in the example of FIG. 3.

[0056] The video encoder (503) may receive video samples from a video source (501) (not part of the electronic device (520) in the example of FIG. 5), which may capture video images to be encoded by the video encoder (503). In other examples, the video source (501) is part of the electronic device (520).

[0057] The video source (501) may provide a source video sequence to be encoded by the video encoder (503) in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media presentation system, the video source (501) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (501) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc., being used. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0058] According to one embodiment, the video encoder (503) may encode and compress pictures of a source video sequence into an encoded video sequence (543) in real-time or under any other time constraint required by an application. Achieving an appropriate encoding rate is one function of the controller (550). In some embodiments, the controller (550) controls and is operatively coupled to other functional units, as described below. Coupling is not shown for clarity. Parameters set by the controller (550) may include rate control related parameters (picture skip, quantization, lambda values ​​for rate distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum MV allowed reference region, etc. The controller (550) may be configured with other appropriate functions associated with the video encoder (503) optimized for a particular system design.

[0059] In some embodiments, the video encoder (503) is configured to operate in an encoding loop. As a very simplified description, in one example, the encoding loop may include a source coder (530) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be encoded and reference pictures) and a (local) decoder (533) embedded in the video encoder (503). The decoder (533) reconstructs the symbols to generate sample data as the (remote) decoder would generate them (so that any compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (534). Since the decoding of the symbol stream produces bit-wise accurate results independent of the location of the decoder (local or remote), the contents in the reference picture memory (534) are also bit-wise accurate between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder "sees" when using the prediction during decoding. This basic principle of reference picture synchronization (including the resulting drift when synchronization cannot be maintained, for example, due to channel errors) is used in some related technologies as well.

[0060] The operation of the "local" decoder (533) may be the same as a "remote" decoder, such as the video decoder (410), which has already been described in detail above in connection with Figure 4. However, with brief reference to Figure 4, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (545) and parser (520) may be lossless, the entropy decoding portion of the video decoder (410), including the buffer memory (515) and parser (520), may not be fully implemented in the local decoder (533).

[0061] An observation that can be made at this point is that any decoder technique, other than analysis / entropy decoding, present in the decoder must necessarily be present in the corresponding encoder in substantially the same functional form. For this reason, the subject matter of the disclosure focuses on the decoder operation. A description of the encoder techniques can be omitted, as they are the inverse of the decoder techniques, which are described generically. Only in certain areas are more detailed descriptions necessary, and are provided below.

[0062] In some examples, during operation, the source coder (530) may perform motion-compensated predictive encoding, which predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as “reference pictures.” In this manner, the encoding engine (532) encodes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0063] The local video decoder (533) may decode the encoded video data of pictures that may be designated as reference pictures based on the symbols generated by the source coder (530). The operation of the encoding engine (532) may advantageously be a lossy process. If the encoded video data can be decoded in a video decoder (not shown in FIG. 5), the reconstructed video sequence may be a replica of the source video sequence, typically with some errors. The local video decoder (533) may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in a reference picture cache (534). In this way, the video encoder (503) may locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures (without transmission errors) obtained by the far-end video decoder.

[0064] The predictor (535) may perform a prediction search for the coding engine (532). That is, for a new picture to be coded, the predictor (535) may search the reference picture memory (534) for sample data (as candidate reference pixel blocks) or specific metadata (reference picture MV, block shape, etc.), which may serve as suitable prediction references for the new picture. The predictor (535) may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, the input picture determined by the search results obtained by the predictor (535) may have prediction references drawn from multiple reference pictures stored in the reference picture memory (534).

[0065] The controller (550) may manage the encoding operations of the source coder (530), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0066] The output of all the above functional units may undergo entropy coding in an entropy coder (545), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0067] The transmitter (540) may buffer the encoded video sequence produced by the entropy coder (545) and prepare it for transmission over a communication channel (560), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (540) may merge the encoded video data from the video coder (503) with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (not shown).

[0068] A controller (550) may manage the operation of the video encoder (503). During encoding, the controller (550) may assign each coded picture a particular coded picture type, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0069] An Intra picture (I-picture) may be one that can be coded and decoded without using other pictures in a sequence as a source of prediction. Some video codecs allow different types of Intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will recognize these variations of I-pictures and their respective uses and characteristics.

[0070] A predictive picture (P picture) may be one that can be encoded and decoded using intra prediction or inter prediction, using at most one MV and a reference index to predict the sample values ​​of each block.

[0071] Bidirectionally predicted pictures (B-pictures) may be encoded and decoded using intra- or inter-prediction, using up to two MVs and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0072] In general, a source picture may be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8 or 16x16 samples, respectively) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective picture of the block. For example, blocks of an I-picture may be non-predictively coded or may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P-picture may be predictively coded via spatial or temporal prediction with reference to the reference picture coded one step before. Blocks of a B-picture may be predictively coded via spatial or temporal prediction with reference to the reference picture coded one or two steps before.

[0073] The video encoder (503) may perform encoding operations according to a given video encoding technique or standard, such as ITU-T Rec. H.265. In its operations, the video encoder (503) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard being used.

[0074] In one embodiment, the transmitter (540) may transmit additional data along with the coded video. The source coder (530) may include such data as part of the coded video sequence. The additional data may include other types of redundant data, such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0075] A video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being coded / decoded, called the current picture, is divided into blocks. If a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture may be coded by a vector called a MV. The MV refers to a reference block in the reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0076] In some embodiments, bidirectional prediction techniques may be used in inter-picture prediction. According to bidirectional prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both preceding in decoding order to the current picture in the video (but may be past and future, respectively, in display order). A block in the current picture may be coded by a first MV that points to a first reference block in the first reference picture and a second MV that points to a second reference block in the second reference picture. A block may be predicted by a combination of the first reference block and the second reference block.

[0077] Furthermore, to improve coding efficiency, merge mode techniques can be used in inter-picture prediction.

[0078] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be partitioned into one CU of 64×64 pixels, four CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PU, prediction unit) depending on temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB, prediction block) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, a prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0079] 6 shows a diagram of a video encoder (603) according to another embodiment of the present disclosure. The video encoder (603) is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video picture in a sequence of video pictures and to encode the processed block into a coded picture that is part of a coded video sequence. In one example, the video encoder (603) is used in place of the video encoder (303) of the example of FIG. 3.

[0080] In an HEVC example, the video encoder (603) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8x8 samples. The video encoder (603) determines whether the processing block is best coded using an intra mode, an inter mode, or a bidirectional predictive mode, e.g., using rate-distortion optimization. If the processing block is coded in an intra mode, the video encoder (603) may use intra prediction techniques to code the processing block into a coded picture. If the processing block is coded in an inter mode or a bidirectional predictive mode, the video encoder (603) may use inter prediction techniques or bidirectional prediction techniques, respectively, to code the processing block into a coded picture. In certain video coding techniques, the merge mode may be an inter picture prediction sub-mode in which the MV is derived from one or more MV predictors without the benefit of any coded MV components other than the MV predictors. In certain other video coding techniques, there may be motion vector components applicable to the block of interest. In one example, the video encoder (603) includes other components, such as a mode decision module (not shown) for determining the mode of the processing blocks.

[0081] In the example of FIG. 6, the video encoder (603) includes an inter-encoder (630), an intra-encoder (622), a residual calculator (623), a switch (626), a residual encoder (624), an overall controller (621), and an entropy encoder (625) coupled together as shown in FIG.

[0082] The inter-encoder (630) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-prediction information (e.g., a description of redundant information due to an inter-coding technique, MV, merge mode information), and calculate an inter-prediction result (e.g., a prediction block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that is decoded based on the encoded video information.

[0083] The intra encoder (622) is configured to receive samples of a current block (e.g., a processing block), possibly compare the block with previously encoded blocks in the same picture, generate quantized coefficients after transformation, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). In one example, the intra encoder (622) also calculates an intra prediction result (e.g., a prediction block) based on the intra prediction information and a reference block in the same picture.

[0084] The global controller (621) is configured to determine global control data and control other components of the video encoder (603) based on the global control data. In one example, the global controller (621) determines a mode of the block and provides a control signal to the switch (626) based on the mode. For example, if the mode is an intra mode, the global controller (621) controls the switch (626) to select an intra mode result to be used by the residual calculator (623) and controls the entropy encoder (625) to select intra prediction information and include the intra prediction information in the bitstream. If the mode is an inter mode, the global controller (621) controls the switch (626) to select an inter prediction result to be used by the residual calculator (623) and controls the entropy encoder (625) to select inter prediction information and include the inter prediction information in the bitstream.

[0085] The residual calculator (623) is configured to calculate a difference (residual data) between the received block and a prediction result selected from the intra-encoder (622) or the inter-encoder (630). The residual encoder (624) operates on the residual data and is configured to encode the residual data to generate transform coefficients. In one example, the residual encoder (624) is configured to transform the residual data from a spatial domain to a frequency domain to generate transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be used by the intra-encoder (622) and the inter-encoder (630) as appropriate. For example, the inter-encoder (630) may generate decoded blocks based on the decoded residual data and the inter-prediction information, and the intra-encoder (622) may generate decoded blocks based on the decoded residual data and the intra-prediction information. The decoded blocks are suitably processed to generate a decoded picture, which may be buffered in a memory circuit (not shown) and used as a reference picture in some examples.

[0086] The entropy encoder (625) is configured to format the bitstream to include the coded block. The entropy encoder (625) is configured to include various information according to a suitable standard, such as HEVC. In one example, the entropy encoder (625) is configured to include global control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other suitable information in the bitstream. It should be noted that, according to the disclosed subject matter, when encoding a block in a merged sub-mode of either the inter mode or the bi-prediction mode, the residual information is not present.

[0087] 7 shows a diagram of a video decoder (710) according to another embodiment of the present disclosure. The video decoder (710) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (710) is used in place of the video decoder (310) of the example of FIG. 3.

[0088] In the example of FIG. 7, the video decoder (710) includes an entropy decoder (771), an inter decoder (780), a residual decoder (773), a reconstruction module (774), and an intra decoder (772) coupled together as shown in FIG.

[0089] The entropy decoder (771) may be configured to recover from the coded picture certain symbols that represent the syntax elements of which the coded picture is composed. Such symbols may include, for example, prediction information (e.g., intra-mode, inter-mode, bi-predictive mode, the latter two in merged submode or other submode, etc.) that can identify the mode in which the block is coded, the specific samples or metadata used for prediction by the intra decoder (772) or the inter decoder (780), respectively, such as intra prediction information or inter prediction information, such as residual information in the form of quantized transform coefficients, etc. In one example, if the prediction mode is an inter mode or bi-predictive mode, the inter prediction information is provided to the inter decoder (780), and if the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (772). The residual information may undergo inverse quantization and is provided to the residual decoder (773).

[0090] The inter decoder (780) is configured to receive the inter prediction information and to generate inter prediction results based on the inter prediction information.

[0091] The intra decoder (772) is configured to receive intra prediction information and to generate a prediction result based on the intra prediction information.

[0092] The residual decoder (773) is configured to perform inverse quantization to extract inverse quantized transform coefficients, and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require certain control information, including a quantizer parameter (QP), which may be provided by the entropy decoder (771) (data path not shown as this may be only low volume control information).

[0093] The reconstruction module (774) is configured to combine, in the spatial domain, the residual output by the residual decoder (773) and the prediction result (possibly output by an inter-prediction module or an intra-prediction module) to form a reconstruction block, which may be part of a reconstructed picture or part of a reconstructed video. It should be noted that other suitable operations, such as a debooking operation, may be performed to improve the visual quality.

[0094] It should be noted that the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using any suitable technology. In one embodiment, the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using one or more processors executing software instructions.

[0095] [II. Intra Prediction] In some related examples such as VP9, ​​eight directional modes are supported, corresponding to angles from 45 degrees to 207 degrees. In some related examples such as AV1 (AOMedia Video 1), the directional intra mode is extended to a set of angles with finer granularity to take advantage of more diverse spatial redundancy in directional textures. The original eight angles are slightly modified and called nominal angles, and these eight nominal angles are named V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED and D67_PRED.

[0096] FIG. 8 shows an exemplary nominal angle according to one embodiment of the present disclosure. Each nominal angle may be associated with seven finer angles, so there may be a total of 56 directional angles in some related examples such as AV1. The prediction angle is represented by the nominal intra angle plus an angle delta, which is derived by multiplying a coefficient (ranging from -3 to 3) with a step size of 3 degrees. To implement the directional prediction mode in AV1 through a general method, all 56 directional intra prediction angles in AV1 can be implemented with a unified directional predictor that projects each pixel to a reference sub-pixel position and interpolates the reference sub-pixel by a 2-tap bilinear filter.

[0097] In some related examples such as AV1, there are five non-directional smooth intra prediction modes, which are DC, PAETH, SMOOOTH, SMOOTH_V and SMOOTH_H. For DC prediction, the average of the left neighboring sample and the upper neighboring sample is used as the predictor of the block to be predicted. For PAETH prediction, the upper reference sample, the left reference sample and the upper left reference sample are first taken, and then the closest value to (upper+left-upper left) is set as the predicted value of the pixel to be predicted.

[0098] 9 shows the locations of the top sample, the left sample, and the top-left sample for one pixel in the current block according to one embodiment of the present disclosure. For SMOOTH, SMOOTH_V, and SMOOTH_H modes, the block is predicted using quadratic interpolation in the vertical or horizontal direction, or an average in both directions.

[0099] FIG. 10 illustrates an exemplary recursive filter intra mode according to one embodiment of the present disclosure.

[0100] To capture the decaying spatial correlation with the reference on the edge, FILTER INTRA mode is designed for luma blocks. In AV1, five filter intra modes are defined, each represented by a set of eight 7-tap filters that reflect the correlation between the pixels of a 4×2 patch and the seven neighbors adjacent to the patch. For example, the weighting coefficients of the 7-tap filters are position-dependent. As shown in FIG. 10, an 8×8 block is divided into eight 4×2 patches, which are denoted by B0, B1, B2, B3, B4, B5, B6, and B7. For each patch, the seven neighbors, denoted by R0-R6, are used to predict the pixels in the respective patch. For patch B0, all neighbors have already been restored. However, for other patches, if all neighbors have not been restored, the predicted values ​​of the nearest neighbors are used as the reference values. For example, since all neighbors of patch B7 have not been restored, the predicted samples of patch B7's neighbors (i.e., B5 and B6) are used instead.

[0101] For the chroma components, a chroma-only intra prediction mode, called CfL (chroma from luma) mode, models chroma pixels as linear functions of concurrent reconstructed luma pixels. CfL prediction may be expressed as follows: CfL(α)=α×L AC +DC formula (1) Here, L ACdenotes the AC contribution of the luma component, α denotes a parameter of the linear model, and DC denotes the DC contribution of the chroma component. In one example, the restored luma pixels are subsampled to the chroma resolution and then the mean is subtracted to form the AC contribution. Instead of requiring the decoder to calculate scaling parameters to approximate the chroma AC components from the AC contributions, as in some related examples, the CfL mode in AC1 determines the parameters α based on the original chroma pixels and conveys these in the bitstream. This reduces the decoder complexity and results in a more accurate prediction. As for the DC contributions of the chroma components, they are calculated using the intra-DC mode, which is sufficient for most chroma content and has a mature and fast implementation.

[0102] [III. Loop Filtering] In some related examples such as AV1, three in-loop filters may be applied to the reconstructed frames, in this order: deblocking, a constrained directional enhancement filter (CDEF), and a loop-restoration filter, which include a Wiener filter and a self-guided projection (SGRPRJ) filter, one of which may be adaptively selected.

[0103] Deblocking may be applied across transform block boundaries to reduce blocky artifacts caused by quantization of transform coefficients. In some examples, 4-, 8-, and 14-tap finite impulse response (FIR) filters may be used for luma blocks, and 4- and 6-tap FIR filters may be used for chroma blocks.

[0104] The filter length is initially determined by the minimum transform block size at the boundary. A conditional check may be performed using variance to avoid blurring the actual edges. In addition, a flatness check may also be incorporated to finally determine the filter length.

[0105] CDEF is a nonlinear deringing filter applied along directional features detected in a target region. In some related examples, an 8×8 region is the unit size on which CDEF is performed. Standard direction detection may be performed as shown in FIG. 11, which shows some exemplary directions in the CDEF process.

[0106] In FIG. 11, a candidate direction d (0 to 7) that minimizes the following quantity may be selected as the main direction.

[0107]

number

[0108]

number

[0109]

number

[0110]

number

[0111]

number

[0112] When applying filtering, each 64x64 block in a tile may select one of the presets, and filtering may be performed on each 8x8 unit within each 64x64 block.

[0113] Note that during the decoding process, several variables related to the signal characteristics of local regions may be parsed or derived from the bitstream. These variables include dir, var, priStr, secStr, and damping. The variable dir represents the main edge direction of the 8x8 block. The variable var represents the variance of the signal values ​​within the 8x8 block and is defined as the normalized difference between the cost in the main direction and the cost along the direction orthogonal to the main direction. The variable priSt represents the primary filter strength S of the 64x64 block containing the 8x8 filtering unit. p The variable secStr represents the secondary filter strength S of a 64x64 block that contains an 8x8 filtering unit. s The variable damping represents the damping parameter D of a 64×64 block containing 8×8 filtering units. These values ​​may be obtained separately for the luma and chroma channels.

[0114] After the deblocking and CDEF processes are performed, in some related examples such as AV1, mutually exclusive application of two types of restoration filters may be performed. The two types of restoration filters include a Wiener filter and an SGRPRJ filter. A square loop-restoration unit (LRU) size may be selected from 64×64 to 256×256.

[0115] In the Wiener filter, the quality of each reconstructed pixel in a coded frame can be improved through non-causal filtering using neighboring pixels in a W×W window around the respective pixel. The 2D filter taps of the Wiener filter are denoted as F and are determined as follows: F=H -1 M formula (7) Here, H=E[XX T ] is the autocovariance of x, and W 2 contains samples, M=E[YX T ] is the cross-correlation between x and the original source sample y.

[0116] In some related examples, such as AV1, the separability of F and the symmetry and normalization of the filter coefficients of the Wiener filter may be imposed as constraints. 2 The vector x1 may be defined as follows: F=column_vectorize[ab T ] Formula (8) where a and b are W×1 vertical and horizontal filters such that a(i)=a(W−1-i), b(i)=b(W−1-i) and Σa(i)=Σb(i)=1, for i=0,1,...,r−1. The coefficient vectors a and b may be searched in the encoder and coded into the bitstream.

[0117] In SGRPRJ filtering, we convert the degraded image x into a simple restored version as follows:

[0118]

number

[0119]

number

[0120] More specifically, the local mean (μ) and variance (σ) of a pixel within a (2r+1) × (2r+1) window around the pixel 2 ) may be calculated and each pixel x may be filtered as follows:

[0121]

number

[0122] Given two simple restorations X1 and X2 from a degraded image X according to equation (9), the final output X r The following subspace projections are performed to construct X r =X+α(X1-X)+β(X2-X) Equation (11) Given X, X1, X2 and a source Y, the encoder may calculate α and β as follows: [α β] T =(A T A) -1 A T b Equation (12) Here, A={X1-X, X2-X} and b=YX.

[0123] The encoder may then send a 6-tuple (r1, e1, r2, e2, α, β) for each LRU.

[0124] [IV. Reusing feature information for enhancement restoration filtering] In some relevant examples, such as AV1, the Wiener filter may be performed in square-sized units from 64x64 to 256x256 by uniformly dividing the frame / tile into LRUs. In one example, the filter coefficients of the Wiener filter may be obtained by assuming that the signal statistics are stationary. It is therefore desirable to classify the target region of filtering into one of the classified statistical types, for which the stationarity assumption can reasonably hold. Possible approaches to classify the target region include using quantities such as local variance or edge information. These quantities themselves or the associated categorical information may be computed at the decoder or conveyed in the bitstream, but may be costly in terms of computation or bitrate overhead.

[0125] In some related examples such as AV1, the SGRPRJ filter may be implemented on square-sized units ranging from 64x64 to 256x256 by uniformly dividing the frame / tile into LRUs. In the SGRPRJ filter, a form of simple edge-preserving filtering is performed to construct a simple reconstructed image using a fixed radius and noise parameter pair for each LRU. In addition, fixed projection parameters α and β may be used for each LRU as weighting coefficients in the error image to form the final reconstruction. However, regions in the error image may have different statistical properties reflecting local signal features such as edges and textures. Thus, if a single set of SGRPRJ filter parameters such as radius, noise parameters, α and β is used or estimated over an LRU region covering pixels with widely varying signal statistics, the quality of the final reconstruction may be compromised. On the other hand, incorporating signal classification for better adaptability may impose essentially the same challenges as in the case of Wiener filtering in terms of additional cost in computational load or bitrate overhead.

[0126] The present disclosure includes methods for improving the performance of reconstruction filtering techniques by reusing feature information, such as directional information derived from the CDEF process and / or intra-prediction modes. For example, adaptive reconstruction filtering techniques may effectively reuse signal features and statistics already available at the decoder.

[0127] In this disclosure, a restoration filter (or filtering) process may be defined as a filtering process used to operate on a noisy image and estimate a clean, original image based on the noisy image. A restoration filter process may include a process used to blur an image or an inverse process used to reverse blur an image. Examples of a restoration filter process include, but are not limited to, a Wiener filtering process and an SGRPRJ filtering process. A restoration filter (or filtering) unit is the domain in which the restoration filter process is performed.

[0128] In this disclosure, a directional information unit may be defined as a group of pixels having a specified shape and size, and may provide the main direction of a feature represented by the pixel values ​​of the group of pixels. In an example such as AV1, each directional information unit in the CDEF process may be an 8x8 block. The main direction and variance values ​​of the pixels in each 8x8 block may be derived in a standard manner. In other examples, directional intra-prediction modes in AV1 may provide such information in units of various shapes and sizes corresponding to intra-prediction blocks.

[0129] According to aspects of the present disclosure, the directional information derived at the decoder may be reused to estimate the presence and directionality of boundary edges for a reconstruction filter, such as a Wiener filter or an SGRPRJ filter. For example, the directional information may be derived from a CDEF process.

[0130] According to some embodiments, the shape and size of the reconstruction filtering unit (e.g., Wiener or SGRPRJ filtering unit) may be defined using multiple available directional information units (e.g., 8x8 blocks used in CDEF direction detection and filtering in AV1). In this way, finer granularity of directional adaptability can be achieved. Thus, the reconstruction filtering process may be performed with a unit size smaller than one of the fixed square types of 64x64, 128x128 or 256x256 used in some related examples such as AV1.

[0131] In one embodiment, the size of the restoration filtering unit may be, for example, the same as the LRU size defined in AV1.

[0132] In one embodiment, the size of the restoration filtering unit may be the same as the size of the directionality information unit (eg, 8×8).

[0133] In one embodiment, the restoration filtering unit may be further divided into square, rectangular, T-shaped or four-way sub-LRUs from a given LRU size, similar to or consistent with the division in some related embodiments, such as the division in AV1 as shown in FIG. 12.

[0134] In one embodiment, blocks each having a size of a directional information unit (e.g., 8×8) may be merged by following various scan orders to form a filtering unit. FIG. 13 shows an example in which blocks each having a size of an 8×8 directional unit are merged in a raster scan order into a filtering unit each having a size of 32×8. For example, four 8×8 directional unit blocks (1301)-(1304) may be merged into a 32×8 filtering unit block (1310), and four 8×8 directional unit blocks (1305)-(1308) may be merged into a 32×8 filtering unit block (1320).

[0135] In one embodiment, blocks each having a size of a directional information unit (e.g., 8×8) and having similar directionality may be merged by the following various scan orders to form a filtering unit. FIG. 14 shows an example in which blocks each having a size of an 8×8 directional unit are merged into a filtering unit of variable size in a raster scan order. The sizes of the filtering unit include 8×8, 16×8, and 32×8. For example, one 8×8 directional unit block (1401) may be an 8×8 filtering unit block (1410), two 8×8 directional unit blocks (1403)-(1404) may be merged into a 16×8 filtering unit block (1420), and four 8×8 directional unit blocks (1405)-(1408) may be merged into a 32×8 filtering unit block (1430).

[0136] According to some embodiments, each available directionality of the CDEF process may be used directly as a class index for a signal class for which a unique set of reconstruction filter shapes and sizes can be defined, i.e., the selection of the reconstruction filter may depend on the available directionality of the CDEF process.

[0137] In one embodiment, a solution to an equation in the calculation of the reconstruction filter, such as equation (7), may be applied for each signal class.

[0138] In one embodiment, different shaped 2D filters with different numbers of filter taps for each signal class may be used, with or without symmetry.

[0139] In one embodiment, the separability of the 2D filter (separable or non-separable filter) may depend on the available directionality of the CDEF process.

[0140] In one embodiment, multiple directions that are out of the available directions of the CDEF process may be merged into a single class, resulting in a reduction in the number of directional classes. For each merged class, a unique set of reconstruction filter shapes and sizes may be defined.

[0141] According to some embodiments, in addition to directionality, block variance information may be combined to further refine the directionality-based classes.

[0142] In one embodiment, the directional information unit with a size of 8×8 in the CDEF process may be further classified into different subclasses. The classification may be based on the variance value of the directional information unit. For example, if the number of classes based on directionality is 5 and the number of classes based on variance is 3, there may be 15 signal classes, for which a set of restoration filters (e.g., Wiener or SGRPRJ) can be designed, respectively.

[0143] According to some embodiments, in addition to the directionality, the filter strength may be combined to determine the signal class of the reconstruction filtering unit. For example, the transmitted primary CDEF filter strength and secondary CDEF filter strength may be combined to determine the signal class of the reconstruction filtering unit. Different filter strength presets selected by the encoder may indicate different signal characteristics of the target region.

[0144] In one embodiment, one of the first and second filter strength presets signaled in the bitstream may be directly used as the other dimension of the signal class index. For example, if the number of classes based on directionality is 5 and the number of classes based on filter strength presets is 4, there may be 20 signal classes for which a set of reconstruction filters (e.g., Wiener or SGRPRJ) can be designed, respectively.

[0145] According to some embodiments, a directional majority vote or consistency check included in the reconstruction filtering unit may be performed to determine the filter class.

[0146] In one embodiment, if the size and shape of the reconstruction filtering unit are fixed and the number of directional information units (e.g., 8×8 for CDEF) included in the reconstruction filtering unit is greater than a predetermined number, a majority vote or consistency check of the directionalities included in the reconstruction filtering unit may be performed to determine the filter class.

[0147] In one embodiment, in case of majority vote, the most frequent directionality among the available and possibly merged directionality may be selected. In one example, a certain margin may be set between the first and second most frequent directionality.

[0148] In one embodiment, before taking the majority vote, it is determined whether the number of classes in the reconstruction filtering unit is greater than a predetermined number: if true, an inconsistency may be declared and either an explicit signal for the reconstruction filtering unit or the use of a smaller reconstruction filtering unit may be selected.

[0149] According to some embodiments, the same scheme and directional information used for the restoration filtering unit in the luma component may be used for the restoration filtering unit in the chroma components if such information is only available for the luma component from the CDEF process.

[0150] According to some embodiments, the restoration filtering unit in a chroma component may use its own filter strengths, such as preset values ​​of primary and secondary filter strengths included in the chroma component, if such information is available for the chroma component.

[0151] According to some embodiments, the restoration filtering unit in a chroma component may use its own variance information if such information is available for the chroma component.

[0152] In one embodiment, when the CDEF process is turned off and the restoration filter is turned on, the direction search process of the CDEF process described in Section III (loop filtering section) may be applied to derive the presence and directionality of the boundary edges of the restoration filtering unit.

[0153] In one embodiment, when the CDEF process is turned off and the reconstruction filter is turned on, a default signal class is selected, or alternatively, explicit signaling of the filter class index may be performed.

[0154] According to aspects of the disclosure, directional information indicated by the intra-prediction modes available at the decoder may be reused as a guide for labeling signal classes in a reconstruction filtering unit (e.g., a Wiener or SGRPRJ filtering unit), each of which may define a unique set of reconstruction filter shapes and sizes. In some embodiments, such directional information may be provided by the encoder for intra-prediction at various unit sizes.

[0155] According to some embodiments, the shape and size of the reconstruction filtering unit may be defined using multiple available directional information units (e.g., 8x8 for directional intra-prediction units in AV1). In this way, finer-grained directional adaptability can be achieved. Thus, the reconstruction filtering process may be performed with a unit size smaller than one of the fixed square types of 64x64, 128x128 or 256x256 used in some related examples such as AV1.

[0156] In one embodiment, the size of the restoration filtering unit may be, for example, the same as the LRU size defined in AV1.

[0157] In one embodiment, the size of the restoration filtering unit may be the same as the size of the directionality information unit (eg, 8×8).

[0158] In one embodiment, the restoration filtering unit may be further divided into square, rectangular, T-shaped or four-way sub-LRUs from a given LRU size, similar to or consistent with the division in some related embodiments, such as the division in AV1 as shown in FIG. 12.

[0159] In one embodiment, blocks each having a size of a directional information unit (e.g., 8×8) may be merged by following various scanning orders to form a filtering unit, as shown in FIG. 13.

[0160] In one embodiment, blocks having the size of a directionality information unit (e.g., 8×8) and similar directionality may be merged by the following various scan orders to form a filtering unit, as shown in FIG. 14: According to some embodiments, a fixed number of signal classes may be defined, and blocks of each signal class may have intra-prediction modes with similar directionality. A unique set of reconstruction filter shapes and sizes may be defined for each signal class. If non-angular intra modes such as SMOOTH (including SMOOTH, SMOOTH_H, SMOOOTH_V modes), Paeth predictor, or DC mode are used, each or combination of the non-angular intra modes may be associated with its own signal class.

[0161] In one embodiment, each of the eight nominal angles, along with the associated seven possible delta angles in AV1, may be grouped together to form a total of eight directional classes.

[0162] In one embodiment, the directional class may depend on both the nominal angle and the delta angle relative to the nominal angle.

[0163] According to some embodiments, if the reconstruction filtering unit region is not predicted using directional intra mode or intra prediction mode, the signal class of the reconstruction filtering unit may be determined based on the neighboring blocks of the reconstruction filtering unit. A default signal class may be selected, or an explicit signaling of a signal class index or a filter coefficient may be performed based on whether the neighboring blocks are coded in directional intra mode or whether the directionality of the neighboring blocks is inconsistent. For example, if the neighboring blocks are not coded in directional intra mode or the directionality of the neighboring blocks is inconsistent, a default signal class may be selected, or an explicit signaling of a signal class index or a filter coefficient may be performed.

[0164] According to some embodiments, a directional majority vote or consistency check included in the reconstruction filtering unit may be performed to determine the filter class.

[0165] In one embodiment, if the size and shape of the reconstruction filtering unit are fixed and the number of directional information units (e.g., 8×8 for CDEF) included in the reconstruction filtering unit is greater than a predetermined number, a majority vote or consistency check of the directional information units included in the reconstruction filtering unit may be performed to determine the filter class. The directional information units may have various sizes and different directional intra-prediction modes.

[0166] In one embodiment, in case of majority vote, the most frequent directionality among the available and possibly merged directionality may be selected. In one example, a certain margin may be set between the first and second most frequent directionality.

[0167] In one embodiment, before taking the majority vote, it is determined whether the number of classes in the reconstruction filtering unit is greater than a predetermined number: if true, an inconsistency may be declared and either an explicit signal for the reconstruction filtering unit or the use of a smaller reconstruction filtering unit may be selected.

[0168] According to some embodiments, the reconstruction filtering unit in a chroma component may use its own directional information from a directional intra prediction mode if such information is available separately for the chroma component.

[0169] According to an aspect of the present disclosure, when directional information is available from both the intra prediction mode and the CDEF process, the directional mode of the intra prediction mode, together with the directional information from the CDEF process, may be used as a guide to identify and classify a reconstruction filtering unit (e.g., a Wiener or SGRPRJ filtering unit).

[0170] In one embodiment, when directionality information is available from both the intra-prediction direction and the CDEF process, a mapping of directionality from both sources may be introduced when checking the consistency of the directionality information. The classification-based restoration filtering process may be performed only if the directionality information is consistent. Figure 15 shows an example in which the intra-prediction directions corresponding to the seven delta angles associated with one of the eight basic angles in AV1 may be mapped to a single directionality class, and thus a one-to-one correspondence can be formed with the eight directions derived during the CDEF process.

[0171] In one embodiment, directionality information from one of the intra-prediction modes or the CDEF process may be used as a first source, and the other of the intra-prediction modes or the CDEF process may be used only if a consistent direction cannot be determined with the directionality information from the first source.

[0172] Note that in some examples, such as AV1, the Wiener filter and the SGRPRJ filter may be adaptively selected per LRU. The determination of the filter parameters and filtering process may be performed per LRU in the same way for both filters. The difference may be the type and number of filter parameters.

[0173] In one embodiment, when the SGRPRJ filter is implemented, a search window size r and a noise parameter e may be defined and used for each signal class, similar to the Wiener filter parameters.

[0174] In one embodiment, when an SGRPRJ filter is implemented, the projection parameters α and β may be defined and used for each signal class, similar to the Wiener filter parameters.

[0175] In one embodiment, when the SGRPRJ filter is implemented, the search window size r, the noise parameter e, and the projection parameters α and β may be defined and used for each signal class, similar to the Wiener filter parameters.

[0176] [V. Flowchart] FIG. 16 shows a flow chart outlining an example process (1600) according to one embodiment of the present disclosure. In various embodiments, the process (1600) is performed by a processing circuit such as a processing circuit in the terminal devices (210), (220), (230), and (240), a processing circuit performing the function of a video encoder (303), a processing circuit performing the function of a video decoder (310), a processing circuit performing the function of a video decoder (410), a processing circuit performing the function of an intra prediction module (452), a processing circuit performing the function of a video encoder (503), a processing circuit performing the function of a predictor (535), a processing circuit performing the function of an intra encoder (622), a processing circuit performing the function of an intra decoder (772), etc. In some embodiments, the process (1600) is implemented with software instructions, such that the processing circuit performs the process (1600) when the processing circuit executes the software instructions.

[0177] The process (1600) may generally begin at step (S1610), where the process (1610) determines directional information for a reconstruction filter unit included in a video frame based on at least one of a CDEF process or an intra-prediction mode. The process (1600) then proceeds to step (S1620).

[0178] In step (S1620), the process (1600) determines one of a plurality of sets of filter parameters of the reconstruction filter process based on the directional information of the reconstruction filter unit. Then, the process (1600) proceeds to step (S1630).

[0179] In step (S1630), the process (1600) performs a reconstruction filter process for the reconstruction filter unit based on one of the sets of filter parameters. Then, the process (1600) proceeds to step (S1640).

[0180] In step S1640, the process reconstructs the video frame based on the filtered reconstruction filter unit, and then the process ends.

[0181] In one embodiment, the reconstruction filter unit includes one or more directional information units, and at least one of the CDEF process or intra prediction modes is performed on one of the one or more directional information units.

[0182] In one embodiment, each of the multiple sets of filter parameters of the reconstruction filter is associated with at least one directionality of the CDEF process.

[0183] In one embodiment, the process (1600) determines one of a plurality of sets of filter parameters for the reconstruction filter process based on the directional information and the block variance information of the reconstruction filter unit.

[0184] In one embodiment, the process (1600) determines one of a plurality of sets of filter parameters of the reconstruction filter process based on the directional information of the reconstruction filter unit and the filter strength of the CDEF process.

[0185] In one embodiment, the process (1600) determines the directional information of the reconstruction filter units based on at least one of a majority vote or a consistency check of the directionalities in the reconstruction filter units.

[0186] In one embodiment, based on the fact that the reconstruction filter unit is not intra-coded and the neighboring blocks are intra-coded, the process (1600) determines the directional information of the reconstruction filter unit based on the intra-prediction mode performed on the neighboring blocks of the reconstruction filter unit.

[0187] In one embodiment, the process (1600) performs a reconstruction filter process for a reconstruction filter unit based on directional information determined from the CDEF process that matches directional information determined from the intra prediction mode.

[0188] In one embodiment, the reconstruction filter process is one of a Wiener filter process and a SGRPRJ filter process.

[0189] In one embodiment, the process determines one of multiple sets of filter parameters for the reconstruction filter process based on a default set of filter parameters, a communicated index indicating a set of filter parameters, or one of the communicated sets of filter parameters.

[0190] [VI. Computer Systems] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 17 illustrates a computer system (1700) suitable for implementing certain embodiments of the disclosed subject matter.

[0191] Computer software may be encoded using any suitable machine code or computer language, which may be subjected to assembly, compilation, linking, or similar mechanisms to generate code including instructions, which may be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., either directly or through an interpreter, microcode execution, etc.

[0192] The instructions may be executed on various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0193] 17 for computer system (1700) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1700).

[0194] The computer system (1700) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, through tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), and olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly associated with conscious human input, such as audio (e.g., speech, music, ambient sounds), images (scanned images, photographic images obtained from a still camera, etc.), and video (2D video, 3D video including stereoscopic pictures, etc.).

[0195] The input human interface devices may include one or more of a keyboard (1701), a mouse (1702), a trackpad (1703), a touch screen (1710), a data glove (not shown), a joystick (1705), a microphone (1706), a scanner (1707), and a camera (1708).

[0196] The computer system (1700) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1710), data gloves (not shown), or joystick (1705), although there may be haptic feedback devices that do not function as input devices), audio output devices (speakers (1709), headphones (not shown), etc.), visual output devices (screens (1710), including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have touch screen input capability, each of which may or may not have haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or three or more dimensional output through such means as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown). These visual output devices, such as a screen 1710, may be connected to the system bus 1748 through a graphics processor 1750.

[0197] The computer system (1700) may also include human accessible storage devices and associated media, such as optical media, including CD / DVD ROM / RW (1720) with CD / DVD or similar media (1721), thumb drives (1722), removable hard drives or solid state drives (1723), legacy magnetic media, such as tapes and floppy disks (not shown), specialized ROM / ASIC / PLD based devices, such as security dongles (not shown), etc.

[0198] Additionally, those skilled in the art should understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other non-transitory signals.

[0199] The computer system (1700) may also include a network interface (1754) to one or more communications networks (1755). The one or more communications networks (1755) may be, for example, wireless, wired, optical. The one or more communications networks (1755) may be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of the one or more communications networks (1755) include Ethernet, wireless LAN, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), TV wired or wireless wide area digital networks (including cable TV, satellite TV, and terrestrial broadcast TV), vehicular and industrial (including CANBus), etc. Certain networks typically require an external network interface adapter (such as a USB port on the computer system (1700)) attached to a particular general-purpose data port or peripheral bus (1749), while other network interface adapters are typically integrated into the core of the computer system (1700) by being attached to a system bus (such as an Ethernet interface to a PC computer system or a cellular network to a smartphone computer system) as described below. Using any of these networks, the computer system (1700) can communicate with other entities. Such communications may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to certain CANbus devices), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used in each of these networks and network interfaces.

[0200] The above-mentioned human interface devices, human accessible storage devices and network interfaces may be attached to a core (1740) of the computer system (1700).

[0201] The core (1740) may include one or more central processing units (CPUs) (1741), graphics processing units (GPUs) (1742), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (1743), hardware accelerators for specific tasks (1774), graphics adapters (1750), etc. These devices may be connected through a system bus (1748), along with read only memory (ROM) (1745), random access memory (1746), and internal mass storage device (1747) (such as an internal non-user accessible hard drive, SSD, etc.). In some computer systems, the system bus (1748) may be accessible in the form of one or more physical plugs to allow expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1748) or through a peripheral bus (1749). In one example, the screen 1710 may be attached to a graphics adapter 1750. Peripheral bus architectures include PCI, USB, etc.

[0202] The CPU (1741), GPU (1742), FPGA (1743), and accelerator (1744) may execute certain instructions, which in combination may constitute the above computer code. The computer code may be stored in a ROM (1745) or a RAM (1746). Also, temporary data may be stored in the RAM (1746), while persistent data may be stored in, for example, an internal mass storage device (1747). A cache memory, which may be closely associated with one or more of the CPU (1741), GPU (1742), mass storage device (1747), ROM (1745), RAM (1746), etc., may be used to enable fast storage and retrieval in any of the memory devices.

[0203] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0204] By way of example and not limitation, the architecture (1700), and in particular a computer system having a core (1740), may provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with a user-accessible mass storage device as described above, as well as specific storage of the core (1740) of a non-transitory nature, such as mass storage device (1747) internal to the core or ROM (1745). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (1740). The computer-readable media may include one or more memory devices or chips according to particular needs. The software may cause the core (1740), and in particular the processor (including a CPU, GPU, FPGA, etc.) therein, to perform certain operations or certain portions of certain operations described herein, including defining data structures stored in RAM (1746) and modifying such data structures according to operations defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of hardwired or otherwise embodied logic in circuitry (e.g., accelerator (1744)), which may operate in place of or in conjunction with software to perform particular operations or portions of particular operations described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, where appropriate. The present disclosure includes any appropriate combination of hardware and software.

[0205] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be recognized that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

[0206] [Appendix A: Abbreviations] ALF: Adaptive Loop Filter AMVP: Advanced Motion Vector Prediction APS: Adaptation Parameter Set ASIC: Application-Specific Integrated Circuit ATMVP: Alternative / Advanced Temporal Motion Vector Prediction AV1: AOMedia Video 1 AV2: AOMedia Video 2 BMS: Benchmark Set BV: Block Vector CANBus: Controller Area Network Bus CB: Coding Block CC-ALF: Cross-Component Adaptive Loop Filter CD: Compact Disc CDEF: Constrained Directional Enhancement Filter CPR: Current Picture Referencing CPU: Central Processing Unit CRT: Cathode Ray Tube CTB: Coding Tree Block CTU: Coding Tree Unit CU: Coding Unit DPB: Decoder Picture Buffer DPCM: Differential Pulse-Code Modulation DPS: Decoding Parameter Set DVD: Digital Video Disc FPGA: Field Programmable Gate Area JCCR: Joint CbCr Residual Coding JVET: Joint Video Exploration Team GOP: Groups of Pictures GPU: Graphics Processing Unit GSM: Global System for Mobile communications HDR: High Dynamic Range HEVC: High Efficiency Video Coding HRD: Hypothetical Reference Decoder IBC: Intra Block Copy IC: Integrated Circuit ISP: Intra Sub-Partitions JEM: Joint Exploration Model LAN: Local Area Network LCD: Liquid-Crystal Display LR: Loop Restoration Filter LRU: Loop Restoration Unit LTE: Long-Term Evolution MPM: Most Probable Mode MV: Motion Vector OLED: Organic Light-Emitting Diode PBs: Prediction Blocks PCI: Peripheral Component Interconnect PDPC: Position Dependent Prediction Combination PLD: Programmable Logic Device PPS: Picture Parameter Set PU: Prediction Unit RAM: Random Access Memory ROM: Read-Only Memory SAO: Sample Adaptive Offset SCC: Screen Content Coding SDR: Standard Dynamic Range SEI: Supplementary Enhancement Information SNR: Signal Noise Ratio SPS: Sequence Parameter Set SSD: Solid-state Drive TU: Transform Unit USB: Universal Serial Bus VPS: Video Parameter Set VUI: Video Usability Information VVC: Versatile Video Coding WAIP: Wide-Angle Intra Prediction

Claims

1. A method of video decoding in a decoder, comprising: determining directionality information of the reconstruction filter unit included in the video frame based on directionality of each of one or more groups of pixels in the reconstruction filter unit determined by at least one of a previously performed Constrained Directional Emphasis Filter (CDEF) process or a previously performed intra-prediction coding process; selecting a set of filter parameters from among a plurality of sets of filter parameters of a reconstruction filter process based on the directional information of the reconstruction filter unit; performing one of a Wiener filter process or an autoguided projection filter process as the reconstruction filter process for the reconstruction filter unit based on the selected set of filter parameters after the at least one of the previously performed CDEF process or the previously performed intra-prediction coding process; Reconstructing the video frame based on the filtered reconstruction filter unit; The method includes:

2. Each of the one or more groups of pixels is a directional information unit; The reconstruction filter unit includes one or more directional information units; The method of claim 1 , wherein the at least one of the previously performed CDEF process or the previously performed intra-predictive coding process is performed on one of the one or more directional information units.

3. A method as described in claim 1 or 2, wherein each of the multiple sets of filter parameters of the restoration filter process is related to at least one directionality of the previously executed CDEF process.

4. The step of selecting a set of filter parameters comprises: The method according to claim 1 , further comprising determining one of the sets of filter parameters of the reconstruction filter process based on the directional information and block variance information of the reconstruction filter unit.

5. The step of selecting a set of filter parameters comprises:

4. The method according to claim 1, further comprising determining one of the sets of filter parameters of the reconstruction filter process based on the directional information of the reconstruction filter unit and a filter strength of the previously performed CDEF process.

6. The step of determining the directional information of the reconstruction filter unit, comprising:

6. The method according to claim 1 , further comprising determining the directional information of the reconstruction filter unit based on at least one of a majority vote or a consistency check of the directionalities of the one or more groups of pixels in the reconstruction filter unit.

7. The method of claim 1, further comprising the steps of: determining whether the reconstruction filter unit is not intra-coded and whether an adjacent block of the reconstruction filter unit is intra-coded; The step of determining the directional information of the reconstruction filter unit comprises: The method according to claim 1 , comprising determining the directional information of the reconstruction filter unit based on a previously performed intra-predictive coding on the neighboring blocks of the reconstruction filter unit.

8. The step of executing 8. The method according to claim 1 , further comprising: performing the one of the Wiener filter process or the self-guided projection filter process on the reconstruction filter unit based on the directional information determined from the previously performed CDEF process being consistent with the directional information determined from the previously performed intra-prediction coding process.

9. A method according to any one of claims 1 to 8, wherein the performing step includes a step of determining boundary edges of the restoration filter process based on the selected set of filter parameters.

10. The step of selecting a set of filter parameters comprises:

4. The method of claim 1, further comprising determining one of the sets of filter parameters of the reconstruction filter process based on (i) a default set of filter parameters, (ii) a communicated index indicating a set of filter parameters, or (ii) one of the communicated sets of filter parameters.

11. An apparatus including a processing circuit, comprising: Apparatus, wherein the processing circuitry is configured to perform a method according to any one of claims 1 to 10.

12. A program which, when executed by at least one processor, causes the at least one processor to execute a method according to any one of claims 1 to 10.

13. A method of video encoding in an encoder, comprising: determining directionality information of the reconstruction filter unit included in the video frame based on directionality of each of one or more groups of pixels in the reconstruction filter unit determined by at least one of a previously performed Constrained Directional Emphasis Filter (CDEF) process or a previously performed intra-prediction coding process; selecting a set of filter parameters from among a plurality of sets of filter parameters of a reconstruction filter process based on the directional information of the reconstruction filter unit; Including, a reconstruction filter unit configured to perform, after the at least one of the previously performed CDEF process or the previously performed intra-prediction coding process, one of a Wiener filter process or an autoguided projection filter process based on the selected set of filter parameters as the reconstruction filter process for the reconstruction filter unit.

Images