Method, apparatus and program for video coding

Abstract

To provide a method and apparatus for video coding, and a program.SOLUTION: Aspects of the disclosure include methods, apparatuses, and non-transitory computer-readable storage mediums for video encoding / decoding. An apparatus includes processing circuitry that determines directionality information of a restoration filter unit included in a video frame based on at least one of a constrained directional enhancement filter (CDEF) process and an intra prediction mode. The processing circuitry determines one of a plurality of sets of filter parameters of a restoration filter process based on the directionality information of the restoration filter unit. The processing circuitry performs the restoration filter process on the restoration filter unit based on the one of the plurality of sets of filter parameters. The processing circuitry reconstructs the video frame based on the filtered restoration filter unit.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] [Incorporation of Related Applications] This application claims the benefit of priority to U.S. Patent Application No. 17 / 362,048, entitled "METHOD AND APPARATUS FOR VIDEO CODING," filed June 29, 2021, which claims the benefit of priority to U.S. Provisional Application No. 63 / 091,707, entitled "FEATURE INFORMATION REUSE FOR ENHANCED RESTORATION FILTERING," filed October 14, 2020. The entire disclosure of the prior application is incorporated by reference in its entirety.

[0002] [Technical field] This disclosure describes embodiments generally related to video coding. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context for the present disclosure. The work of the present inventors is not admitted explicitly or implicitly as prior art to the present disclosure, to the extent that such work is described in this background section, nor are any aspects of the description that do not specifically qualify as prior art at the time of filing.

[0004] Video encoding and decoding can be performed using motion-compensated inter-picture prediction. Uncompressed digital video can include a series of pictures, each with spatial dimensions of, for example, 1920 x 1080 luma samples and associated chroma samples. The series of pictures can have a fixed or variable picture rate (also informally known as frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video (1920 x 1080 luma sample resolution at a 60 Hz frame rate) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0005] One goal of video encoding and decoding is to reduce redundancy in an input video signal through compression. Compression can help reduce the bandwidth or storage space requirements, sometimes by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations of these, can be used. Lossless compression refers to a technique in which an exact copy of the original signal can be reconstructed from a compressed version of the original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signal is small enough to make the reconstructed signal useful for the intended application. For video, lossy compression is widely used. The amount of acceptable distortion depends on the application. For example, users of a particular consumer streaming application may tolerate higher distortion than users of a television distribution application. The achievable compression ratio can reflect higher acceptable distortion / acceptable distortion can result in a higher compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding.

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. If all blocks of samples are coded in intra mode, the picture can be an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and video session or as a still image. Samples of intra-blocks can be transformed, and the transform coefficients can be quantized before entropy coding. Intra-prediction can be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and AC coefficients after the transform, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0008] For example, conventional intra-coding, such as that known from MPEG-2 generation encoding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to predict intra-prediction from surrounding sample data and / or metadata obtained, for example, during encoding and / or decoding of spatially adjacent and previous blocks of data in decoding order. Such techniques are hereinafter referred to as "intra-prediction" techniques. It should be noted that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and does not use reference data from reference pictures.

[0009] There may be many forms of intra-prediction. If one or more of such techniques are available in a given video coding technique, the technique used may be coded as an intra-prediction mode. In some cases, a mode may have sub-modes and / or parameters, which may be coded separately or may be included in a mode codeword. The codeword used for a given mode, sub-mode, and / or parameter combination may affect the coding efficiency gain through intra-prediction, which in turn may affect the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain intra-prediction modes were introduced in H.264, improved in H.265, and further refined in newer coding techniques such as joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). A prediction block can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of the neighboring samples are copied into the prediction block according to a direction. A reference to the direction in use can be coded in the bitstream or may itself be predicted.

[0011] Referring to FIG. 1A, shown at the bottom right is a subset of nine known predictor directions from the 33 possible predictor directions in H.265 (corresponding to the 33 angular modes out of the 35 intra modes). The point where the arrows converge (101) represents the sample to be predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right at an angle of 45 degrees from horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left of sample (101) at an angle of 22.5 degrees from horizontal.

[0012] Still referring to FIG. 1A , a square block (104) of 4×4 samples is shown in the upper left (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with an “S” and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. Because the block is 4×4 samples in size, S44 is located in the lower right. Also shown are reference samples following a similar numbering scheme. The reference samples are labeled with an R and their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, the prediction samples are adjacent to the block being reconstructed, so negative values ​​do not need to be used.

[0013] Intra-picture prediction can work by copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating the prediction direction for this block, consistent with the arrow (102). That is, assume that the samples are predicted from one or more prediction samples to the upper right at an angle of 45 degrees from horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In some cases, especially when the orientation is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example by interpolation, to calculate the reference sample.

[0015] As video coding technology advances, the number of possible directions is increasing. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), and as of the time of disclosure, JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding have been used to represent these likely directions with a small number of bits, accepting a certain penalty for less likely directions. Furthermore, in some cases, the direction itself can be predicted from neighboring directions used in adjacent, already decoded blocks.

[0016] FIG. 1B shows a schematic diagram (105) illustrating the 65 intra-prediction directions according to JEM, showing the increasing number of prediction directions over time.

[0017] The mapping of intra-prediction direction bits in a coded video bitstream to represent directions can vary between video coding techniques, ranging, for example, from simple direct mapping of prediction directions to complex adaptive schemes including intra-prediction modes, codewords, most-probable modes, and similar techniques. However, in all cases, there may be certain directions that are statistically less likely to occur in the video content than certain other directions. Because the goal of video compression is to reduce redundancy, in well-performing video coding techniques, these less likely directions are represented by more bits than more likely directions.

[0018] Motion compensation is a lossy compression technique, and can be associated with a technique in which blocks of sample data from a previously reconstructed picture or part thereof (reference picture) are used to predict a newly reconstructed picture or part thereof after being spatially shifted in a direction indicated by a motion vector (hereinafter referred to as MV). In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or it may have three dimensions, with the third dimension indicating the reference picture in use (the latter may indirectly be the temporal dimension).

[0019] In some video compression techniques, the MV applicable to a particular region of sample data can be predicted from other MVs, e.g., from MVs associated with other regions of sample data that are spatially adjacent to the region being reconstructed and precede that MV in decoding order. This can significantly reduce the amount of data required to encode the MV, thereby removing redundancy and increasing compression. For example, when encoding an input video signal derived from a camera (known as natural video), MV prediction can work effectively because there is a statistical possibility that regions larger than the region to which a single MV is applicable move in a similar direction and therefore can potentially be predicted using a similar MV derived from the MV of a neighboring region. As a result, the detected MV for a given region will be similar or identical to the MV predicted from the surrounding MVs, and after entropy coding, the MV can be represented with fewer bits than would be used to directly encode the MV. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., a sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating a predictor from several surrounding MVs.

[0020] H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016) describes various MV prediction mechanisms. Among the many MV prediction mechanisms provided by H.265, a technique called "spatial merge" will be described in this specification.

[0021] Referring to Figure 1C, the current block (111) may contain samples that the encoder found during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of directly encoding its MV, the MV can be derived from metadata associated with one or more reference pictures, e.g., from the most recent reference picture (in decoding order) using the MV associated with any one of five surrounding samples denoted A0, A1, and B0, B1, B2 (112-116, respectively). In H.265, MV prediction can use predictors from the same reference picture as neighboring blocks. Summary of the Invention

[0022] An aspect of the present disclosure provides an apparatus for video encoding / decoding. The apparatus includes a processing circuit that determines directionality information of a reconstruction filter unit included in a video frame based on at least one of a constrained directional enhancement filter (CDEF) process or an intra-prediction mode. The processing circuit determines one of a plurality of sets of filter parameters for the reconstruction filter process based on the directionality information of the reconstruction filter unit. The processing circuit performs the reconstruction filter process on the reconstruction filter unit based on the one of the plurality of sets of filter parameters. The processing circuit reconstructs the video frame based on the filtered reconstruction filter unit.

[0023] In one embodiment, the reconstruction filter unit includes one or more directional information units, and at least one of the CDEF process or intra prediction mode is performed on one of the one or more directional information units.

[0024] In one embodiment, each of the sets of filter parameters of the reconstruction filter is associated with at least one directionality of the CDEF process.

[0025] In one embodiment, the processing circuit determines one of a plurality of sets of filter parameters for the reconstruction filter process based on the directionality information and the block variance information of the reconstruction filter unit.

[0026] In one embodiment, the processing circuit determines one of a plurality of sets of filter parameters for the reconstruction filter process based on the directional information of the reconstruction filter unit and the filter strength of the CDEF process.

[0027] In one embodiment, the processing circuit determines the directional information of the reconstruction filter unit based on at least one of a majority vote or a consistency check of the directionalities in the reconstruction filter unit.

[0028] In one embodiment, based on the fact that the reconstruction filter unit is not intra-coded and the neighboring blocks are intra-coded, the processing circuit determines the directionality information of the reconstruction filter unit based on the intra-prediction mode performed on the neighboring blocks of the reconstruction filter unit.

[0029] In one embodiment, the processing circuitry performs a reconstruction filter process for the reconstruction filter unit based on directional information determined from the CDEF process that matches directional information determined from the intra prediction mode.

[0030] In one embodiment, the restoration filter process is one of a Wiener filter process and a self-guided projection (SGRPRJ) filter process.

[0031] In one embodiment, the processing circuit determines one of multiple sets of filter parameters for the reconstruction filter process based on a default set of filter parameters, a communicated index indicating a set of filter parameters, or one of the communicated sets of filter parameters.

[0032] An aspect of the present disclosure provides a method for video encoding / decoding, in which directionality information of a reconstruction filter unit included in a video frame is determined based on at least one of a CDEF process or an intra-prediction mode. One of a plurality of sets of filter parameters of the reconstruction filter process is determined based on the directionality information of the reconstruction filter unit. The reconstruction filter process is performed on the reconstruction filter unit based on the one of the plurality of sets of filter parameters. The video frame is reconstructed based on the filtered reconstruction filter unit.

[0033] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform any one or combination of methods for video decoding. [Brief explanation of the drawings]

[0034] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Figure 1A] FIG. 2 is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B] FIG. 1 is a diagram of an exemplary intra-prediction direction. [Figure 1C]FIG. 1 is a schematic diagram of a current block and its surrounding spatial merge candidates in one example. [Figure 2] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Figure 3] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Figure 4] FIG. 2 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 5] FIG. 2 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 6] 10 shows a block diagram of an encoder according to another embodiment; [Figure 7] 10 shows a block diagram of a decoder according to another embodiment; [Figure 8] 1 illustrates an exemplary nominal angle according to one embodiment of the present disclosure. [Figure 9] 1 illustrates the locations of the top sample, left sample, and top-left sample for one pixel in the current block, according to one embodiment of the present disclosure. [Figure 10] 1 illustrates an exemplary recursive filter intra mode according to one embodiment of the present disclosure. [Figure 11] 1 illustrates some example directions in a constrained directional enhancement filter (CDEF) process according to some embodiments of the present disclosure. [Figure 12] 1 illustrates some example block divisions according to some embodiments of the present disclosure. [Figure 13] 10 illustrates an example in which a directional unit block is merged into a filtering unit according to one embodiment of the present disclosure. [Figure 14] 10 illustrates another example in which a directional unit block is merged into a filtering unit according to an embodiment of the present disclosure. [Figure 15] 10 illustrates an example mapping of directionality information between directional intra-prediction modes and main directions derived during the CDEF process, according to some embodiments of this disclosure. [Figure 16] 1 illustrates an exemplary flowchart according to one embodiment of the present disclosure. [Figure 17] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0035] [I. Video Decoder and Encoder Systems] FIG. 2 illustrates a simplified block diagram of a communication system (200) according to one embodiment of the present disclosure. The communication system (200) includes, for example, multiple terminal devices that can communicate with each other via a network (250). For example, the communication system (200) includes a first pair of terminal devices (210) and (220) interconnected via the network (250). In the example of FIG. 2, the first pair of terminal devices (210) and (220) perform unidirectional data transmission. For example, the terminal device (210) may encode video data (e.g., a stream of video pictures captured by the terminal device (210)) for transmission to another terminal device (220) via the network (250). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (220) may receive the coded video data from the network (250), decode the coded video data, reconstruct the video pictures, and display the video pictures according to the reconstructed video data. One-way data transmission may be common, such as in media presentation applications.

[0036] In another example, the communication system 200 includes a second pair of terminal devices 230 and 240 that perform bidirectional transmission of encoded video data, such as may occur during a video conference. For the bidirectional transmission of data, in one example, each of the terminal devices 230 and 240 may encode video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices 230 and 240 over the network 250. Each of the terminal devices 230 and 240 may also receive the encoded video data transmitted by the other of the terminal devices 230 and 240, decode the encoded video data to reconstruct the video pictures, and display the video pictures on an accessible display device in accordance with the reconstructed video data.

[0037] In the example of FIG. 2 , the terminal devices 210, 220, 230, and 240 may be depicted as a server, a personal computer, and a smartphone, although the principles of the present disclosure are not limited thereto. Embodiments of the present disclosure may also be applied to laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. The network 250 represents any number of networks that convey coded video data between the terminal devices 210, 220, 230, and 240, including, for example, wired (hardwired) and / or wireless communication networks. The communication network 250 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of the network 250 are not important to the operation of the present disclosure, unless otherwise described herein.

[0038] 3 illustrates the arrangement of a video encoder and a video decoder in a streaming environment as an example of an application of the disclosed subject matter, which is equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.), etc.

[0039] The streaming system may include a capture subsystem (313), which may include, for example, a video source (301) (e.g., a digital camera) that generates an uncompressed stream of video pictures (302). In one example, the stream of video pictures (302) includes samples captured by the digital camera. The stream of video pictures (302), depicted as a thick line to emphasize its higher data volume compared to the encoded video data (304) (or encoded video bitstream), may be processed by an electronic device (320) that includes a video encoder (303) coupled to the video source (301). The video encoder (303), as described in more detail below, may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter. The encoded video data (304) (or encoded video bitstream (304)), depicted as a thin line to emphasize its lower data volume compared to the stream of video pictures (302), may be stored on a streaming server (305) for future use. One or more streaming client subsystems, such as the client subsystems (306) and (308) in Figure 3, may access the streaming server (305) to obtain copies (307) and (309) of the encoded video data (304). The client subsystem (306) may include, for example, a video decoder (310) within an electronic device (330). The video decoder (310) decodes the input copy (307) of the encoded video data and generates an output stream (311) of video pictures that can be rendered on a display (312) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (304), (307), and (309) (e.g., a video bitstream) may be encoded according to a particular video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265.In one example, a developing video coding standard is informally known as Versatile Video Coding (VVC), and the disclosed subject matter may be used in the context of VVC.

[0040] It should be noted that the electronic devices 320 and 330 may include other components (not shown). For example, the electronic device 320 may include a video decoder (not shown), and the electronic device 330 may include a video encoder (not shown).

[0041] 4 shows a block diagram of a video decoder (410) according to one embodiment of the present disclosure. The video decoder (410) may be included in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., a receiving circuit). The video decoder (410) may be used in place of the video decoder (310) in the example of FIG. 3.

[0042] The receiver (431) may receive one or more coded video sequences to be decoded by the video decoder (410), and in the same or other embodiments, may receive one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel (401), which may be a hardware or software link to a storage device that stores the coded video data. The receiver (431) may receive the coded video data along with other data (e.g., coded audio data and / or auxiliary data streams), which may be forwarded to respective using entities (not shown). The receiver (431) may separate the coded video sequences from the other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420) (hereinafter referred to as "parser (420)"). In certain applications, the buffer memory (415) is part of the video decoder (410). In other cases, it may be external to the video decoder (410) (not shown). In still other cases, there may be a buffer memory (not shown) external to the video decoder (410), for example, to prevent network jitter, and in addition, there may be another buffer memory (415) internal to the video decoder (410), for example, to address playback timing. If the receiver (431) is receiving data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isochronous network, the buffer memory (415) may not be needed or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (415) may be needed and may be relatively large, advantageously adaptively sized, and at least partially implemented in an operating system or similar element (not shown) external to the video decoder (410).

[0043] The video decoder (410) may include a parser (420) for recovering symbols (421) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (410) and potentially include information for controlling a rendering device, such as a rendering device (412) (e.g., a display screen). The rendering device (412) may not be an integral part of the electronic device (430) but may be coupled to the electronic device (430), as shown in FIG. 4. The rendering device control information may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (420) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (420) may extract a set of subgroup parameters for at least one subgroup of pixels in a video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (420) may also extract information from the coded video sequence, such as transform coefficients, quantization parameter values, MVs, etc.

[0044] The parser (420) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (415) to generate symbols (421).

[0045] The recovery of the symbols (421) may involve several different units, depending on the type of coded video picture or portion thereof (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors. Which units are involved and how may be controlled by subgroup control information parsed from the coded video sequence by the parser (420). The flow of such subgroup control information between the parser (420) and the following units is not shown for clarity.

[0046] In addition to the functional blocks described above, the video decoder (410) may be conceptually subdivided into multiple functional units, as described below. In a practical implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0047] The first unit is a scalar / inverse transform unit (451), which receives quantized transform coefficients as symbols (421) from the parser (420), along with control information (including which transform to use, block size, quantization coefficients, quantization scaling matrix, etc.) The scalar / inverse transform unit (451) may output blocks containing sample values ​​that can be input to an aggregator (455).

[0048] In some cases, the output samples of the scaler / inverse transform (451) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) generates blocks of the same size and shape as the block being reconstructed using surrounding, already reconstructed information retrieved from the current picture buffer (458). The current picture buffer (458), for example, buffers the partially reconstructed and / or fully reconstructed current picture. In some cases, the aggregator (455) adds, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451).

[0049] In other cases, the output samples of the scaler / inverse transform unit (451) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion-compensated prediction unit (453) may access the reference picture memory (457) to retrieve samples used for prediction. After motion-compensating the retrieved samples according to the symbols (421) associated with the block, these samples may be added by the aggregator (455) to the output of the scaler / inverse transform unit (451) (in this case, referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (457) available to the motion-compensated prediction unit (453) from which the motion-compensated prediction unit (453) retrieves prediction samples may be controlled by MVs, for example, in the form of symbols (421) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory (457) when sub-sample accurate MVs are used, MV prediction mechanisms, etc.

[0050] The output samples of the aggregator (455) may be subjected to various loop filtering techniques in a loop filter unit (456). The video compression techniques may include in-loop filtering techniques controlled by parameters contained in the coded video sequence (also called a coded video bitstream) and made available to the loop filter unit (456) as symbols (421) from the parser (420), which may be responsive to meta-information obtained during decoding of previous portions of the coded picture or coded video sequence (in decoding order), as well as to previously reconstructed loop-filtered sample values.

[0051] The output of the loop filter unit (456) may be a sample stream that may be output to a rendering device (412) and stored in a reference picture memory (457) for use in future inter-picture prediction.

[0052] Once a particular coded picture is fully reconstructed, it may be used as a reference picture for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (420)), the current picture buffer (458) may become part of the reference picture memory (457), and a new current picture buffer may be reallocated before beginning reconstruction of the subsequent coded picture.

[0053] The video decoder (410) may perform decoding operations according to a predetermined video compression technique in a standard, such as ITU-T Rec. H.265. A coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, a profile may select certain tools from all tools available in the video compression technique or standard as the only tools available for use with that profile. Compliance also requires that the complexity of the coded video sequence be within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further constrained through a hypothetical reference decoder (HRD) specification and metadata about HRD buffer management conveyed in the coded video sequence.

[0054] In one embodiment, the receiver (431) may receive additional (redundant) data along with the coded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (410) to properly decode the data and / or to more accurately recover the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0055] 5 shows a block diagram of a video encoder (503) according to one embodiment of the present disclosure. The video encoder (503) is included in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmission circuit). The video encoder (503) may be used in place of the video encoder (303) in the example of FIG. 3.

[0056] The video encoder (503) may receive video samples from a video source (501) (which in the example of FIG. 5 is not part of the electronic device (520)), which may capture video images to be encoded by the video encoder (503). In other examples, the video source (501) is part of the electronic device (520).

[0057] The video source (501) may provide a source video sequence to be encoded by the video encoder (503) in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media presentation system, the video source (501) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (501) may be a camera that captures local image information as a video sequence. Video data may be provided as multiple individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, and each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following discussion will focus on samples.

[0058] According to one embodiment, the video encoder (503) may encode and compress pictures of a source video sequence into a coded video sequence (543) in real time or under any other time constraint required by the application. Achieving an appropriate coding rate is one function of the controller (550). In some embodiments, the controller (550) controls and is operatively coupled to other functional units, as described below. Coupling is not shown for clarity. Parameters set by the controller (550) may include rate control-related parameters (e.g., picture skip, quantization, lambda value for rate-distortion optimization techniques), picture size, group-of-picture (GOP) layout, maximum MV-allowed reference region, etc. The controller (550) may also be configured with other appropriate functionality associated with the video encoder (503) optimized for a particular system design.

[0059] In some embodiments, the video encoder (503) is configured to operate in an encoding loop. As a very simplified description, in one example, the encoding loop may include a source coder (530) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be encoded and reference pictures) and a (local) decoder (533) embedded in the video encoder (503). The decoder (533) reconstructs the symbols to generate sample data similar to what a (remote) decoder would generate (so that any compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (534). Because decoding of the symbol stream yields bit-for-bit accurate results independent of the location (local or remote) of the decoder, the contents in the reference picture memory (534) are also bit-for-bit accurate between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder "sees" when using the prediction during decoding. This basic principle of reference picture synchronization (including the resulting drift when synchronization cannot be maintained, for example, due to channel error) is used in some related technologies as well.

[0060] The operation of the "local" decoder (533) may be the same as a "remote" decoder, such as the video decoder (410), which has already been described in detail above in connection with Figure 4. However, with brief reference to Figure 4, because symbols are available and the encoding / decoding of symbols into an encoded video sequence by the entropy coder (545) and parser (520) may be lossless, the entropy decoding portion of the video decoder (410), including the buffer memory (515) and parser (520), may not be fully implemented in the local decoder (533).

[0061] An observation that can be made at this point is that any decoder technology, other than analysis / entropy decoding, that exists in the decoder must necessarily exist in substantially the same functional form in the corresponding encoder. For this reason, the subject matter of this disclosure focuses on decoder operation. A description of the encoder technology can be omitted, as it is the reverse of the decoder technology, which is described generically. Only in certain areas is more detailed explanation necessary, and is provided below.

[0062] In some examples, during operation, the source coder (530) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (532) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0063] The local video decoder (533) may decode the coded video data of pictures that may be designated as reference pictures based on symbols generated by the source coder (530). The operation of the coding engine (532) may advantageously be a lossy process. When the coded video data can be decoded by a video decoder (not shown in FIG. 5), the reconstructed video sequence may typically be a replica of the source video sequence, with some errors. The local video decoder (533) may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in a reference picture cache (534). In this way, the video encoder (503) may locally store copies of reconstructed reference pictures that have common content with reconstructed reference pictures (without transmission errors) obtained by a far-end video decoder.

[0064] The predictor (535) may perform a predictive search for the coding engine (532). That is, for a new picture to be coded, the predictor (535) may search the reference picture memory (534) for sample data (as candidate reference pixel blocks) or specific metadata (reference picture MV, block shape, etc.), which may serve as suitable prediction references for the new picture. The predictor (535) may operate sample block-by-pixel block to find suitable prediction references. In some cases, the input picture determined by the search results obtained by the predictor (535) may have prediction references drawn from multiple reference pictures stored in the reference picture memory (534).

[0065] The controller (550) may manage the encoding operations of the source coder (530), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0066] The output of all the above functional units may undergo entropy coding in an entropy coder (545), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0067] The transmitter (540) may buffer the coded video sequence produced by the entropy coder (545) and prepare it for transmission over a communication channel (560), which may be a hardware or software link to a storage device that stores the coded video data. The transmitter (540) may merge the coded video data from the video coder (503) with other data to be transmitted, such as coded audio data and / or an auxiliary data stream (not shown).

[0068] The controller (550) may manage the operation of the video encoder (503). During encoding, the controller (550) may assign each coded picture a particular coding picture type, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0069] An intra picture (I-picture) may be one that can be coded and decoded without using other pictures in the sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will recognize these variations of I-pictures and their respective uses and characteristics.

[0070] A predicted picture (P picture) may be coded and decoded using intra- or inter-prediction, using at most one MV and reference index to predict the sample values ​​of each block.

[0071] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, using up to two MVs and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0072] In general, a source picture may be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the respective picture of the blocks. For example, blocks of an I-picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P-picture may be predictively coded via spatial or temporal prediction with reference to a reference picture coded one step earlier. Blocks of a B-picture may be predictively coded via spatial or temporal prediction with reference to a reference picture coded one or two steps earlier.

[0073] The video encoder (503) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T Rec. H.265. In doing so, the video encoder (503) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard being used.

[0074] In one embodiment, the transmitter (540) may transmit additional data along with the coded video. The source coder (530) may include such data as part of the coded video sequence. The additional data may include other types of redundant data, such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0075] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. If a block in the current picture is similar to a reference block in a previously encoded and still buffered reference picture in the video, the block in the current picture may be coded by a vector called a MV. The MV refers to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0076] In some embodiments, bidirectional prediction techniques may be used in inter-picture prediction. According to bidirectional prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both preceding the current picture in decoding order (but may also be past and future, respectively, in display order) in the video. A block in the current picture may be coded by a first MV that points to a first reference block in the first reference picture and a second MV that points to a second reference block in the second reference picture. A block may be predicted by a combination of the first reference block and the second reference block.

[0077] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0078] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Typically, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree-decomposed into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one CU of 64x64 pixels, four CUs of 32x32 pixels, or 16 CUs of 16x16 pixels. In one example, each CU is analyzed to determine the CU's prediction type, such as an inter-prediction type or an intra-prediction type. A CU is divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0079] 6 shows a diagram of a video encoder (603) according to another embodiment of this disclosure. The video encoder (603) is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video picture in a sequence of video pictures and to encode the processed block into a coded picture that is part of the coded video sequence. In one example, the video encoder (603) is used in place of the example video encoder (303) of FIG. 3.

[0080] In an HEVC example, the video encoder (603) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8x8 samples. The video encoder (603) determines, for example, using rate-distortion optimization, whether the processing block is best coded using intra mode, inter mode, or bidirectional prediction mode. If the processing block is coded in intra mode, the video encoder (603) may use intra prediction techniques to code the processing block into a coded picture. If the processing block is coded in inter mode or bidirectional prediction mode, the video encoder (603) may use inter prediction or bidirectional prediction techniques, respectively, to code the processing block into a coded picture. In certain video coding techniques, merge mode may be an inter-picture prediction sub-mode in which MVs are derived from one or more MV predictors without the benefit of coded MV components other than the MV predictors. In certain other video coding techniques, there may be motion vector components applicable to the block of interest. In one example, the video encoder (603) includes other components, such as a mode decision module (not shown) for determining the mode of the processing block.

[0081] In the example of Figure 6, the video encoder (603) includes an inter-encoder (630), an intra-encoder (622), a residual calculator (623), a switch (626), a residual encoder (624), an overall controller (621), and an entropy encoder (625) coupled together as shown in Figure 6.

[0082] The inter-encoder (630) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-prediction information (e.g., a description of redundant information due to inter-coding techniques, MV, merge mode information), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the coded video information.

[0083] The intra encoder (622) is configured to receive samples of a current block (e.g., a processing block), optionally compare the block with previously coded blocks in the same picture, generate quantized coefficients after transformation, and optionally generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). In one example, the intra encoder (622) also calculates intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same picture.

[0084] The overall controller (621) is configured to determine overall control data and control other components of the video encoder (603) based on the overall control data. In one example, the overall controller (621) determines the mode of the block and provides a control signal to the switch (626) based on the mode. For example, if the mode is intra mode, the overall controller (621) controls the switch (626) to select the intra mode result to be used by the residual calculator (623) and controls the entropy encoder (625) to select intra prediction information and include the intra prediction information in the bitstream. If the mode is inter mode, the overall controller (621) controls the switch (626) to select the inter prediction result to be used by the residual calculator (623) and controls the entropy encoder (625) to select inter prediction information and include the inter prediction information in the bitstream.

[0085] The residual calculator (623) is configured to calculate the difference (residual data) between the received block and a prediction result selected from the intra-encoder (622) or inter-encoder (630). The residual encoder (624) operates based on the residual data and is configured to encode the residual data to generate transform coefficients. In one example, the residual encoder (624) is configured to transform the residual data from the spatial domain to the frequency domain to generate transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is configured to perform an inverse transform to generate decoded residual data. The decoded residual data may be used by the intra-encoder (622) and inter-encoder (630), as appropriate. For example, the inter-encoder (630) may generate decoded blocks based on the decoded residual data and inter-prediction information, and the intra-encoder (622) may generate decoded blocks based on the decoded residual data and intra-prediction information. The decoded blocks are processed appropriately to generate decoded pictures, which may be buffered in a memory circuit (not shown) and, in some examples, used as reference pictures.

[0086] The entropy encoder (625) is configured to format the bitstream to include the coded block. The entropy encoder (625) is configured to include various information according to an appropriate standard, such as HEVC. In one example, the entropy encoder (625) is configured to include global control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. It should be noted that, according to the disclosed subject matter, when encoding a block in a merged sub-mode of either the inter mode or the bi-prediction mode, the residual information is not present.

[0087] 7 shows a diagram of a video decoder (710) according to another embodiment of this disclosure. The video decoder (710) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (710) is used in place of the video decoder (310) of the example of FIG. 3.

[0088] In the example of Figure 7, the video decoder (710) includes an entropy decoder (771), an inter decoder (780), a residual decoder (773), a reconstruction module (774), and an intra decoder (772) coupled together as shown in Figure 7.

[0089] The entropy decoder (771) may be configured to recover, from a coded picture, specific symbols that represent the syntax elements of which the coded picture is composed. Such symbols may include, for example, prediction information (e.g., intra-mode, inter-mode, bi-predictive mode, merged submode, or the latter two in other submodes) that can identify the mode in which the block is coded (e.g., intra-mode, inter-mode, bi-predictive mode, merged submode, or the latter two in other submodes), specific samples or metadata used for prediction by the intra-decoder (772) or inter-decoder (780), respectively (e.g., intra-predictive information or inter-predictive information), residual information in the form of quantized transform coefficients, etc. In one example, if the prediction mode is an inter-mode or bi-predictive mode, the inter-predictive information is provided to the inter-decoder (780), and if the prediction type is an intra-predictive type, the intra-predictive information is provided to the intra-decoder (772). The residual information may undergo inverse quantization and be provided to the residual decoder (773).

[0090] The inter decoder (780) is configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0091] The intra decoder (772) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0092] The residual decoder (773) is configured to perform inverse quantization to extract inverse quantized transform coefficients and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require certain control information (including quantizer parameters (QP)), which may be provided by the entropy decoder (771) (data path not shown as this may only be low-volume control information).

[0093] The reconstruction module (774) is configured to combine, in the spatial domain, the residual output by the residual decoder (773) and the prediction result (possibly output by an inter-prediction module or an intra-prediction module) to form a reconstruction block, which may be part of a reconstructed picture or part of a reconstructed video. It should be noted that other suitable operations, such as a debooking operation, may also be performed to improve visual quality.

[0094] It should be noted that the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using any suitable technology. In one embodiment, the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (303), (503), and (603) and the video decoders (310), (410), and (710) may be implemented using one or more processors executing software instructions.

[0095] [II. Intra Prediction] In some related examples, such as VP9, ​​eight directional modes are supported, corresponding to angles from 45 degrees to 207 degrees. In some related examples, such as AV1 (AOMedia Video 1), the directional intra mode is extended to a finer set of angles to take advantage of more diverse spatial redundancies in directional textures. The original eight angles are slightly modified and called nominal angles, and these eight nominal angles are named V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED.

[0096] FIG. 8 illustrates exemplary nominal angles according to one embodiment of the present disclosure. Each nominal angle may be associated with seven smaller angles, so that in some related examples such as AV1, there may be a total of 56 directional angles. The prediction angle is expressed as the nominal intra angle plus an angle delta, which is derived by multiplying a coefficient (ranging from -3 to 3) with a step size of 3 degrees. To implement directional prediction modes in AV1 in a general way, all 56 directional intra prediction angles in AV1 can be implemented with a unified directional predictor that projects each pixel to a reference subpixel position and interpolates the reference subpixel with a 2-tap bilinear filter.

[0097] In some related examples such as AV1, there are five non-directional smoothed intra prediction modes, which are DC, PAETH, SMOOOTH, SMOOTH_V, and SMOOTH_H. For DC prediction, the average of the left neighboring sample and the above neighboring sample is used as the predictor of the block to be predicted. For PAETH prediction, the above reference sample, the left reference sample, and the above-left reference sample are first taken, and then the closest value to (above + left - above-left) is set as the predicted value of the pixel to be predicted.

[0098] 9 shows the locations of the top sample, left sample, and top-left sample for one pixel in the current block according to one embodiment of the present disclosure. For SMOOTH, SMOOTH_V, and SMOOTH_H modes, the block is predicted using quadratic interpolation in the vertical or horizontal direction, or an average in both directions.

[0099] FIG. 10 illustrates an exemplary recursive filter intra mode according to one embodiment of the present disclosure.

[0100] To capture the decaying spatial correlation with the reference on the edge, a FILTER INTRA mode is designed for luma blocks. In AV1, five filter intra modes are defined, each represented by a set of eight 7-tap filters that reflect the correlation between pixels in a 4x2 patch and the seven neighbors adjacent to the patch. For example, the weighting coefficients of the 7-tap filters depend on the position. As shown in Figure 10, an 8x8 block is divided into eight 4x2 patches, denoted by B0, B1, B2, B3, B4, B5, B6, and B7. For each patch, seven neighbors, denoted by R0 through R6, are used to predict pixels within the respective patch. For patch B0, all neighbors have already been reconstructed. However, for other patches, if all neighbors have not been reconstructed, the predicted values ​​of the nearest neighbors are used as reference values. For example, because all neighbors of patch B7 have not been reconstructed, the predicted samples of patch B7's neighbors (i.e., B5 and B6) are used instead.

[0101] For the chroma components, a chroma-only intra prediction mode, called CfL (chroma from luma) mode, models chroma pixels as linear functions of simultaneous reconstructed luma pixels. CfL prediction may be expressed as follows: CfL(α)=α×L AC +DC formula (1) where L ACwhere α denotes the AC contribution of the luma component, α denotes a parameter of the linear model, and DC denotes the DC contribution of the chroma component. In one example, the restored luma pixels are subsampled to the chroma resolution and then averaged to form the AC contribution. Instead of requiring the decoder to calculate scaling parameters to approximate the chroma AC components from the AC contributions, as in some related examples, the CfL mode in AC1 determines the parameter α based on the original chroma pixels and transmits them in the bitstream. This reduces decoder complexity and produces more accurate predictions. As for the DC contribution of the chroma components, it is calculated using intra-DC mode, which is sufficient for most chroma content and has mature, fast implementations.

[0102] [III. Loop Filtering] In some related examples, such as AV1, three in-loop filters may be applied to the reconstructed frames in the following order: deblocking, a constrained directional enhancement filter (CDEF), and a loop-restoration filter, which include a Wiener filter and a self-guided projection (SGRPRJ) filter, one of which may be adaptively selected.

[0103] Deblocking may be applied across transform block boundaries to reduce blocky artifacts caused by quantization of transform coefficients. In some examples, 4-, 8-, and 14-tap finite impulse response (FIR) filters may be used for luma blocks, and 4- and 6-tap FIR filters may be used for chroma blocks.

[0104] The filter length is initially determined by the minimum transform block size at the boundary. A conditional check may be performed using variance to avoid blurring the actual edge. Furthermore, a flatness check may also be incorporated to finally determine the filter length.

[0105] The CDEF is a nonlinear deringing filter applied along the directional features detected in the target region. In some related examples, an 8x8 region is the unit size on which the CDEF is performed. Standard direction detection may be performed as shown in Figure 11, which shows some example directions in the CDEF process.

[0106] In FIG. 11, a candidate direction d (0 to 7) that minimizes the following quantity may be selected as the main direction.

[0107]

number

[0108]

number

[0109]

number

[0110]

number

[0111]

number

[0112] When applying filtering, each 64x64 block in the tile may select one of the presets, and filtering may be performed on each 8x8 unit within each 64x64 block.

[0113] Note that during the decoding process, several variables related to the signal characteristics of local regions may be analyzed or derived from the bitstream. These variables include dir, var, priStr, secStr, and damping. The variable dir represents the main edge direction of the 8x8 block. The variable var represents the variance of signal values ​​within the 8x8 block and is defined as the normalized difference between the cost in the main direction and the cost along the direction orthogonal to the main direction. The variable priStr represents the primary filter strength S of the 64x64 block containing the 8x8 filtering unit. p The variable secStr represents the secondary filter strength S of a 64x64 block containing an 8x8 filtering unit. s The variable damping represents the damping parameter D of a 64x64 block containing 8x8 filtering units. These values ​​may be obtained separately for the luma and chroma channels.

[0114] After the deblocking and CDEF processes are performed, in some related examples such as AV1, mutually exclusive application of two types of restoration filters may be performed. The two types of restoration filters include a Wiener filter and an SGRPRJ filter. The square loop-restoration unit (LRU) size may be selected from 64x64 to 256x256.

[0115] In a Wiener filter, the quality of each reconstructed pixel in a coded frame can be improved through non-causal filtering using neighboring pixels in a W × W window around each pixel. The 2D filter taps of the Wiener filter, denoted F, are determined as follows: F=H -1 M formula (7) Here, H=E[XX T ] is the autocovariance of x, and W is a column vector in a W × W window. 2 contains samples, M=E[YX T ] is the cross-correlation between x and the original source sample y.

[0116] In some related examples, such as AV1, the separability of F and the symmetry and normalization of the filter coefficients of the Wiener filter may be imposed as constraints. 2 , formed as a 1× vector, may be defined as follows: F=column_vectorize[ab T ] Formula (8) where a and b are W×1 vertical and horizontal filters such that a(i)=a(W−1−i), b(i)=b(W−1−i) and Σa(i)=Σb(i)=1, for i=0, 1,..., r−1. The coefficient vectors a and b may be searched in the encoder and coded into the bitstream.

[0117] SGRPRJ filtering converts a degraded image x into a simple restored version as follows:

[0118]

number

[0119]

number

[0120] More specifically, the local mean (μ) and variance (σ) of a pixel within a (2r+1) × (2r+1) window around the pixel 2 ) may be calculated and each pixel x may be filtered as follows:

[0121]

number

[0122] Given two simple restorations X1 and X2 from a degraded image X by equation (9), the final output X r The following subspace projections are performed to construct X r =X+α(X1-X)+β(X2-X) Equation (11) Given X, X1, X2 and source Y, the encoder may calculate α and β as follows: [α β] T =(A T A) -1 A T b Equation (12) where A={X1-X, X2-X} and b=YX.

[0123] The encoder may then send a 6-tuple (r1, e1, r2, e2, α, β) for each LRU.

[0124] [IV. Reusing feature information for enhancement restoration filtering] In some related examples, such as AV1, the Wiener filter may be implemented in square-sized units ranging from 64x64 to 256x256 by uniformly dividing the frame / tile into LRUs. In one example, the filter coefficients of the Wiener filter may be obtained by assuming that the signal statistics are stationary. Therefore, it is desirable to classify the target region for filtering into one of several statistical types for which the stationarity assumption can reasonably be made. Possible approaches to classifying the target region include using quantities such as local variance or edge information. These quantities themselves or related categorical information may be calculated at the decoder or conveyed in the bitstream, but this may be costly in terms of computational or bitrate overhead.

[0125] In some related examples, such as AV1, the SGRPRJ filter may be implemented on square-sized units ranging from 64 × 64 to 256 × 256 by uniformly dividing the frame / tile into LRUs. In the SGRPRJ filter, a form of simple edge-preserving filtering is performed to construct a simple reconstruction image using a fixed radius and noise parameter pair for each LRU. Furthermore, fixed projection parameters α and β may be used as weighting coefficients for the error image to form the final reconstruction for each LRU. However, regions within the error image may have different statistical properties reflecting local signal features such as edges and texture. Therefore, if a single set of SGRPRJ filter parameters, such as radius, noise parameters, α and β, is used or estimated across LRU regions covering pixels with widely varying signal statistics, the quality of the final reconstruction may be compromised. On the other hand, incorporating signal classification for better adaptability may impose essentially the same challenges as Wiener filtering in terms of additional computational load or bitrate overhead.

[0126] This disclosure includes methods for improving the performance of reconstruction filtering techniques by reusing feature information, such as directional information derived from the CDEF process and / or intra-prediction modes. For example, adaptive reconstruction filtering techniques may effectively reuse signal features and statistical information already available at the decoder.

[0127] In this disclosure, a restoration filter (or filtering) process may be defined as a filtering process used to operate on a noisy image and estimate a clean, original image based on the noisy image. A restoration filter process may include a process used to blur an image or an inverse process used to reverse blur an image. Examples of restoration filter processes include, but are not limited to, a Wiener filtering process and an SGRPRJ filtering process. A restoration filter (or filtering) unit is the domain in which the restoration filter process is performed.

[0128] In this disclosure, a directional information unit may be defined as a group of pixels having a specified shape and size, and may provide the main direction of a feature represented by the pixel values ​​of the group of pixels. In an example such as AV1, each directional information unit in the CDEF process may be an 8x8 block. The main direction and variance values ​​of the pixels within each 8x8 block may be derived in a standard manner. In other examples, directional intra-prediction modes in AV1 may provide such information in units of various shapes and sizes corresponding to intra-prediction blocks.

[0129] According to aspects of the present disclosure, the directional information derived at the decoder may be reused to estimate the presence and directionality of boundary edges for a reconstruction filter, such as a Wiener filter or an SGRPRJ filter. For example, the directional information may be derived from a CDEF process.

[0130] According to some embodiments, the shape and size of the reconstruction filtering unit (e.g., a Wiener or SGRPRJ filtering unit) may be defined using multiple available directionality information units (e.g., the 8x8 blocks used in CDEF direction detection and filtering in AV1). In this way, finer-granularity direction adaptability can be achieved. Therefore, the reconstruction filtering process may be performed with a unit size smaller than one of the fixed square types of 64x64, 128x128, or 256x256 used in some related examples such as AV1.

[0131] In one embodiment, the size of the reconstruction filtering unit may be the same as the LRU size defined in AV1, for example.

[0132] In one embodiment, the size of the restoration filtering unit may be the same as the size of the directionality information unit (eg, 8x8).

[0133] In one embodiment, the restoration filtering unit may be further divided into square, rectangular, T-shaped or four-way sub-LRUs from a given LRU size, similar to or consistent with the division in some related embodiments, such as the division in AV1 as shown in FIG. 12.

[0134] In one embodiment, blocks each having a size of a directional information unit (e.g., 8x8) may be merged by following various scan orders to form a filtering unit. Figure 13 shows an example in which blocks each having a size of an 8x8 directional unit are merged in a raster scan order into filtering units each having a size of 32x8. For example, four 8x8 directional unit blocks (1301)-(1304) may be merged into a 32x8 filtering unit block (1310), and four 8x8 directional unit blocks (1305)-(1308) may be merged into a 32x8 filtering unit block (1320).

[0135] In one embodiment, blocks having the same directionality and each having the size of a directional information unit (e.g., 8x8) may be merged in various scan orders to form a filtering unit. Figure 14 shows an example in which blocks each having the size of an 8x8 directional unit are merged in a raster scan order into a filtering unit of variable size. The filtering unit sizes include 8x8, 16x8, and 32x8. For example, one 8x8 directional unit block (1401) may be an 8x8 filtering unit block (1410), two 8x8 directional unit blocks (1403)-(1404) may be merged into a 16x8 filtering unit block (1420), and four 8x8 directional unit blocks (1405)-(1408) may be merged into a 32x8 filtering unit block (1430).

[0136] According to some embodiments, each available directionality of the CDEF process may be used directly as a class index for a signal class for which a unique set of reconstruction filter shapes and sizes can be defined, i.e., the selection of the reconstruction filter may depend on the available directionality of the CDEF process.

[0137] In one embodiment, a solution to an equation in the calculation of the reconstruction filter, such as equation (7), may be applied for each signal class.

[0138] In one embodiment, different shaped 2D filters with different numbers of filter taps may be used for each signal class, with or without symmetry.

[0139] In one embodiment, the separability of the 2D filter (separable or non-separable filter) may depend on the available directionality of the CDEF process.

[0140] In one embodiment, multiple directions outside the available directions of the CDEF process may be merged into a single class, resulting in a reduction in the number of directional classes, and for each merged class, a unique set of reconstruction filter shapes and sizes may be defined.

[0141] According to some embodiments, in addition to directionality, block variance information may be combined to further refine the directionality-based classes.

[0142] In one embodiment, the directional information units having a size of 8x8 in the CDEF process may be further classified into different subclasses. The classification may be based on the variance value of the directional information units. For example, if the number of classes based on directionality is 5 and the number of classes based on variance is 3, there may be 15 signal classes, each of which can be designed with a set of restoration filters (e.g., Wiener or SGRPRJ).

[0143] According to some embodiments, in addition to the directionality, the filter strength may be combined to determine the signal class of the reconstruction filtering unit. For example, the transmitted primary CDEF filter strength and secondary CDEF filter strength may be combined to determine the signal class of the reconstruction filtering unit. Different filter strength presets selected by the encoder may indicate different signal characteristics of the target region.

[0144] In one embodiment, one of the first-order filter strength and second-order filter strength presets conveyed in the bitstream may be directly used as the other dimension of the signal class index. For example, if the number of classes based on directionality is 5 and the number of classes based on filter strength presets is 4, there may be 20 signal classes for which a set of restoration filters (e.g., Wiener or SGRPRJ) can be designed, respectively.

[0145] According to some embodiments, a directional majority vote or consistency check included in the reconstruction filtering unit may be performed to determine the filter class.

[0146] In one embodiment, if the size and shape of the reconstruction filtering unit are fixed and the number of directional information units (e.g., 8x8 for CDEF) included in the reconstruction filtering unit is greater than a predetermined number, a majority vote or consistency check of the directionalities included in the reconstruction filtering unit may be performed to determine the filter class.

[0147] In one embodiment, in the case of a majority vote, the most frequent directionality among the available, possibly merged, directions may be selected, and in one example, a certain margin may be set between the first and second most frequent directionality.

[0148] In one embodiment, before taking the majority vote, it is determined whether the number of classes in the reconstruction filtering unit is greater than a predetermined number, and if true, an inconsistency may be declared and either explicit signaling for the reconstruction filtering unit or the use of a smaller reconstruction filtering unit may be selected.

[0149] According to some embodiments, the same scheme and directionality information used for the reconstruction filtering unit in the luma component may be used for the reconstruction filtering unit in the chroma component if such information is only available for the luma component from the CDEF process.

[0150] According to some embodiments, the restoration filtering unit in a chroma component may use its own filter strengths, such as preset values ​​of primary and secondary filter strengths included in the chroma component, if such information is available for the chroma component.

[0151] According to some embodiments, the restoration filtering unit in a chroma component may use its own variance information if such information is available for the chroma component.

[0152] In one embodiment, when the CDEF process is turned off and the reconstruction filter is turned on, the direction search process of the CDEF process described in Section III (Loop Filtering Section) may be applied to derive the existence and directionality of the boundary edges of the reconstruction filtering unit.

[0153] In one embodiment, when the CDEF process is turned off and the reconstruction filter is turned on, a default signal class is selected, or explicit signaling of the filter class index may be performed.

[0154] According to aspects of this disclosure, the directional information indicated by the intra-prediction modes available at the decoder may be reused as a guide for labeling signal classes in a reconstruction filtering unit (e.g., a Wiener or SGRPRJ filtering unit), each of which may define a unique set of reconstruction filter shapes and sizes. In some embodiments, such directional information may be provided by the encoder for various unit sizes for intra-prediction.

[0155] According to some embodiments, the shape and size of the reconstruction filtering unit may be defined using multiple available directionality information units (e.g., 8x8 for the directional intra-prediction unit in AV1). In this way, finer-granularity direction adaptability can be achieved. Therefore, the reconstruction filtering process may be performed with a unit size smaller than one of the fixed square types of 64x64, 128x128, or 256x256 used in some related examples such as AV1.

[0156] In one embodiment, the size of the reconstruction filtering unit may be the same as the LRU size defined in AV1, for example.

[0157] In one embodiment, the size of the restoration filtering unit may be the same as the size of the directionality information unit (eg, 8x8).

[0158] In one embodiment, the restoration filtering unit may be further divided into square, rectangular, T-shaped or four-way sub-LRUs from a given LRU size, similar to or consistent with the division in some related embodiments, such as the division in AV1 as shown in FIG. 12.

[0159] In one embodiment, blocks each having a size of a directional information unit (e.g., 8x8) may be merged by following various scanning orders to form a filtering unit, as shown in FIG. 13.

[0160] In one embodiment, blocks each having the size of a directionality information unit (e.g., 8x8) and having similar directionality may be merged by the following various scanning orders to form a filtering unit, as shown in FIG. 14: According to some embodiments, a fixed number of signal classes may be defined, and blocks of each signal class may have intra-prediction modes with similar directionality. A unique set of reconstruction filter shapes and sizes may be defined for each signal class. When non-angular intra-modes such as SMOOTH (including SMOOTH, SMOOTH_H, and SMOOOTH_V modes), Paeth predictor, or DC mode are used, each or a combination of the non-angular intra-modes may be associated with its own signal class.

[0161] In one embodiment, each of the eight nominal angles, along with the associated seven possible delta angles in AV1, may be grouped together to form a total of eight directional classes.

[0162] In one embodiment, the directional class may depend on both the nominal angle and the delta angle relative to the nominal angle.

[0163] According to some embodiments, if a reconstruction filtering unit region is not predicted using a directional intra-mode or an intra-prediction mode, the signal class of the reconstruction filtering unit may be determined based on neighboring blocks of the reconstruction filtering unit. A default signal class may be selected, or explicit signaling of signal class indexes or filter coefficients may be performed based on whether the neighboring blocks are coded in directional intra-mode or whether the directionality of the neighboring blocks is inconsistent. For example, if the neighboring blocks are not coded in directional intra-mode or the directionality of the neighboring blocks is inconsistent, a default signal class may be selected, or explicit signaling of signal class indexes or filter coefficients may be performed.

[0164] According to some embodiments, a directional majority vote or consistency check included in the reconstruction filtering unit may be performed to determine the filter class.

[0165] In one embodiment, if the size and shape of the reconstruction filtering unit are fixed and the number of directional information units (e.g., 8x8 for CDEF) included in the reconstruction filtering unit is greater than a predetermined number, a majority vote or consistency check of the directional information units included in the reconstruction filtering unit may be performed to determine the filter class. The directional information units may have various sizes and different directional intra-prediction modes.

[0166] In one embodiment, in the case of a majority vote, the most frequent directionality among the available, possibly merged, directions may be selected, and in one example, a certain margin may be set between the first and second most frequent directionality.

[0167] In one embodiment, before taking the majority vote, it is determined whether the number of classes in the reconstruction filtering unit is greater than a predetermined number, and if true, an inconsistency may be declared and either explicit signaling for the reconstruction filtering unit or the use of a smaller reconstruction filtering unit may be selected.

[0168] According to some embodiments, the reconstruction filtering unit in a chroma component may use its own directional information from a directional intra prediction mode if such information is available for the chroma component separately.

[0169] According to an aspect of the present disclosure, when directional information is available from both the intra prediction mode and the CDEF process, the directional mode of the intra prediction mode, together with the directional information from the CDEF process, may be used as a guide to identify and classify a reconstruction filtering unit (e.g., a Wiener or SGRPRJ filtering unit).

[0170] In one embodiment, if directionality information is available from both the intra-prediction direction and the CDEF process, a mapping of directions from both sources may be incorporated when checking the consistency of the directionality information. The classification-based restoration filtering process may be performed only if the directionality information is consistent. Figure 15 shows an example in which the intra-prediction directions corresponding to seven delta angles associated with one of the eight cardinal angles in AV1 may be mapped to a single directionality class, thus forming a one-to-one correspondence with the eight directions derived during the CDEF process.

[0171] In one embodiment, directionality information from either an intra-prediction mode or a CDEF process may be used as a first source, and the other of the intra-prediction mode or the CDEF process may be used only if a consistent direction cannot be determined with the directionality information from the first source.

[0172] Note that in some examples, such as AV1, the Wiener filter and the SGRPRJ filter may be adaptively selected per LRU. The determination of the filter parameters and filtering process may be performed per LRU in the same way for both filters. The difference may be the type and number of filter parameters.

[0173] In one embodiment, when an SGRPRJ filter is implemented, a search window size r and a noise parameter e may be defined and used for each signal class, similar to the Wiener filter parameters.

[0174] In one embodiment, when an SGRPRJ filter is implemented, projection parameters α and β may be defined and used for each signal class, similar to the Wiener filter parameters.

[0175] In one embodiment, when an SGRPRJ filter is implemented, the search window size r, noise parameter e, and projection parameters α and β may be defined and used for each signal class, similar to the Wiener filter parameters.

[0176] [V. Flowchart] 16 shows a flowchart outlining an exemplary process (1600) according to one embodiment of the present disclosure. In various embodiments, the process (1600) is performed by a processing circuit such as a processing circuit in the terminal devices (210), (220), (230), and (240), a processing circuit performing the functions of the video encoder (303), a processing circuit performing the functions of the video decoder (310), a processing circuit performing the functions of the video decoder (410), a processing circuit performing the functions of the intra prediction module (452), a processing circuit performing the functions of the video encoder (503), a processing circuit performing the functions of the predictor (535), a processing circuit performing the functions of the intra encoder (622), a processing circuit performing the functions of the intra decoder (772), etc. In some embodiments, the process (1600) is implemented with software instructions, and thus, the processing circuit performs the process (1600) when the processing circuit executes the software instructions.

[0177] The process (1600) may generally begin at step (S1610), where the process (1610) determines directional information for a reconstruction filter unit included in a video frame based on at least one of a CDEF process or an intra-prediction mode. The process (1600) then proceeds to step (S1620).

[0178] In step (S1620), the process (1600) determines one of a plurality of sets of filter parameters of the reconstruction filter process based on the directional information of the reconstruction filter unit. Then, the process (1600) proceeds to step (S1630).

[0179] In step S1630, the process 1600 performs a reconstruction filter process on the reconstruction filter unit based on one of the sets of filter parameters. The process 1600 then proceeds to step S1640.

[0180] In step S1640, the process 1600 reconstructs the video frame based on the filtered reconstruction filter unit, and then the process 1600 ends.

[0181] In one embodiment, the reconstruction filter unit includes one or more directional information units, and at least one of the CDEF process or intra prediction mode is performed on one of the one or more directional information units.

[0182] In one embodiment, each of the sets of filter parameters of the reconstruction filter is associated with at least one directionality of the CDEF process.

[0183] In one embodiment, the process (1600) determines one of multiple sets of filter parameters for the reconstruction filter process based on the directionality information and block variance information of the reconstruction filter unit.

[0184] In one embodiment, the process (1600) determines one of a plurality of sets of filter parameters for the reconstruction filter process based on the directional information of the reconstruction filter unit and the filter strength of the CDEF process.

[0185] In one embodiment, the process (1600) determines the directional information of the reconstruction filter units based on at least one of a majority vote or a consistency check of the directionalities in the reconstruction filter units.

[0186] In one embodiment, based on the fact that the reconstruction filter unit is not intra-coded and the neighboring blocks are intra-coded, the process (1600) determines the directionality information of the reconstruction filter unit based on the intra-prediction mode performed on the neighboring blocks of the reconstruction filter unit.

[0187] In one embodiment, the process (1600) performs a reconstruction filter process for a reconstruction filter unit based on directional information determined from the CDEF process that matches directional information determined from the intra prediction mode.

[0188] In one embodiment, the reconstruction filter process is one of a Wiener filter process and an SGRPRJ filter process.

[0189] In one embodiment, the process determines one of multiple sets of filter parameters for the reconstruction filter process based on a default set of filter parameters, a communicated index indicating a set of filter parameters, or one of the communicated sets of filter parameters.

[0190] [VI. Computer Systems] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 17 illustrates a computer system (1700) suitable for implementing certain embodiments of the disclosed subject matter.

[0191] Computer software may be encoded using any suitable machine code or computer language, which may undergo assembly, compilation, linking, or similar mechanisms to generate code including instructions that may be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., either directly or through interpretation, microcode execution, etc.

[0192] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0193] 17 for computer system (1700) are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components shown in the exemplary embodiment of computer system (1700).

[0194] The computer system 1700 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic pictures).

[0195] The input human interface devices may include one or more of a keyboard (1701), a mouse (1702), a trackpad (1703), a touch screen (1710), a data glove (not shown), a joystick (1705), a microphone (1706), a scanner (1707), and a camera (1708).

[0196] The computer system (1700) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1710), data gloves (not shown), or joystick (1705), although haptic feedback devices that do not function as input devices may also be present), audio output devices (speakers (1709), headphones (not shown), etc.), visual output devices (including screens (1710) including CRT, LCD, plasma, and OLED screens, each of which may or may not have touchscreen input capability, each of which may or may not have haptic feedback capability, and some of which may be capable of outputting two-dimensional visual output or three- or more-dimensional output through means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown). These visual output devices (such as a screen 1710) may be connected to the system bus 1748 through a graphics processor 1750.

[0197] The computer system (1700) may also include human-accessible storage devices and associated media such as optical media including CD / DVD ROM / RW (1720) with CD / DVD or similar media (1721), thumb drives (1722), removable hard drives or solid state drives (1723), legacy magnetic media such as tape and floppy disks (not shown), specialized ROM / ASIC / PLD-based devices such as security dongles (not shown), etc.

[0198] Additionally, those skilled in the art should understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other non-transitory signals.

[0199] The computer system 1700 may also include a network interface 1754 to one or more communication networks 1755. The one or more communication networks 1755 may be, for example, wireless, wired, optical, or the like. The one or more communication networks 1755 may be local, wide area, metropolitan, vehicular and industrial, real-time, delay-tolerant, or the like. Examples of the one or more communication networks 1755 include Ethernet, wireless LAN, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), TV wired or wireless wide area digital networks (including cable TV, satellite TV, and terrestrial broadcast TV), vehicular and industrial (including CANBus), and the like. Certain networks typically require external network interface adapters (e.g., USB ports on the computer system 1700) attached to specific general-purpose data ports or peripheral buses 1749, while other network interface adapters are typically integrated into the core of the computer system 1700 by attaching them to a system bus (e.g., an Ethernet interface to a PC computer system or a cellular network to a smartphone computer system), as described below. Using any of these networks, the computer system 1700 can communicate with other entities. Such communications may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to certain CANbus devices), or bidirectional, for example, to other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks may be used in each of these networks and network interfaces.

[0200] The above-mentioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (1740) of the computer system (1700).

[0201] The core (1740) may include one or more central processing units (CPUs) (1741), graphics processing units (GPUs) (1742), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (1743), task-specific hardware accelerators (1774), graphics adapters (1750), etc. These devices, along with read-only memory (ROM) (1745), random access memory (1746), and internal mass storage devices (1747) (such as internal non-user-accessible hard drives, SSDs, etc.), may be connected through a system bus (1748). In some computer systems, the system bus (1748) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1748) or through a peripheral bus (1749). In one example, the screen 1710 may be attached to a graphics adapter 1750. Peripheral bus architectures include PCI, USB, etc.

[0202] The CPU (1741), GPU (1742), FPGA (1743), and accelerator (1744) may execute specific instructions, which in combination may constitute the above-mentioned computer code. The computer code may be stored in ROM (1745) or RAM (1746). Temporary data may be stored in RAM (1746), while persistent data may be stored in, for example, an internal mass storage device (1747). A cache memory, which may be closely associated with one or more of the CPU (1741), GPU (1742), mass storage device (1747), ROM (1745), RAM (1746), etc., may be used to enable fast storage and retrieval of data from any of the memory devices.

[0203] The computer-readable medium may include computer code for performing various computer-implemented operations, and the medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0204] By way of example and not limitation, the architecture (1700), and in particular a computer system having a core (1740), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with such user-accessible mass storage devices, as well as specific storage of the core (1740) that is non-transitory in nature, such as the core's internal mass storage (1747) or ROM (1745). Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core (1740). The computer-readable media may include one or more memory devices or chips according to particular needs. The software may cause the core (1740), and in particular the processor (including a CPU, GPU, FPGA, etc.) therein, to perform particular processes or portions of particular processes described herein, including defining data structures stored in RAM (1746) and modifying such data structures according to software-defined operations. Additionally or alternatively, a computer system may provide functionality as a result of hardwired or otherwise embodied logic in circuitry (e.g., accelerator (1744)), which may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure includes any appropriate combination of hardware and software.

[0205] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be recognized that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

[0206] [Appendix A: Abbreviations] ALF: Adaptive Loop Filter AMVP: Advanced Motion Vector Prediction APS: Adaptation Parameter Set ASIC: Application-Specific Integrated Circuit ATMVP: Alternative / Advanced Temporal Motion Vector Prediction AV1: AOMedia Video 1 AV2: AOMedia Video 2 BMS: Benchmark Set BV: Block Vector CANBus: Controller Area Network Bus CB: Coding Block CC-ALF: Cross-Component Adaptive Loop Filter CD: Compact Disc CDEF: Constrained Directional Enhancement Filter CPR: Current Picture Referencing CPU: Central Processing Unit CRT: Cathode Ray Tube CTB: Coding Tree Block CTU: Coding Tree Unit CU: Coding Unit DPB: Decoder Picture Buffer DPCM: Differential Pulse-Code Modulation DPS: Decoding Parameter Set DVD: Digital Video Disc FPGA: Field Programmable Gate Area JCCR: Joint CbCr Residual Coding JVET: Joint Video Exploration Team GOP: Groups of Pictures GPU: Graphics Processing Unit GSM: Global System for Mobile communications HDR: High Dynamic Range HEVC: High Efficiency Video Coding HRD: Hypothetical Reference Decoder IBC: Intra Block Copy IC: Integrated Circuit ISP: Intra Sub-Partitions JEM: Joint Exploration Model LAN: Local Area Network LCD: Liquid-Crystal Display LR: Loop Restoration Filter LRU: Loop Restoration Unit LTE: Long-Term Evolution MPM: Most Probable Mode MV: Motion Vector OLED: Organic Light-Emitting Diode PBs: Prediction Blocks PCI: Peripheral Component Interconnect PDPC: Position Dependent Prediction Combination PLD: Programmable Logic Device PPS: Picture Parameter Set PU: Prediction Unit RAM: Random Access Memory ROM: Read-Only Memory SAO: Sample Adaptive Offset SCC: Screen Content Coding SDR: Standard Dynamic Range SEI: Supplementary Enhancement Information SNR: Signal Noise Ratio SPS: Sequence Parameter Set SSD: Solid-state Drive TU: Transform Unit USB: Universal Serial Bus VPS: Video Parameter Set VUI: Video Usability Information VVC: Versatile Video Coding WAIP: Wide-Angle Intra Prediction

Claims

1. 1. A method of video decoding in a decoder, comprising: determining, based on the directionality of each of a plurality of groups of pixels of a video frame determined by at least one of a constrained directional emphasis filter (CDEF) process or an intra-prediction coding process, directionality information of a reconstruction filter unit formed by merging the plurality of groups of pixels according to a scan order; selecting a set of filter parameters from a plurality of sets of filter parameters for a reconstruction filter process based on the directional information of the reconstruction filter unit; After the at least one of the CDEF process or the intra-prediction coding process, performing one of a Wiener filter process or a self-guided projection filter process as the reconstruction filter process for the reconstruction filter unit based on the selected set of filter parameters; Reconstructing the video frame based on the filtered reconstruction filter unit; A method comprising:

2. each of the plurality of groups of pixels is a directionality information unit; the reconstruction filter unit includes a plurality of directional information units; The method of claim 1 , wherein the at least one of the CDEF process or the intra-predictive coding process is performed on one of the plurality of directional information units.

3. The method of claim 1 or 2, wherein each of the sets of filter parameters of the reconstruction filter process is associated with at least one directionality of the CDEF process.

4. The step of selecting a set of filter parameters comprises:

4. The method according to claim 1, further comprising determining one of the sets of filter parameters of the reconstruction filter process based on the directional information and block variance information of the reconstruction filter unit.

5. The step of selecting a set of filter parameters comprises:

4. The method according to claim 1, further comprising determining one of the sets of filter parameters of the reconstruction filter process based on the directional information of the reconstruction filter unit and a filter strength of the CDEF process.

6. The step of determining the directional information of the reconstruction filter unit comprises:

6. The method according to claim 1, further comprising determining the directional information of the reconstruction filter unit based on at least one of a majority vote or a consistency check of the directionalities of the groups of pixels in the reconstruction filter unit.

7. If the reconstruction filter unit is not intra-coded and the neighboring blocks of the reconstruction filter unit are intra-coded, The step of determining the directional information of the reconstruction filter unit comprises:

6. The method according to claim 1, comprising determining the directionality information of the reconstruction filter unit based on intra-predictive coding previously performed on the neighboring blocks of the reconstruction filter unit.

8. The performing step includes:

8. The method according to claim 1, further comprising: performing the one of the Wiener filter process or the self-guided projection filter process on the reconstruction filter unit based on whether the directional information determined from the CDEF process matches the directional information determined from the intra-prediction coding process.

9. The method of claim 1 , wherein the performing step comprises determining boundary edges of the reconstruction filter process based on the selected set of filter parameters.

10. The step of selecting a set of filter parameters comprises:

4. The method of claim 1, further comprising determining one of the sets of filter parameters of the reconstruction filter process based on (i) a default set of filter parameters, (ii) a communicated index indicating a set of filter parameters, or (iii) one of the communicated sets of filter parameters.

11. 1. An apparatus including a processing circuit, Apparatus, wherein the processing circuitry is configured to perform the method of any one of claims 1 to 10.

12. A program which, when executed by at least one processor, causes said at least one processor to carry out the method of any one of claims 1 to 10.

13. 1. A method of video encoding in an encoder, comprising: determining, based on the directionality of each of the groups of pixels determined by at least one of a constrained directional emphasis filter (CDEF) process or an intra-prediction coding process, directionality information of a reconstruction filter unit formed by merging the groups of pixels according to a scan order; selecting a set of filter parameters from a plurality of sets of filter parameters for a reconstruction filter process based on the directional information of the reconstruction filter unit; Including, after the at least one of the CDEF process or the intra-prediction coding process, one of a Wiener filter process or a self-guided projection filter process is performed as the reconstruction filter process for the reconstruction filter unit based on the selected set of filter parameters.

Images