Video coding method and device

Abstract

Aspects of the present disclosure include a video encoding / decoding method, an apparatus, and a computer-readable non-transient storage medium. The apparatus includes a processing circuit that determines directional information of a restoration filter unit included in a video frame based on at least one of a constrained direction enhancement filter (CDEF) process or an intra-prediction mode. The processing circuit determines one of a plurality of filter parameter sets of a restoration filter process based on the directional information of the restoration filter unit. The processing circuit performs the restoration filter process on the restoration filter unit based on one of the plurality of filter parameter sets. The processing circuit reconstructs the video frame based on the filtered restoration filter unit.

Description

Technology Field

[0001] Integration by reference

[0002] The present application claims the benefit of priority to U.S. Provisional Application No. 63 / 079,322 "Cross-Component Filtering on Block-Level Reconstruction," filed on September 16, 2020, and the benefit of priority to U.S. Patent Application No. 17 / 315,989 "Method and Apparatus for Video Coding," filed on May 10, 2021.

[0003] The present application claims priority to U.S. Provisional Application No. 63 / 091,707, "FEATURE INFORMATION REUSE FOR ENHANCED RESTORATION FILTERING," filed October 14, 2020, and the benefit of priority to U.S. Patent Application No. 17 / 362,048, "METHOD AND APPARATUS FOR VIDEO CODEDING," filed June 29, 2021. The entire disclosures of these prior applications are incorporated into the present application by reference.

[0004] This disclosure generally relates to video coding. Background Technology

[0005] The background description provided in this specification is generally intended to provide the context of the present disclosure. To the extent described in this background section, the works of the currently named inventors and aspects of the description that may not be recognized as prior art at the time of filing are not recognized as prior art of the present disclosure, either expressly or impliedly.

[0006] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video may contain a series of pictures, each having, for example, a spatial dimension of 1920×1080 luminance samples (also called luminance samples) and associated chrominance samples (also called chroma samples). The series of pictures may have, for example, 60 pictures per second or a fixed or variable picture rate of 60Hz (informally also known as the frame rate). Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at a 60Hz frame rate) requires bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600GB of storage space.

[0007] One objective of video coding and decoding may be to reduce the redundancy of the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage requirements by more than double orders of magnitude in some cases. Both lossless compression and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application. For video, lossy compression is widely employed. The amount of acceptable distortion varies depending on the application. For example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio may reflect that the higher the acceptable / tolerated distortion, the higher the compression ratio can be.

[0008] Video encoders and decoders can utilize techniques from a wide range of categories, including, for example, motion compensation, transform, quantization, and entropy coding.

[0009] Video codec technologies may include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples of a previously reconstructed reference picture or other data. In some video codecs, a picture is spatially subdivided into sample blocks. If all blocks of samples are coded in intra mode, that picture may be an intra-picture. Derivations such as intra-pictures and independent decoder refresh pictures can be used to reset the decoder state and thus serve as the first picture or still image of the coded video bitstream and video session. Samples in an intra-block can be exposed to transformation, and transformation coefficients can be quantized before entropy coding. Intra-prediction may be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after conversion and the smaller the AC coefficient, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0010] For example, traditional intra-coding, such as that known from MPEG-2 generative coding techniques, does not use intra-prediction. However, some new video compression techniques include methods that attempt to use, for example, surrounding sample data and / or metadata acquired during the encoding / decoding of spatially neighboring and preceding data blocks in the decoding order. These techniques will henceforth be referred to as "intra-prediction" techniques. In at least some cases, intra-prediction uses only reference data from the current picture being reconstructed, rather than the reference picture.

[0011] There may be various forms of intra-prediction. If one or more of these techniques can be used in a given video coding technique, the technique in use may be coded in an intra-prediction mode. In some cases, a mode may have submodes and / or parameters, which may be coded individually or included in a mode codeword. Which codeword is used for a given combination of mode, submode, and / or parameter can affect the coding efficiency gain through intra-prediction, and thus may also affect the entropy coding technique used to convert the codeword into a bitstream.

[0012] Intra prediction of specific modes was introduced with H.264, improved in H.265, and further enhanced in modern coding techniques such as the Joint Exploration Model (JEM), Universal Video Coding (VVC), and Benchmark Set (BMS). A predictor block can be formed using neighbor samples belonging to already available samples. The sample values ​​of neighbor samples are copied to the predictor block according to the direction. References to the direction in use can be encoded in the bitstream or predicted themselves.

[0013] Referring to FIG. 1a, a subset of nine known predictor directions from the 33 possible predictor directions of H.265 (corresponding to the 33 angle modes of the 35 intra modes) is shown in the lower right. The point where the arrows converge (101) indicates the predicted sample. The arrows indicate the direction in which the sample is predicted. For example, arrow (102) indicates that the sample (101) is predicted to the upper right at an angle of 45 degrees from the horizontal from one sample or sample. Similarly, arrow (103) indicates that the sample (101) is predicted to the lower left of the sample (101) at an angle of 22.5 degrees from the horizontal from one sample or sample.

[0014] Referring still to FIG. 1a, a square block (104) of 4×4 samples (indicated by a thick dashed line) is shown in the upper left corner. The square block (104) contains 16 samples, each labeled "S", a position in the Y dimension (e.g., column index), and a position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample of the block (104) in both the Y and X dimensions. Since the block size is 4×4 samples, S44 is located in the lower right corner. Additional reference samples following a similar numbering scheme are shown. The reference samples are labeled R, a Y position (e.g., row index) and an X position (column index) relative to the block (104). In both H.264 and H.265, the predicted sample is adjacent to the block being rebuilt; therefore, there is no need to use negative values.

[0015] Intra-picture prediction can be performed by copying reference sample values ​​from appropriate neighbor samples by the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating a prediction direction corresponding to the arrow (102) for this block—that is, samples are predicted from one prediction sample or samples located at the top right at a 45-degree angle from the horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0016] In some cases, to calculate reference samples, especially when the direction cannot be divided equally into 45 degrees, the values ​​of multiple reference samples may be combined through interpolation, for example.

[0017] As video coding technology has advanced, the number of possible directions has increased. H.264 (2003) could represent nine different directions. H.265 (2013) increased this to 33, and at the time of its release, JEM / VVC / BMS could support up to 65 directions. Experiments were conducted to identify the most likely direction, and specific techniques of entropy coding were used to represent such most likely directions with a small number of bits, thereby accommodating a specific penalty for less likely directions. Additionally, the direction itself can be predicted from neighboring directions used in already decoded neighboring blocks.

[0018] FIG. 1b illustrates a schematic diagram (105) showing 65 intra-predicted directions according to JEM to explain the number of predicted directions increasing over time.

[0019] The mapping of intra-predicted direction bits within a coded video bitstream representing direction can vary depending on the video coding technique; for example, it ranges from a simple direct mapping of the predicted direction to complex adaptive schemes involving intra-predicted modes, codewords, and the most probable mode, as well as similar techniques. However, in all cases, there may be a specific direction in the video content that is statistically less likely to occur than other specific directions. Since the goal of video compression is to reduce redundancy, in a well-functioning video coding technique, less likely directions are represented by a larger number of bits than more likely directions.

[0020] Motion compensation may be a lossy compression technique and may relate to a technique used to predict a newly reconstructed picture or part of a picture after a block of sample data from a previously reconstructed picture or part thereof (reference picture) has been spatially moved in a direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be identical to the picture currently being reconstructed. The MV may be two or three dimensions in X and Y, with a third being an indication of the reference picture in use (the latter may indirectly be a time dimension).

[0021] In some video compression techniques, an MV applicable to a specific area of ​​sample data can be predicted from other MVs, for example, from MVs associated with other areas of sample data that are spatially adjacent to the area being reconstructed and precede it in the decoding order. This significantly reduces the amount of data required to code the MV, thereby eliminating redundancy and increasing the compression ratio. For example, when coding an input video signal derived from a camera (referred to as natural video), MV prediction can work effectively because there is a statistical probability that an area larger than the area where a single MV can be applied will shift in a similar direction; thus, in some cases, a similar MV derived from the MVs of neighboring areas can be used for prediction. As a result, the MV found for a given area is similar or identical to the MV predicted from surrounding MVs, which can be represented by fewer bits than would have been used if the MV had been coded directly after entropy coding. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself may be lost, for example, because rounding error occurs when calculating the predictor from several surrounding MVs.

[0022] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms provided by H.265, the technique described here is hereinafter referred to as "spatial merge".

[0023] Referring to FIG. 1c, the current block (111) may include samples discovered by the encoder during the motion search process so as to be predictable from a previous block of the same size that is spatially shifted. Instead of directly coding the MV, the MV may be derived from metadata associated with one or more reference pictures, for example from the most recent (in decoding order) reference picture, using an MV associated with one of five neighboring samples denoted as A0, A1 and B0, B1, B2 (112–116, respectively). In H.265, the MV prediction may use a predictor from the same reference picture that is being used by the neighboring block.

[0024] The aspects of the present disclosure provide a video encoding / decoding device. The device includes a processing circuit that determines directional information of a restoration filter unit included in a video frame based on at least one of a constrained directional enhancement filter (CDEF) process or an intra-prediction mode. The processing circuit determines one of a plurality of filter parameter sets of a restoration filter process based on the directional information of the restoration filter unit. The processing circuit performs the restoration filter process on the restoration filter unit based on one of the plurality of filter parameter sets. The processing circuit reconstructs the video frame based on the filtered restoration filter unit.

[0025] In one embodiment, the restoration filter unit includes one or more directional information units, and at least one of the CDEF process or the intra prediction mode is performed for one of the one or more directional information units.

[0026] In one embodiment, each of the plurality of filter parameter sets of the restoration filter is associated with at least one directionality of the CDEF process.

[0027] In one embodiment, the processing circuit determines one of a plurality of filter parameter sets of the restoration filter process based on directional information and block distribution information of the restoration filter unit.

[0028] In one embodiment, the processing circuit determines one of a plurality of filter parameter sets of the restoration filter process based on the directional information of the restoration filter unit and the filter strength of the CDEF process.

[0029] In one embodiment, the processing circuit determines the directionality information of the restoration filter unit based on at least one of a majority vote or a consistency check regarding the directionality in the restoration filter unit.

[0030] In one embodiment, the processing circuit determines directional information of the restoration filter unit based on an intra prediction mode performed on neighbor blocks of the restoration filter unit and based on the restoration filter unit that is not intra-coded and neighbor blocks of the restoration filter unit that are not intra-coded.

[0031] In some possible embodiments, the processing circuit performs a restoration filter process on the restoration filter unit based on directional information determined in the CDEF process that matches the directional information determined from the intra prediction mode.

[0032] In one embodiment, the filter restoration process is one of a Wiener filter process and a self-guided projection (SGRPRJ) filter process.

[0033] In one embodiment, the processing circuit determines one of a plurality of filter parameter sets of the restoration filter process based on one of a default filter parameter set, a signaled index indicating a filter parameter set, or a signaled filter parameter set.

[0034] The aspects of the present disclosure provide a video encoding / decoding method. In the method, directional information of a restoration filter unit included in a video frame is determined based on at least one of a CDEF process or an intra-prediction mode. Based on the directional information of the restoration filter unit, one of a plurality of filter parameter sets of a restoration filter process is determined. Based on the one of the plurality of filter parameter sets, the restoration filter process is performed on the restoration filter unit. The video frame is reconstructed based on the filtered restoration filter unit.

[0035] The aspects of the present disclosure also provide a computer-readable, non-transient storage medium for storing instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to perform any one or a combination of the video decoding methods. Brief explanation of the drawing

[0036] Additional features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. Figure 1a is a schematic diagram of an exemplary subset of intra-prediction modes. Figure 1b is a diagram of an exemplary intra-predicted direction. Figure 1c is a schematic diagram of the current block and its surrounding space merge candidates in one example. FIG. 2 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. FIG. 3 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. FIG. 4 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. FIG. 5 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. FIG. 6 illustrates a block diagram of an encoder according to another embodiment. FIG. 7 illustrates a block diagram of a decoder according to another embodiment. FIG. 8 illustrates an exemplary nominal angle according to one embodiment of the present disclosure. FIG. 9 illustrates the positions of the top, left, and top-left samples for a single pixel in the current block according to one embodiment of the present disclosure. FIG. 10 illustrates an exemplary recursive filter intra mode according to one embodiment of the present disclosure. FIG. 11 illustrates some exemplary directions of a constrained direction enhancement filter (CDEF) process according to some embodiments of the present disclosure. FIG. 12 illustrates some exemplary block partitions according to some embodiments of the present disclosure. FIG. 13 illustrates an example in which a directional unit block is merged into a filtering unit according to one embodiment of the present disclosure. FIG. 14 illustrates another example in which a directional unit block is merged into a filtering unit according to one embodiment of the present disclosure. FIG. 15 illustrates an exemplary mapping of directional information between a directional intra-prediction mode and a dominant direction derived during a CDEF process according to some embodiments of the present disclosure. FIG. 16 illustrates an exemplary flowchart according to one embodiment of the present disclosure. FIG. 17 is a schematic diagram of a computer system according to one embodiment. Specific details for implementing the invention

[0037] I. Video Decoder and Encoder System

[0038] FIG. 2 shows a simplified block diagram of a communication system (200) according to one embodiment of the present disclosure. The communication system (200) includes a plurality of terminal devices capable of communicating with each other, for example, through a network (250). For example, the communication system (200) includes a first pair of terminal devices (210, 220) interconnected through the network (250). In the example of FIG. 2, the first pair of terminal devices (210, 220) perform unidirectional transmission of data. For example, the terminal device (210) may encode video data (e.g., a stream of video pictures captured by the terminal device (210)) for transmission to another terminal device (220) through the network (250). The encoded video data may be transmitted in the form of one or more coded video bit streams. The terminal device (220) receives coded video data from the network (250), decodes the coded video data to restore a video picture, and can display the video picture according to the restored video data. Unidirectional data transmission may be common in media serving applications, etc.

[0039] In another example, the communication system (200) includes a second pair of terminal devices (230, 240) that perform bidirectional transmission of coded video data, which may occur, for example, during a video conference. For bidirectional transmission of data, in one example, each terminal device of the pair of terminal devices (230, 240) may code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to another terminal device of the pair of terminal devices (230, 240) via a network (250). Each terminal device of the pair of terminal devices (230, 240) may also receive coded video data transmitted by another terminal device of the pair of terminal devices (230, 240), decode the coded video data to restore the video, and display the video picture on an accessible display device according to the restored video data.

[0040] In the example of FIG. 2, the terminal devices (210, 220, 230, 240) may be exemplified as servers, personal computers, and smartphones, but the principles of the present disclosure may not be limited thereto. Embodiments of the present disclosure find applications using laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (250) represents any number of networks that transmit coded video data between terminal devices (210, 220, 230, 240), including, for example, wired and / or wireless communication networks. The communication network (250) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include communication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (250) may not be important to the operation of the present disclosure unless described below.

[0041] FIG. 3 illustrates the placement of a video encoder and a video decoder in a streaming environment as an example of an application for the disclosed subject. The disclosed subject may be equally applicable to other video-enabled applications, such as video conferencing, digital TV, CD, DVD, memory stick, etc., including the storage of compressed video on digital media.

[0042] A streaming system may include a video source (301) and a capture subsystem (313) which may include, for example, a digital camera, to generate, for example, a stream (302) of uncompressed video pictures. In one example, the stream (302) of video pictures includes samples captured by a digital camera. The stream (302) of video pictures, indicated by a bold line to emphasize a high data volume compared to encoded video data (304) (or coded video bitstream), may be processed by an electronic device (320) comprising a video encoder (303) coupled to the video source (301). The video encoder (303) may include hardware, software, or a combination thereof that can enable or implement aspects of the disclosed subject matter as described in more detail below. The encoded video data (304) (or encoded video bitstream (304)) is indicated by a thin line to emphasize a lower data volume compared to the stream (302) of the video picture and can be stored in a streaming server (305) for later use. One or more streaming client subsystems, such as the client subsystems (306, 308) in FIG. 3, can access the streaming server (305) to retrieve copies (307, 309) of the encoded video data (304). The client subsystem (306) may include, for example, a video decoder (310) in an electronic device (330). The video decoder (310) decodes the incoming copy (307) of the encoded video data and generates an outgoing stream of the video picture (311) that can be rendered on a display (312) (e.g., a display screen) or another rendering device (not shown). In some streaming systems, encoded video data (304, 307, 309) (e.g., video bitstream) may be encoded according to a specific video coding / compression standard.Examples of such standards include ITU-T Recommendation H.265. For instance, a video coding standard under development is informally known as VVC (Versatile Video Coding). The disclosed topics may be used in the context of VVC.

[0043] Note that the electronic device (320, 330) may include other components (not shown). For example, the electronic device (320) may include a video decoder (not shown), and the electronic device (330) may also include a video encoder (not shown).

[0044] FIG. 4 illustrates a block diagram of a video decoder (410) according to one embodiment of the present disclosure. The video decoder (410) may be included in an electronic device (430). The electronic device (430) may include a receiver (431) (e.g., a receiving circuit). The video decoder (410) may be used instead of the video decoder (310) in the example of FIG. 3.

[0045] The receiver (431) may receive one or more coded video sequences to be decoded by the video decoder (410); in the same or different embodiments, at one coded video sequence at a time, the decoding of each coded video sequence is independent of other coded video sequences. The coded video sequences may be received from a channel (401) which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (431) may receive the encoded video data along with other data, e.g., encoded audio data and / or auxiliary data streams, which may be forwarded to each user entity (not shown). The receiver (431) may separate the coded video sequences from the other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (431) and the entropy decoder / parser (420) (hereinafter "parser (420)"). In certain applications, the buffer memory (415) is part of the video decoder (410). In other cases, it may be outside the video decoder (410) (not shown). In yet another case, the buffer memory (not shown) may be outside the video decoder (410), for example, to prevent network jitter, and additionally, another buffer memory (415) may be inside the video decoder (410), for example, to handle playout timing. When the receiver (431) is receiving data from a storage / forwarding device or an isosynchronous network that has sufficient bandwidth and controllability, the buffer memory (415) may not be needed or may be small.For use in a best-effort packet network such as the Internet, the buffer memory (415) may be required, may be relatively large, may have an adaptive size, and may be implemented at least partially in a similar element (not shown) outside of the operating system or video decoder (410).

[0046] The video decoder (410) may include a parser (420) to reconstruct symbols (421) from a coded video sequence. Categories of these symbols include information used to manage the operation of the video decoder (410) and potential information for controlling a rendering device (412) (e.g., a display picture) that is not an integrated part of the electronic device (430) but can be connected to the electronic device (430) as illustrated in FIG. 4. The control information for the rendering device(s) may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (420) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (420) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to a group. The subgroups may include a Group of Picture (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (420) may also extract transform coefficients, quantizer parameter values, MV, etc. from the coded video sequence information.

[0047] The parser (420) can generate symbols (421) by performing entropy decoding / parsing operations on a video sequence received from the buffer memory (415).

[0048] The reconstruction of the symbol (421) may include several different units depending on the type of the coded video picture or a part thereof (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors. Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by the parser (420). The flow of such subgroup control information between the parser (420) and the number of units below it is not shown for clarity.

[0049] Beyond the functional blocks already mentioned, the video decoder (410) can be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these units interact closely with one another and may be integrated with one another, at least partially. However, to explain the subject of disclosure, it is appropriate to conceptually subdivide it into the functional units below.

[0050] The first unit is a scaler / inverse transform unit (451). The scaler / inverse transform unit (451) receives quantized transform coefficients as well as control information, including transforms for use as symbol(s) (421) from the parser (420), block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (451) can output a block containing sample values ​​that can be input to an aggregator (455).

[0051] In some cases, the output sample of the scaler / inverse conversion unit (451) may be related to an intra-coded block, that is, a block that does not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by an intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) uses already reconstructed surrounding information drawn from the current picture buffer (458) to generate a block of the same size and shape as the block being reconstructed. The current picture buffer (458) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (455) adds, in some cases, the prediction information generated by the intra-prediction unit (452) to the output sample information provided by the scaler / inverse conversion unit (551) on a sample basis.

[0052] In other cases, the output samples of the scaler / inverse unit (451) may belong to an intercoded and potentially motion-compensated block. In this case, the motion-compensated prediction unit (453) may access the reference picture memory (457) to retrieve samples used for prediction. After motion-compensating the samples retrieved according to the symbols (421) belonging to the block, these samples may be added to the output of the scaler / inverse unit (451) (in this case, referred to as residual samples or residual signals) by the aggregator (455) to generate output sample information. The address in the reference picture memory (457) from which the motion-compensated prediction unit (453) retrieves the prediction samples may be controlled by the MV and may be used by the motion-compensated unit (453) in the form of a symbol (421) that may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​drawn from reference picture memory (457) when the accurate MV of the subsample is used, an MV prediction mechanism, etc.

[0053] The output sample of the aggregator (455) may be subject to various loop filtering techniques in the loop filter unit (456). The video compression technique may include an in-loop filter technique that is controlled by parameters contained in the coded video sequence (also referred to as the coded video bitstream) and made available to the loop filter unit (456) as a symbol (421) from the parser (420), but may also respond to meta-information obtained while decoding a previous (in decoding order) part of the coded picture or coded video sequence, as well as respond to previously reconstructed and loop-filtered sample values.

[0054] The output of the loop filter unit (456) may be a sample stream that can be output to the render device (412) as well as stored in the reference picture memory (457) for use in future inter-picture prediction.

[0055] Once a specific coded picture has been completely reconstructed, it can later be used as a reference picture for prediction. For example, when a coded picture corresponding to the current picture is completely reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (420)), the current picture buffer (458) can become part of the reference picture memory (457), and the new current picture buffer can be reallocated before starting the reconstruction of the next coded picture.

[0056] The video decoder (410) can perform decoding operations according to a video compression technology predetermined in a standard, such as ITU-T Rec. H.265. The coded video sequence may follow the syntax specified by the video compression technology or standard used in that it complies with both the syntax of the video compression technology or standard and the profile documented in the video compression technology. In particular, the profile may select a specific tool as the only tool available in that profile among all tools available in the video compression technology or standard. Additionally, for compliance to be required, the complexity of the coded video sequence may be within the range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further limited by HRD specifications and metadata for managing the virtual reference decoder (HRD) buffer signaled in the coded video sequence.

[0057] In one embodiment, the receiver (431) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (410) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, a temporal layer, a spatial layer or an SNR enhancement layer, a redundant slice, a redundant picture, a forward error correction code, etc.

[0058] FIG. 5 illustrates a block diagram of a video encoder (503) according to one embodiment of the present disclosure. The video encoder (503) is included in an electronic device (520). The electronic device (520) includes a transmitter (540) (e.g., a transmitting circuit). The video encoder (503) may be used instead of the video encoder (403) in the example of FIG. 3.

[0059] The video encoder (503) can receive video samples from a video source (501) (not part of the electronic device (520) in the example of FIG. 5) capable of capturing video image(s) to be encoded by the video encoder (503). In another example, the video source (501) is part of the electronic device (520).

[0060] A video source (501) may provide a source video sequence to be coded by a video encoder (503) in the form of a digital video sample stream, which may have any appropriate bit depth (e.g., 8-bit, 10-bit, 12-bit,…), any color space (e.g., BT.601 Y CrCB, RGB,…), and any appropriate sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (501) may be a storage device that stores a pre-prepared video. In a video conferencing system, the video source (501) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that impart motion when viewed sequentially. The picture itself may consist of a spatial array of pixels, and each pixel may contain one or more samples depending on the sampling structure, color space, etc. being used. A person skilled in the art will easily understand the relationship between pixels and samples. The following description focuses on the sample.

[0061] According to one embodiment, the video encoder (503) can code and compress a picture of a source video sequence into a coded video sequence (543) in real time or under any other time constraint required by the application. Enforcing an appropriate coding rate is one of the functions of the controller (550). In some embodiments, the controller (550) controls other function units and is functionally coupled to other function units as described below. For clarity, the coupling is not shown. Parameters set by the controller (550) may include rate control-related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique,…), picture size, picture group (GOP) layout, maximum MV allowable reference area, etc. The controller (550) may be configured to have other appropriate functions belonging to the video encoder (503) optimized for a specific system design.

[0062] In some embodiments, the video encoder (503) is configured to operate in a coding loop. For an overly simplified description, in one example, the coding loop may include a source coder (530) (e.g., responsible for generating symbol and reference picture(s), such as a symbol stream, based on the input picture to be coded), and a (local) decoder (533) embedded in the video encoder (503). The decoder (533) reconstructs the symbols to generate sample data in the same way that the (remote) decoder also generates (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques considered in the disclosed subject). The reconstructed sample stream (sample data) is input into the reference picture memory (534). Since the decoding of the symbol stream leads to a bit-exact result regardless of the decoder location (local or remote), the contents of the reference picture memory (534) are also bit-exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "recognizes" as reference picture samples exactly the same sample values ​​that the decoder "sees" when using prediction during decoding. The basic principle of reference picture synchronicity (and, for example, where simultaneity cannot be maintained due to channel error, resulting drift) is also used in some related techniques.

[0063] The operation of the “local” decoder (533) may be the same as the operation of the “remote” decoder, such as the video decoder (410), which has already been described in detail in relation to FIG. 4. Briefly referring also to FIG. 4, since the encoding / decoding of symbols into a video sequence coded by the entropy coder (545) and the parser (420) may be lossless, the entropy decoding part of the video decoder (410) may not be fully implemented in the local decoder (533), including the buffer memory (415) and the parser (420).

[0064] What can be observed at this point is that, with the exception of parsing / entropy decoding present in the decoder, all decoder techniques must exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject focuses on decoder operation. Descriptions of encoder techniques may be omitted as they are the opposite of the comprehensively described decoder techniques. Further details are required only in specific areas and are provided below.

[0065] In operation, in some examples, the source coder (530) may perform motion-compensated predictive coding to predictively code an input picture by referencing one or more previously coded pictures from a video sequence designated as a "reference picture." In this way, the coding engine (532) codes the difference between a pixel block of the input picture and a pixel block of the reference picture(s) that can be selected as predictive reference(s) for the input picture.

[0066] The local video decoder (533) can decode the coded video data of a picture that can be designated as a reference picture based on symbols generated by the source coder (530). The operation of the coding engine (532) can advantageously be a lossy process. When the coded video data can be decoded by a video decoder (not shown in FIG. 5), the reconstructed video sequence may generally be a replica of the source video sequence with some errors. The local video decoder (533) can replicate the decoding process that can be performed by the video decoder on the reference picture and allow the reconstructed reference picture to be stored in the reference picture cache (534). In this way, the video encoder (503) can locally store a copy of the reconstructed reference picture having common content as the reconstructed reference picture to be acquired by the far-end video decoder (no transmission errors).

[0067] The predictor (535) can perform a predictive search for the coding engine (532). That is, for a new picture to be coded, the predictor (535) can search the reference picture memory (534) for sample data (which is a candidate reference pixel block) that can serve as a suitable predictive reference for the new picture, or for specific metadata such as the reference picture MV, block shape, etc. The predictor (535) can operate on a sample block-by-pixel block basis to find a suitable predictive reference. In some cases, as determined by the search result obtained by the predictor (535), the input picture may have a predictive reference drawn from a number of reference pictures stored in the reference picture memory (534).

[0068] The controller (550) can manage the coding operation of the source coder (530), including the settings of parameters and subgroup parameters used to encode video data, for example.

[0069] The output of all the aforementioned function units may be subject to entropy coding in the entropy coder (545). The entropy coder (545) converts symbols generated by various function units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0070] The transmitter (540) may buffer the coded video sequence(s) generated by the entropy coder (545) and prepare for transmission through a communication channel (560), which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter (540) may merge the coded video data from the video coder (503) with other data to be transmitted, e.g., coded audio data and / or an auxiliary data stream (source not shown).

[0071] The controller (550) can manage the operation of the video encoder (503). During coding, the controller (550) can assign a specific coded picture type to each coded picture, which can affect the coding technique that can be applied to each picture. For example, a picture can often be designated as one of the following picture types:

[0072] An Intra Picture (I Picture) may be one that can be coded and decoded without using any other picture within the sequence as a prediction source. Some video codecs allow different types of Intra Pictures, including, for example, Independent Decoder Refresh Picture ("IDR"). Those skilled in the art are aware of these variations of I Pictures and their respective applications and characteristics.

[0073] The predictive picture (P picture) may be coded and decoded using intra-prediction or inter-prediction, which uses up to one MV and reference index to predict the sample values ​​of each block.

[0074] A bidirectionally predictive picture (B picture) may be coded and decoded using intra-prediction or inter-prediction, using up to two MVs and reference indices to predict the sample values ​​of each block. Similarly, a multiple-predictive picture may use more than two reference pictures and associated metadata to reconstruct a single block.

[0075] A source picture is typically subdivided spatially into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks) and can be coded block by block. Blocks can be predictively coded by referencing other (already coded) blocks as determined by the coding assignment applied to each picture in the block. For example, blocks in picture I can be non-predictively coded or predictively coded by referencing already coded blocks (spatial prediction or intra prediction) of the same picture. Pixel blocks in picture P can be predictively coded via spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks in picture B can be predictively coded via spatial prediction or temporal prediction by referencing one or two previously coded reference pictures.

[0076] The video encoder (503) can perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In the operation, the video encoder (503) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data can follow the syntax specified by the video coding technique or standard used.

[0077] In one embodiment, the transmitter (540) may transmit additional data along with the encoded video. The source coder (530) may include this data as part of the encoded video sequence. The additional data may include other forms of redundant data such as time / space / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0078] Video can be captured as a temporal sequence of multiple source pictures (video pictures). Intra-picture prediction (often abbreviated as intra prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes (temporal or other) correlations between pictures. In one example, a specific picture currently being encoded / decoded, referred to as the current picture, is partitioned into blocks. When a block within the current picture is similar to a reference block in a reference picture that was previously encoded and is still buffered in the video, the block within the current picture can be encoded as a vector referred to as MV. MV points to a reference block within the reference picture and, if multiple reference pictures are in use, may have three dimensions that identify the reference picture.

[0079] In some embodiments, a bi-prediction technique may be used for inter-picture prediction. According to the bi-prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture, in which the decoding order of the current picture in the video is both preceding (but may be past and future in display order, respectively). A block of the current picture may be coded by a first MV pointing to a first reference block of the first reference picture and a second MV pointing to a second reference block of the second reference picture. A block may be predicted by a combination of the first reference block and the second reference block.

[0080] In addition, coding efficiency can be improved by using merge mode technology in inter-picture prediction.

[0081] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in blocks. For example, according to the HEVC standard, pictures within a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs within the pictures have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU includes three coding tree blocks (CTBs), one luminance CTB and two chroma CTBs. Each CTU can be repeatedly quad-tree split into one or more coding units (CUs). For example, a 64×64 pixel CTU can be split into one 64×64 pixel CU, four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. The CU is divided into one or more prediction units (PUs) based on temporal and / or spatial predictability. Typically, each PU includes a luminal prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed on a unit of prediction blocks. If a luminal prediction block is used as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luminal values) for pixels, such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0082] FIG. 6 illustrates a video encoder (603) according to another embodiment of the present disclosure. The video encoder (603) is configured to receive a processing block (e.g., a prediction block) of sample values ​​within a current video picture in a video picture sequence and to encode the processing block into a coded picture that is part of a coded video sequence. In one example, the video encoder (603) is used instead of the video encoder (303) in the example of FIG. 3.

[0083] In the HEVC example, the video encoder (603) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (603) determines whether the processing block is best coded using an intra-mode, inter-mode, or positive prediction mode, for example, using rate distortion optimization. When the processing block is coded in intra-mode, the video encoder (603) may use an intra-prediction technique to encode the processing block into a coded picture; when the processing block is coded in inter-mode or positive prediction mode, the video encoder (603) may use an inter-prediction or positive prediction technique, respectively, to encode the processing block into a coded picture. In a particular video coding technique, the merge mode may be an inter-picture prediction submode in which the MV is derived from one or more MV predictors without the benefit of the coded MV component outside the predictor. In a particular other video coding technique, there may be an MV component applicable to the subject block. In one example, the video encoder (603) includes other components such as a mode determination module (not shown) for determining the mode of the processing block.

[0084] In the example of FIG. 6, the video encoder (603) includes an inter-encoder (630), an intra-encoder (622), a residual calculator (623), a switch (626), a residual encoder (624), and a general controller (621), and includes an entropy encoder (625) combined together as shown in FIG. 6.

[0085] The inter encoder (630) receives a sample of the current block (e.g., processing block), compares the block with one or more reference blocks in a reference picture (e.g., blocks in the previous picture and the next picture), generates inter prediction information (e.g., description of redundancy information according to inter encoding technique, MV, merge mode information), and calculates an inter prediction result (e.g., prediction block) based on the inter prediction information using any appropriate technique. In some examples, the reference picture is a decoded reference picture that is decoded based on encoded video information.

[0086] The intra-encoder (622) receives a sample of the current block (e.g., processing block), and in some cases, compares the block with a block already coded in the same picture, generates quantized coefficients after conversion, and in some cases also generates intra-prediction information (e.g., intra-prediction direction information according to one or more intra-encoding techniques). In one example, the intra-encoder (622) also calculates an intra-prediction result (e.g., prediction block) based on the intra-prediction information and reference block of the same picture.

[0087] A general controller (621) is configured to determine general control data and to control other components of the video encoder (603) based on the general control data. In one example, the general controller (621) determines the mode of the block and provides a control signal to the switch (626) according to the mode. For example, if the mode is an intra mode, the general controller (621) controls the switch (626) to select an intra mode result for use in the residual calculator (623) and controls the entropy encoder (625) to select intra prediction information and include the intra prediction information in the bit stream; if the mode is an inter mode, the general controller (621) controls the switch (626) to select an inter prediction result for use in the residual calculator (623) and controls the entropy encoder (625) to select inter prediction information and include the inter prediction information in the bit stream.

[0088] The residual calculator (623) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra-encoder (622) or the inter-encoder (630). The residual encoder (624) is configured to operate based on the residual data to encode the residual data and generate transformation coefficients. In one example, the residual encoder (624) is configured to convert the residual data from the spatial domain to the frequency domain and generate transformation coefficients. The transformation coefficients then undergo quantization processing to obtain quantized transformation coefficients. In various embodiments, the video encoder (603) also includes a residual decoder (628). The residual decoder (628) is configured to perform an inverse transformation and generate decoded residual data. The decoded residual data can be appropriately used by the intra-encoder (622) and the inter-encoder (630). For example, the inter encoder (630) can generate a decoded block based on decoded residual data and inter prediction information, and the intra encoder (622) can generate a decoded block based on decoded residual data and intra prediction information. The decoded block is appropriately processed to generate a decoded picture, and the decoded picture can be buffered in a memory circuit (not shown) and, in some examples, used as a reference picture.

[0089] The entropy encoder (625) is configured to format the bit stream to include the encoded block. The entropy encoder (625) is configured to include various information according to an appropriate standard, such as HEVC. In one example, the entropy encoder (625) is configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other appropriate information in the bit stream. Note that according to the disclosed subject invention, when coding a block in an inter mode or a merged submode of both prediction modes, there is no residual information.

[0090] FIG. 7 illustrates a drawing of a video decoder (710) according to another embodiment of the present disclosure. The video decoder (710) is configured to receive a coded picture that is part of a coded video sequence and to decode the coded picture to generate a reconstructed picture. In one example, the video decoder (710) is used instead of the video decoder (310) in the example of FIG. 3.

[0091] In the example of FIG. 7, the video decoder (710) includes an entropy decoder (771), an inter decoder (780), a residual decoder (773), a reconstruction module (774), and an intra decoder (772) combined together as shown in FIG. 7.

[0092] The entropy decoder (771) may be configured to reconstruct, from the coded picture, specific symbols representing the syntax elements constituting the coded picture. These symbols may include, for example, the mode in which the block is coded (e.g., intra mode, inter mode, positive prediction mode, merge submode, or the latter two in other submodes), prediction information (e.g., intra prediction information or inter prediction information) capable of identifying specific samples or metadata used for prediction by the intra decoder (772) or inter decoder (780), respectively, and residual information in the form of, for example, quantized transformation coefficients. In one example, if the prediction mode is an inter mode or positive prediction mode, the inter prediction information is provided to the inter decoder (780); if the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (772). The residual information may be dequantized and provided to the residual decoder (773).

[0093] The inter decoder (780) is configured to receive inter prediction information and generate an inter prediction result based on the inter prediction information.

[0094] The intra decoder (772) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0095] The residual decoder (773) is configured to perform inverse quantization to extract inverse quantized transform coefficients and process the inverse quantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (773) may also require specific control information (including quantizer parameters (QP)), and that information may be provided by the entropy decoder (771) (the data path may contain only a small amount of control information and is therefore not shown).

[0096] The reconstruction module (774) is configured to combine the residuals and prediction results output by the residual decoder (773) in the spatial domain (which may be output by the inter-prediction module or the intra-prediction module) to form a reconstructed block, and the reconstructed block may be part of a reconstructed picture, and the reconstructed picture may also be part of a reconstructed video. Note that other appropriate actions, such as deblocking, may be performed to improve visual quality.

[0097] It should be noted that the video encoder (303, 503, 603) and video decoder (310, 410, 710) can be implemented using any suitable technology. In one embodiment, the video encoder (303, 503, 603) and video decoder (310, 410, 710) can be implemented using one or more integrated circuits. In another embodiment, the video encoder (303, 503, 603) and video decoder (310, 410, 710) can be implemented using one or more processors that execute software instructions.

[0098] II. Intra-prediction

[0099] In some related examples, such as VP9, ​​eight directional modes corresponding to angles between 45 and 207 degrees are supported. In some related examples, such as AOMedia Video 1 (AV1), to utilize more diverse spatial redundancy in directional textures, directional intra modes are extended to angles set with finer granularity. The original eight angles are slightly modified and referred to as nominal angles, and these eight nominal angles are named V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED.

[0100] FIG. 8 illustrates exemplary nominal angles according to one embodiment of the present disclosure. Since each nominal angle may be associated with seven finer angles, there may be a total of 56 directional angles in some relevant examples, such as AV1. The prediction angle is expressed as a nominal intra angle and an angle delta, which is derived by multiplying a factor (ranging from -3 to 3) by a step size of 3 degrees. To implement a directional prediction mode in AV1 through a general method, all 56 directional intra prediction angles of AV1 may be implemented using a unified directional predictor that projects each pixel onto a reference subpixel location and interpolates the reference subpixel using a 2-tap bilinear filter.

[0101] In some related examples such as AV1, there are five non-directed smooth intra prediction modes: DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H. For DC prediction, the average of the left neighbor samples and the upper neighbor samples is used as the predictor for the block to be predicted. For PAETH prediction, reference samples from the top, left, and upper left are drawn first, and then the value closest to (top + left - upper left) is set as the predictor for the pixel to be predicted.

[0102] FIG. 9 illustrates the positions of the top, left, and top-left samples for a single pixel in the current block according to one embodiment of the present disclosure. For SMOOTH, SMOOTH_V, and SMOOTH_H modes, the block is predicted using quadratic interpolation in the vertical or horizontal direction or bidirectional averaging.

[0103] FIG. 10 illustrates an exemplary circulating filter intra mode according to one embodiment of the present disclosure.

[0104] To capture spatial correlations that attenuate with references on edges, the FILTER INTRA mode is designed for the luminance block. Five filter intra modes are defined in AV1, each represented by a set of eight 7-tap filters that reflect the correlation between a pixel in a 4x2 patch and its seven adjacent neighbors. For example, the weighting coefficients of the 7-tap filters vary by location. As shown in Fig. 10, the 8x8 block is divided into eight 4x2 patches, denoted as B0, B1, B2, B3, B4, B5, B6, and B7. For each patch, the seven neighbors denoted as R0 through R6 are used to predict the pixel of each patch. In the case of patch B0, all neighbors have already been reconstructed. However, for other patches, if not all neighbors have been reconstructed, the predicted value of the immediate neighbor is used as the reference value. For example, since not all neighbors of patch B7 are reconstructed, predicted samples of patch B7's neighbors (i.e., B5 and B6) are used instead.

[0105] For the chroma component, a chroma-specific intra-prediction mode called the chroma from luma (CfL) mode models chroma pixels as a linear function of matching reconstructed luma pixels. CfL prediction can be expressed as follows:

[0106] Equation (1)

[0107] Here L AC represents the AC contribution of the lumina component, and α represents the parameters of the linear model, and DCrepresents the DC contribution of the chroma component. In one example, the reconstructed luminar pixel is subsampled by the chroma resolution, and then the average value is subtracted to form the AC contribution. To approximate the chroma AC component from the AC contribution, instead of requiring the decoder to calculate scaling parameters as in some related examples, the CfL mode in AC1 calculates parameters based on the original chroma pixel. α It determines and signals it as a bitstream. This reduces decoder complexity and yields more accurate predictions. Regarding the DC contribution of chroma components, it is calculated using an intra-DC mode that has a sufficient and mature fast implementation for most chroma content.

[0108] III. Loop Filtering

[0109] In some related examples such as AV1, three in-loop filters may be applied to the reconstructed frame in the order of deblocking, constrained direction enhancement (CDEF), and loop restoration filter. The loop restoration filter includes a Wiener filter and a self-induced projection (SGRPRJ) filter, one of which may be adaptively selected.

[0110] Deblocking can be applied across transform block boundaries to reduce blocky artifacts caused by the quantization of transform coefficients. In some examples, 4-tap, 8-tap, and 14-tap finite impulse response (FIR) filters can be used for the luminance block, and 4-tap and 6-tap FIR filters can be used for the chroma block.

[0111] The filter length is initially determined by the minimum transform block size at the boundary. A conditional check using variance may be performed to prevent actual edges from being blurred. Additionally, a flatness check may be incorporated to finally determine the filter length.

[0112] CDEF is a non-linear de-ringing filter applied along the directional features detected in the target area. In some related examples, an 8x8 area is the unit size on which CDEF is performed. Normative direction detection can be performed as illustrated in Fig. 11, which shows some exemplary directions in the CDEF process.

[0113] In Fig. 11, a candidate direction d (0–7) that minimizes the following amount can be selected as the dominant direction.

[0114] Equation (2)

[0115] Here is the value of pixel p, and is a pixel of line k following direction d, and is obtained as follows It is the average value of.

[0116] Equation (3)

[0117] The sum of all sample values ​​within the block is a constant. Therefore, minimizing Equation (2) corresponds to maximizing the following equation:

[0118] Equation (4)

[0119] Once the dominant direction is determined, the following filter operations can be performed. A primary filter is performed along the selected dominant direction, while a secondary filter is performed along a secondary direction 45 degrees away from the dominant (or primary) direction.

[0120]

[0121] Here and ε₀ and ε₀ are the fixed filter coefficients for the first-order and second-order filters, respectively, and the piecewise linear function is given as follows:

[0122] Equation (6)

[0123] S and D represent the intensity and attenuation values, respectively, and up to 8 presets (S, D) for luminance / chroma are signaled per frame.

[0124] When applying filtering, each 64x64 block within the tile can select one of the presets, and filtering can be performed on each 8x8 unit within each 64x64 block.

[0125] Note that during the decoding process, various variables related to the signal characteristics of the local region may be parsed or derived from the bitstream. These variables include you , our , price , secStr , and steaming Includes d. Variable you represents the dominant edge direction of an 8x8 block. Variable our represents the variance of signal values ​​within an 8x8 block and is defined as the normalized difference between the cost in the dominant direction and the cost along the direction orthogonal to the dominant direction. Variable price The basic filter strength of a 64x64 block containing 8x8 filtering units Represents. Variable secStr The second filter strength of a 64x64 block containing 8x8 filtering units Represents. Variable steaming represents the attenuation parameter D of a 64x64 block containing an 8x8 filtering unit. These values ​​can be obtained individually for the luminance and chroma channels.

[0126] After the deblocking and CDEF processes are performed, in some related examples such as AV1, the mutually exclusive application of two types of restoration filters may be performed.

[0127] These two types of restoration filters include Wiener filters and SGRPRJ filters. The size of the square loop-restoration unit (LRU) can be selected from 64x64 to 256x256.

[0128] In a Wiener filter, the quality of each reconstructed pixel from a coded frame can be enhanced through non-causal filtering with neighboring pixels within a W×W window surrounding each pixel. The 2D filter tab of the Wiener filter is denoted by F and can be determined as follows:

[0129] Equation (7)

[0130] Here is the autocovariance of x, column vectorized in a W×W window Includes dog samples, is the cross-correlation of x with the original source sample y.

[0131] In some related examples, such as AV1, the separability of F and the symmetry and normalization of the filter coefficients of the Wiener filter may be imposed as constraints. Wiener filtering coefficient F( Formed as a vector) can be defined as follows:

[0132] Equation (8)

[0133] Here, a and b are and About , person These are vertical and horizontal filters. Coefficient vectors a and b can be retrieved from the encoder and encoded into a bitstream.

[0134] In SGRPRJ filtering, simple linear filtering described by the following model is performed, resulting in a degraded image as follows. Simple restore version Acquire:

[0135] (9)

[0136] Here, F and G can be obtained using a guide image and a degraded image. In some related examples, such as AV1, a type of self-guided filtering is adopted in which F and G are determined using only the statistics of the degraded image itself, without using a separate guide image.

[0137] More specifically, around the pixel Local average of pixels within the window ( ) and variance( ) can be calculated, and all pixel x can be filtered as follows:

[0138] Equation (10)

[0139] Here, r specifies the search window size, and e is a noise parameter that controls the de-noising strength.

[0140] Image degraded by Equation (9) Two simple reconstructions in and Given, the following subspace projection is performed to produce the final output Build,

[0141] Equation (11)

[0142] and using source Y, the encoder is as follows α and β can calculate:

[0143] Equation (12)

[0144] Here And, am.

[0145] Then, the encoder has a 6-tuple for each LRU ( Can transmit ).

[0146] IV. Reusing Feature Information for Enhanced Restoration Filtering

[0147] In some relevant examples, such as AV1, the Wiener filter can be performed in square units ranging from 64x64 to 256x256 by uniformly partitioning frames / tiles by LRU. In one example, the filter coefficients of the Wiener filter can be obtained by assuming that the signal statistics are stationary. Therefore, it is desirable to classify the filtering target region into one of the categorized statistical types where the stationarity assumption can be reasonably maintained. Possible approaches to classifying the target region include using quantities such as local variance or edge information. These quantities themselves or related categorical information can be computed in the decoder or signaled as a bitstream, but this can be costly in terms of computation or bitrate overhead.

[0148] In some related examples, such as AV1, the SGRPRJ filter can be performed by uniformly dividing frames / tiles into LRUs and operating in square units ranging from 64x64 to 256x256. In the SGRPRJ filter, a form of simple edge-preserving filtering is performed for each LRU to construct a simple reconstructed image using a fixed radius and noise parameter pair. Additionally, the fixed projection parameters α and βIt can be used for each LRU as a weighting factor for the error image to form the final reconstruction. However, regions in the error image may have different statistical characteristics reflecting local signal features such as edges and textures. Therefore, radius, noise parameter, α and β If a single set of SGRPRJ filter parameters, such as those mentioned above, is used or estimated across an LRU region containing pixels where signal statistics vary widely, the quality of the final reconstruction may be compromised. On the other hand, incorporating signal classification for greater adaptability may present essentially the same problems as Wiener filtering in terms of additional costs such as computational burden or bitrate overhead.

[0149] The present disclosure includes a method for improving the performance of a restoration filtering technique by reusing feature information, such as directional information derived from a CDEF process and / or an intra-prediction mode. For example, the adaptive restoration filtering technique can effectively reuse signal characteristics and statistical information already available in the decoder.

[0150] In the present disclosure, a restoration filter (or filtering) process may be defined as a filtering process used to compute a noisy image and estimate a clean original image based on the noisy image. The restoration filter process may include a process used to blur an image or an inverse process used for the inverse of blurring an image. Examples of a restoration filter process include, but are not limited to, Wiener and SGRPRJ filtering processes. A restoration filter (or filtering) unit is a region where the restoration filter process is performed.

[0151] In the present disclosure, a directional information unit may be defined as a group of pixels having a specified shape and size, and may provide the dominant direction of a feature represented by the pixel value of the pixel group. In an example such as AV1, each directional information unit of the CDEF process may be an 8x8 block. In each 8x8 block, the dominant direction and variance value of the pixels may be derived in a prescriptive manner. In another example, the directional intra-prediction mode of AV1 may provide this information in units of various shapes and sizes corresponding to the intra-prediction blocks.

[0152] According to aspects of the present disclosure, directional information derived from a decoder can be reused to infer the presence and directionality of boundary edges for a restoration filter, such as a Wiener filter or an SGRPRJ filter. For example, directional information can be derived from a CDEF process.

[0153] According to some embodiments, the shape and size of a restoration filtering unit (e.g., a Wiener or SGRPRJ filtering unit) may be defined using a number of available directional information units (e.g., an 8x8 block used for CDEF direction detection and filtering in AV1). In this way, finer granularity of directional adaptability can be realized. Thus, the restoration filtering process can be performed with unit sizes smaller than the fixed square types of 64x64, 128x128, or 256x256 used in some related examples such as AV1.

[0154] In one embodiment, the size of the restoration filtering unit may be the same as the LRU size defined in AV1, for example.

[0155] In one embodiment, the size of the restoration filtering unit may be the same as the size of the directional information unit (e.g., 8x8).

[0156] In one embodiment, the restoration filtering unit may be further divided into square, rectangular, T-shaped, or 4-way sub-LRUs in a given LRU size, similar to or aligned with the partition in some related examples, such as the partition in AV1 as shown in FIG. 12.

[0157] In one embodiment, each block having the size of a directional information unit (e.g., 8x8) may be merged by following various scanning sequences to form a filtering unit. FIG. 13 illustrates an example in which each block having the size of an 8x8 directional unit is merged into each filtering unit having a size of 32x8 in a raster scanning sequence. For example, four 8x8 directional unit blocks (1301)–(1304) may be merged into a 32x8 filtering unit block (1310), and four 8x8 directional unit blocks (1305)–(1308) may be merged into a 32x8 filtering unit block (1320).

[0158] In one embodiment, blocks having a size of directional information unit (e.g., 8x8) and similar directionality are merged by following various scanning orders to form a filtering unit. FIG. 14 illustrates an example in which blocks having a size of 8x8 directional units are merged into a variable-sized filtering unit in a raster scanning order. The sizes of the filtering units include 8x8, 16x8, and 32x8. For example, one 8x8 directional unit block (1401) may be an 8x8 filtering unit block (1410), two 8x8 directional unit blocks (1403)–(1404) may be merged into a 16x8 filtering unit block (1420), and four 8x8 directional unit blocks (1405)–(1408) may be merged into a 32x8 filtering unit block (1430).

[0159] According to some embodiments, each available orientation of the CDEF process can be directly used as a class index for a signal class, and a unique set of restoration filter shapes and sizes can be defined for it. That is, the selection of the restoration filter may depend on the available orientation of the CDEF process.

[0160] In one embodiment, the solution to the equation in the calculation of the restoration filter, such as Equation (7), can be applied to each signal class.

[0161] In one embodiment, 2D filters of different shapes having different numbers of filter tabs, symmetric or non-symmetric, may be used for each signal class.

[0162] In one embodiment, the divisibility (dividual or non-dividual filter) of the 2D filter may depend on the available directionality of the CDEF process.

[0163] In one embodiment, multiple of the available directions of the CDEF process may be merged into a single class, and as a result, the number of directional classes is reduced. For each merged class, a unique set of restoration filter shapes and sizes may be defined.

[0164] According to some embodiments, in addition to directionality, block distribution information is combined to further subdivide directionality-based classes.

[0165] In one embodiment, in the CDEF process, a directional information unit of size 8x8 can be further classified into different subclasses. The classification can be based on the variance value of the directional information unit. For example, if the number of classes based on directionality is 5 and the number of classes based on variance is 3, there may be 15 signal classes, and for each of them, a set of reconstruction filters (e.g., Wiener or SGRPRJ) can be designed.

[0166] According to some embodiments, in addition to directionality, filter strengths may be combined to determine the signal class of the reconstructed filtering unit. For example, signaled first and second CDEF filter strengths may be combined to determine the signal class of the reconstructed filtering unit. Different filter strength presets selected by the encoder may represent different signal characteristics of the target area.

[0167] In one embodiment, one of the presets of first and second filter strengths signaled in the bitstream can be directly used as another dimension of the signal class index. For example, if the number of classes based on directionality is 5 and the number of classes based on presets of filter strength is 4, a set of restoration filters (e.g., Wiener or SGRPRJ) can be designed for each of them.

[0168] According to some embodiments, a majority vote or consistency check on the directionality included in the restoration filtering unit may be performed to determine the filter class.

[0169] In one embodiment, the size and shape of the restoration filtering unit are fixed, and if the number of directional information units (e.g., 8x8 for CDEF) included in the restoration filtering unit is greater than or equal to a predefined number, a majority vote or consistency check on the directionality included in the restoration filtering unit may be performed to determine the filter class.

[0170] In one embodiment, in the case of a majority vote, the most frequent direction among the available and potential-to-merge directions may be selected. In one example, a specific margin may be set between the first and second most frequent directions.

[0171] In one embodiment, before taking a majority vote, it is determined whether the number of classes inside the restoration filtering unit is greater than a predefined number. If true, an inconsistency may be declared and explicit signaling for the restoration filtering unit or the use of a smaller restoration filtering unit may be selected.

[0172] According to some embodiments, the same method and direction information used for the restoration filtering unit in the luminance component may be used for the restoration filtering unit of the chroma component when such information is available only for the luminance component from the CDEF process.

[0173] According to some embodiments, the restoration filtering unit in the chroma component may use its own filter strength, such as preset values ​​of primary and secondary filter strengths included in the chroma component, when such information is available for the chroma component.

[0174] According to some embodiments, the chroma component restoration filtering unit may use its dispersion information when such information is available for the chroma component.

[0175] In one embodiment, when the CDEF process is turned off and the restoration filter is turned on, the direction search process of the CDEF process described in Section III (loop filtering section) is applied to derive the existence and directionality of the boundary edge for the restoration filtering unit.

[0176] In one embodiment, when the CDEF process is turned off and the restoration filter is turned on, a default signal class may be selected or explicit signaling of the filter class index may be performed.

[0177] According to aspects of the present disclosure, directional information indicated by intra prediction modes available in the decoder can be reused in a restoration filtering unit (e.g., Wiener or SGRPRJ filtering unit) as a guide for label signal classes, and can define a unique set of restoration filter shapes and sizes for each. In some embodiments, such directional information may be provided in various unit sizes by an encoder for intra prediction.

[0178] According to some embodiments, the shape and size of the restoration filtering unit can be defined using a number of available directional information units (e.g., 8x8 for the directional intra-prediction unit in AV1). In this way, finer granularity of directional adaptability can be realized. Thus, the restoration filtering process can be performed with a unit size smaller than one of the fixed square types of 64x64, 128x128, or 256x256 used in some related examples such as AV1.

[0179] In one embodiment, the size of the restoration filtering unit may be the same as the LRU size defined in AV1, for example.

[0180] In one embodiment, the size of the restoration filtering unit may be the same as the size of the directional information unit (e.g., 8x8).

[0181] In one embodiment, as shown in FIG. 12, the restoration filtering unit may be further divided into square, rectangular, T-shaped, or 4-way sub-LRUs in a given LRU size, similar to or aligned with the partitions in some related examples such as partitions in AVI.

[0182] In one embodiment, each block having the size of a directional information unit (e.g., 8x8) can be merged by following various scanning sequences, as shown in FIG. 13, to form a filtering unit.

[0183] In one embodiment, each block having a size of directional information unit (e.g., 8x8) and similar directionality can be merged by following various scanning sequences, as shown in FIG. 14, to form a filtering unit.

[0184] According to some embodiments, a fixed number of signal classes may be defined, and blocks of each signal class may have intra prediction modes having similar directionality. A unique set of restoration filter shapes and sizes may be defined for each signal class. When non-angular intra modes such as SMOOTH (including SMOOTH, SMOOTH_H, and SMOOTH_V modes), Paeth prediction, or DC mode are used, each non-angular intra mode or combination thereof may be associated with its own signal class.

[0185] In one embodiment, each of the eight nominal angles, along with the seven possible delta angles associated in AV1, can be grouped together to form a total of eight directional classes.

[0186] In one embodiment, the directional class may depend on both the nominal angle and the delta angle associated with the nominal angle.

[0187] According to some embodiments, if the restoration filtering unit region is not predicted using a directional intra mode or an intra prediction mode, the signal class of the restoration filtering unit may be determined based on the neighbor blocks of the restoration filtering unit. A default signal class may be selected based on whether the neighbor blocks are coded in a directional intra mode or whether the directionality of the neighbor blocks matches, or explicit signaling of a signal class index or filter coefficients may be performed. For example, if there are no neighbor blocks coded in a directional intra mode or the directionality of the neighbor blocks does not match, a default signal class may be selected or explicit signaling of a signal class index or filter coefficients may be performed.

[0188] According to some embodiments, a majority vote or consistency check on the directionality included in the restoration filtering unit may be performed to determine the filter class.

[0189] In one embodiment, the size and shape of the restoration filtering unit are fixed, and when the number of directional information units (e.g., 8x8 for CDEF) included in the restoration filtering unit is greater than a predefined number, a majority vote or consistency check on the directionality included in the restoration filtering unit may be performed to determine the filter class. The directional information units may have various sizes and various directional intra-prediction modes.

[0190] In one embodiment, for the case of a majority vote, the most frequent direction among the available and possible merged directions may be selected. For example, a specific margin may be set between the first and second most frequent directions.

[0191] In one embodiment, in the case of a majority vote, the most frequent direction among the available and potential-to-merge directions may be selected. In one example, a specific margin may be set between the first and second most frequent directions.

[0192] In one embodiment, before taking a majority vote, it is determined whether the number of classes inside the restoration filtering unit is greater than a predefined number. If true, inconsistency may be declared and explicit signaling for the restoration filtering unit or the use of a smaller restoration filtering unit may be selected.

[0193] According to some embodiments, the restoration filtering unit in the chroma component may use such information when its directional information from the directional intra-prediction mode is separately available for the chroma component.

[0194] According to aspects of the present disclosure, where directional information is available from both the intra prediction mode and the CDEF process, the directional mode of the intra prediction mode, together with the directional information from the CDEF process, can be used as a guide for identifying and classifying restoration filtering units (e.g., Wiener or SGRPRJ filtering units).

[0195] In one embodiment, when directional information is available from both the intra-predicted direction and the CDEF process, a mapping of directions from both sources may be introduced when checking the consistency of the directional information. The classification-based restoration filtering process may be performed only when the directional information matches. FIG. 15 illustrates an example in which, during the DEF process, an intra-predicted direction corresponding to seven delta angles associated with one of the eight basic angles of AV1 may be mapped to a single directional class, thereby forming a one-to-one correspondence with the eight derived directions.

[0196] In one embodiment, directional information from either the intra-prediction mode or the CDEF process may be used as a first source. The other of the intra-prediction mode or the CDEF process may be used only when a consistent direction cannot be determined using the directional information from the first source.

[0197] Note that Wiener filters and SGRPRJ filters can be adaptively selected on an LRU basis in some examples, such as AV1. The determination of filter parameters and the filtering process can be performed on an LRU basis for both filters in the same manner. The difference lies in the type and number of filter parameters.

[0198] In one embodiment, when an SGRPRJ filter is performed, the search window size r and the noise parameter e can be defined and used per signal class in the same way as the Wiener filter parameters.

[0199] In one embodiment, when the SGRPRJ filter is performed, the projection parameter α and β It can be defined and used per signal class in the same way as Wiener filter parameters.

[0200] In one embodiment, when an SGRPRJ filter is performed, the search window size r, the noise parameter e, and the projection parameter α and β It can be defined and used per signal class in the same way as Wiener filter parameters.

[0201] V. Flowchart

[0202] FIG. 16 illustrates a flowchart schematically showing an exemplary process (1600) according to one embodiment of the present disclosure. In various embodiments, the process (1600) is executed by a processing circuit, such as a processing circuit of a terminal device (210, 220, 230, 240), a processing circuit performing the function of a video encoder (303), a processing circuit performing the function of a video decoder (310), a processing circuit performing the function of a video decoder (410), a processing circuit performing the function of an intra prediction module (452), a processing circuit performing the function of a video encoder (503), a processing circuit performing the function of a predictor (535), a processing circuit performing the function of an intra encoder (622), a processing circuit performing the function of an intra decoder (772), etc. In some embodiments, the process (1600) is implemented as a software instruction, and thus when the processing circuit executes the software instruction, the processing circuit performs the process (1600).

[0203] The process (1600) can generally start at step (S1610), where the process (1600) determines the directional information of the restoration filter unit included in the video frame based on at least one of the CDEF process or the intra prediction mode. Then, the process (1600) proceeds to step (S1620).

[0204] In step (S1620), the process (1600) determines one of a plurality of filter parameter sets of the restoration filter process based on the directional information of the restoration filter unit. Then, the process (1600) proceeds to step (S1630).

[0205] In step (S1630), the process (1600) performs a restoration filter process on the restoration filter unit based on one of a plurality of filter parameter sets. Then, the process (1600) proceeds to step (S1640).

[0206] In step (S1640), the process (1600) restores the video frame based on the filtered restoration filter unit. Then, the process (1600) terminates.

[0207] In one embodiment, the restoration filter unit includes one or more directional information units, and at least one of the CDEF process or intra-prediction mode is performed on one of the one or more directional information units.

[0208] In one embodiment, each of the plurality of filter parameter sets of the restoration filter is associated with at least one directionality of the CDEF process.

[0209] In one embodiment, the process (1600) determines one of a plurality of filter parameter sets of a restoration filter process based on directional information and block distribution information of a restoration filter unit.

[0210] In one embodiment, the process (1600) determines one of a plurality of filter parameter sets of the restoration filter process based on the directional information of the restoration filter unit and the filter strength of the CDEF process.

[0211] In one embodiment, the process (1600) determines the directionality information of the restoration filter unit based on at least one of a majority vote or a consistency check regarding the directionality in the restoration filter unit.

[0212] In one embodiment, the process (1600) determines the direction information of the restoration filter unit based on an intra prediction mode performed on the neighbor blocks of the restoration filter unit and based on the restoration filter unit that is not intra-coded and the neighbor blocks that are intra-coded.

[0213] In one embodiment, the process (1600) performs a restoration filter process on a restoration filter unit based on directional information determined from a CDEF process that matches directional information determined from an intra-prediction mode.

[0214] In one embodiment, the restoration filter process is one of the Wiener filter process and the SGRPRJ filter process.

[0215] In one embodiment, the process (1600) determines one of a plurality of filter parameter sets of a restoration filter process based on a default filter parameter set, a signaled index indicating a filter parameter set, and one of the signaled filter parameter sets.

[0216] VIII. Computer Systems

[0217] The aforementioned technology may be implemented as computer software that uses computer-readable instructions and can be physically stored on one or more computer-readable media. For example, FIG. 17 illustrates a computer system (1700) suitable for implementing a specific embodiment of the disclosed invention.

[0218] Computer software may be coded using any suitable machine code or computer language capable of generating code containing instructions that can be executed directly, or through interpretation, micro-code execution, etc., by a computer central processing unit (CPU), graphics processing unit (GPU), etc., via assembly, compilation, linking, or similar mechanisms.

[0219] The instruction can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0220] The components of the computer system (1700) illustrated in FIG. 17 are essentially exemplary and are not intended to imply any limitation to the scope of use or function of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be interpreted as having any dependency or requirement related to any one or combination of the components shown in the exemplary embodiments of the computer system (1700).

[0221] The computer system (1700) may include a specific human interface input device. This human interface input device may respond to input by one or more human users, for example, tactile input (e.g., keystroke, swip, data glove movement), audio input (e.g., voice, clap), visual input (e.g., gesture), and olfactory input (not shown). The human interface device may also be used to capture specific media that are not necessarily directly related to conscious input by a person, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, pictures acquired from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0222] The input human interface device may include one or more of a keyboard (1701), a mouse (1702), a trackpad (1703), a touchscreen (1710), a data glove (not shown), a joystick (1705), a microphone (1706), a scanner (1707), and a camera (1708) (each only one is shown).

[0223] The computer system (1700) may include a specific human interface output device. Such human interface output device may stimulate the senses of one or more human users, for example, through tactile output, sound, light and smell / taste. These human interface output devices may include haptic output devices (e.g., touch screen (1710), data gloves (not shown), or haptic feedback via a joystick (1705), but there may also be haptic feedback devices that do not serve as input devices), audio output devices (e.g., speakers (1709), headphones (not shown)), visual output devices (e.g., screens (1710) including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touch screen input functions and each with or without haptic feedback functions—some of which may produce two-dimensional visual output or three-dimensional or higher output through means such as stereographic output, virtual-reality glasses (not shown), holographic display and smoke tank (not shown)—), and printers (not shown). These visual output devices (e.g., screen (1710)) can be connected to the system bus (1748) via a graphics adapter (1750).

[0224] The computer system (1700) may also include human-accessible storage devices and associated media, such as optical media including a CD / DVD ROM RW (1720) having a CD / DVD media (1721), a thumb drive (1722), a removable hard drive or solid-state drive (1723), legacy magnetic media such as tape and floppy disk (not shown), and specialized ROM / ASIC / PLD-based devices such as a security dongle (not shown).

[0225] Those skilled in the art should also understand that the term “computer-readable medium” as used in connection with the subject matter disclosed herein does not include a transmitting medium, a carrier wave, or other transient signals.

[0226] The computer system (1700) may also include an interface to one or more communication networks (1755). One or more communication networks (1755) may be, for example, wireless, wired, or optical networks. One or more communication networks (1755) may also be local, wide-area, metropolitan, automotive and industrial, real-time, latency-tolerant, etc. Examples of one or more communication networks (1755) include cellular networks including Ethernet, wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, automotive and industrial including CANBus, etc. A specific network generally requires an external network interface adapter attached to a specific general-purpose data port or peripheral bus (1749) (e.g., a USB port of the computer system (1700)); Others are generally integrated into the core of the computer system (1700) by attaching to the system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1700) can communicate with other networks. This communication may be unidirectional, receive-only (e.g., TV broadcasting), unidirectional transmit-only (e.g., from a CANbus to a specific CANbus device), or bidirectional (e.g., using a local or wide-area digital network to another computer system). Specific protocols and protocol stacks may be used for each of the networks and network interfaces as described above.

[0227] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core (1740) of the computer system (1700).

[0228] The core (1740) may include one or more central processing units (CPUs) (1741), graphics processing units (GPUs) (1742), specialized programmable processing units in the form of a field programmable gate area (FPGA) (1743), hardware accelerators (1744) for specific tasks, graphics adapters (1750), etc. Along with read-only memory (ROM) (1745), random access memory (1746), and internal mass storage devices (1747) such as internal hard drives or SSDs that are not accessible to the user, these devices may be connected via a system bus (1748). In some computer systems, the system bus (1748) may be accessible in the form of one or more physical plugs that enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (1748) or via a peripheral bus (1749). For example, the screen (1710) can be connected to a graphics adapter (1750). Architectures for peripheral buses include PCI, USB, etc.

[0229] The CPU (1741), GPU (1742), FPGA (1743), and accelerator (1744) can be combined to execute specific instructions that can construct the aforementioned computer code. The computer code may be stored in ROM (1745) or RAM (1746). Transitional data may also be stored in RAM (1746), while persistent data may be stored, for example, in an internal mass storage device (1747). Fast storage and retrieval of any of the memory elements may be made possible through the use of a cache memory that may be closely associated with one or more CPUs (1741), GPUs (1742), mass storage devices (1747), ROM (1745), RAM (1746), etc.

[0230] A computer-readable medium may have computer code for performing various computer-implemented operations. The medium and the computer code may be specially designed and constructed for the purposes of this disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0231] As an example, not limited to, the architecture (1700), specifically the core (1740), may provide functionality as a result of processor(s) (including CPU, GPU, FPGA, accelerator, etc.) executing software implemented on one or more types of computer-readable media. Such computer-readable media may be media associated with mass storage devices accessible to users as described above, as well as specific storage devices of the core (1740) of a non-transient nature, such as internal mass storage devices (1747) or ROM (1745). Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core (1740). The computer-readable media may include one or more memory elements or chips as needed. Software may enable the core (1740) and, in particular, the internal processor (including a CPU, GPU, FPGA, etc.) to execute a specific process or a specific part of a specific process described herein, including defining data structures stored in RAM (1746) and modifying such data structures according to a process defined by the software. Additionally or alternatively, the computer system may provide a function that is otherwise implemented in a circuit (e.g., accelerator (1744)) as a result of logic hardwired, which can operate instead of or with the software to execute a specific process or a specific part of a specific process described herein. References to software may include logic, and where appropriate, vice versa. References to computer-readable media may include circuits that store software for execution (e.g., integrated circuits (ICs)), circuits that implement logic for execution, or both, where appropriate. The present disclosure includes any suitable combination of hardware and software.

[0232] Although the present disclosure describes some exemplary embodiments, there are modifications, permutations, and various alternative equivalents that fall within the scope of the present disclosure.

[0233] Appendix A: Abbreviations

[0234] ALF: Adaptive Loop Filter

[0235] AMVP: Advanced Motion Vector Prediction

[0236] APS: Adaptation Parameter Set

[0237] ASIC: Application-Specific Integrated Circuit

[0238] ATMVP: Alternative / Advanced Temporal Motion Vector Prediction

[0239] AV1: AOMedia Video 1

[0240] AV2: AOMedia Video 2

[0241] BMS: Benchmark Set

[0242] BV: Block Vector

[0243] CANBus: Controller Area Network Bus

[0244] CB: Coding Block

[0245] CC-ALF: Cross-Component Adaptive Loop Filter

[0246] CD: Compact Disc

[0247] CDEF: Constrained Directional Enhancement Filter

[0248] CPR: Current Picture Referencing

[0249] CPU: Central Processing Unit

[0250] CRT: Cathode Ray Tube

[0251] CTB: Coding Tree Block

[0252] CTU: Coding Tree Unit

[0253] CU: Coding Unit

[0254] DPB: Decoder Picture Buffer

[0255] DPCM: Differential Pulse-Code Modulation

[0256] DPS: Decoding Parameter Set

[0257] DVD: Digital Video Disc

[0258] FPGA: Field Programmable Gate Area

[0259] JCCR: Joint CbCr Residual Coding

[0260] JVET: Joint Video Exploration Team

[0261] GOP: Groups of Pictures

[0262] GPU: Graphics Processing Unit

[0263] GSM: Global System for Mobile communications

[0264] HDR: High Dynamic Range

[0265] HEVC: High Efficiency Video Coding

[0266] HRD: Hypothetical Reference Decoder

[0267] IBC: Intra Block Copy

[0268] IC: Integrated Circuit

[0269] ISP: Intra Sub-Partitions

[0270] JEM: Joint Exploration Model

[0271] LAN: Local Area Network

[0272] LCD: Liquid-Crystal Display

[0273] LR: Loop Restoration Filter

[0274] LRU: Loop Restoration Unit

[0275] LTE: Long-Term Evolution

[0276] MPM: Most Probable Mode

[0277] MV: Motion Vector

[0278] OLED: Organic Light-Emitting Diode

[0279] PBs: Prediction Blocks

[0280] PCI: Peripheral Component Interconnect

[0281] PDPC: Position Dependent Prediction Combination

[0282] PLD: Programmable Logic Device

[0283] PPS: Picture Parameter Set

[0284] PU: Prediction Unit

[0285] RAM: Random Access Memory

[0286] ROM: Read-Only Memory

[0287] SAO: Sample Adaptive Offset

[0288] SCC: Screen Content Coding

[0289] SDR: Standard Dynamic Range

[0290] SEI: Supplementary Enhancement Information

[0291] SNR: Signal Noise Ratio

[0292] SPS: Sequence Parameter Set

[0293] SSD: Solid-state Drive

[0294] TU: Transform Unit

[0295] USB: Universal Serial Bus

[0296] VPS: Video Parameter Set

[0297] VUI: Video Usability Information

[0298] VVC: Versatile Video Coding

[0299] WAIP: Wide-Angle Intra Prediction

Claims

Claim 1 A video decoding method in a decoder comprises: a step (1610) of determining directional information of a restoration filter unit included in a video frame based on the directionality of each of a plurality of pixel groups of a video frame, determined by at least one of a previously performed constrained directional enhancement filter (CDEF) process or a previously performed intra-predictive coding process, wherein the restoration filter unit is formed by merging the plurality of pixel groups according to the scanning order by comparing the directionality of the pixels; a step (1620) of determining one of a plurality of filter parameter sets of a restoration filter process based on the directional information of the restoration filter unit; and a step (1630) of performing one of a Wiener filter process or a self-guided projection filter process as the restoration filter process for the restoration filter unit based on one of the plurality of filter parameter sets, after at least one of the previously performed CDEF process or the previously performed intra-predictive coding process. A video decoding method comprising the step (1640) of reconstructing the video frame based on a filtered restoration filter unit, wherein the restoration filter unit comprises one or more directional information units, and at least one of the previously performed CDEF process or the previously performed intra-predictive coding process is performed on one of the one or more directional information units, and the step of determining the directional information of the restoration filter unit comprises: determining whether the number of directional classes within the restoration filter unit is greater than a predefined number; and determining the directional information of the restoration filter unit by using an explicitly signaled restoration filter unit or a smaller restoration filter unit in response to the determination that the number of directional classes within the restoration filter unit is greater than the predefined number. Claim 2 delete Claim 3 A video decoding method according to claim 1, wherein each of the plurality of filter parameter sets of the restoration filter process is associated with at least one directionality of the previously performed CDEF process. Claim 4 A video decoding method according to claim 1, wherein the step of determining one of the plurality of filter parameter sets includes the step of determining one of the plurality of filter parameter sets of the restoration filter process based on directional information and block distribution information of the restoration filter unit. Claim 5 A video decoding method according to claim 1, wherein the step of determining one of the plurality of filter parameter sets comprises the step of determining one of the plurality of filter parameter sets of the restoration filter process based on the directional information of the restoration filter unit and the filter strength of the previously performed CDEF process. Claim 6 A video decoding method according to claim 1, wherein the step of determining directional information of the restoration filter unit comprises the step of determining directional information of the restoration filter unit based on at least one of a majority vote or a consistency check regarding the directionality in the restoration filter unit. Claim 7 A video decoding method according to claim 1, wherein the restoration filter unit is not intra-coded, and in response to the neighbor block of the restoration filter unit being intra-coded, the step of determining the directional information of the restoration filter unit includes the step of determining the directional information of the restoration filter unit based on intra-predictive coding previously performed on the neighbor block of the restoration filter unit. Claim 8 A video decoding method according to claim 1, wherein the step of performing one of the Wiener filter process or the self-induced projection filter process comprises the step of performing one of the Wiener filter process or the self-induced projection filter process for the restoration filter unit based on directional information determined in the previously performed CDEF process that matches directional information determined from the previously performed intra-predictive coding process. Claim 9 delete Claim 10 A video decoding method according to claim 1, wherein the step of determining one of the plurality of filter parameter sets comprises the step of determining one of the plurality of filter parameter sets of the restoration filter process based on one of (i) a default filter parameter set, (ii) a signaled index indicating a filter parameter set, or (iii) a signaled filter parameter set. Claim 11 An apparatus comprising a processing circuit configured to perform a method according to any one of claims 1, 3 through 8 and 10. Claim 12 A computer-readable non-transient storage medium for storing instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to perform a method according to any one of claims 1, 3 through 8 and 10. Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images