Video filtering method and device

Abstract

Aspects of the present application provide methods and apparatus for filtering in video encoding / decoding. In some examples, a video filtering apparatus includes processing circuitry. The processing circuitry determines a first offset value for applying a non-linear mapping-based filter based on a first reconstructed sample at a first node along the loop filter chain. The processing circuitry then applies the first offset value to an intermediate reconstructed sample at a second node along the loop filter chain to generate a second reconstructed sample.

Description

Incorporation by Reference

[0001] This application claims priority to U.S. Patent Application No. 17 / 449,126, filed on September 28, 2021, “METHOD AND APPARATUS FOR VIDEO FILTERING,” which claims priority to U.S. Provisional Application No. 63 / 164,478, filed on March 22, 2021, “FLEXIBLE FILTER LOCATION FOR SAMPLE OFFSET.” The entire disclosure of the prior application is incorporated herein by reference. Technical Field

[0002] Embodiments described herein generally relate to video encoding and decoding. Background Art

[0003] The background description provided herein is intended to present the background of the present application. The work of the named inventors, the work described in this background section, and the contents within the scope of the embodiments of this specification may not be considered as prior art at the time of filing, and are not explicitly or implicitly admitted as prior art against the present application.

[0004] Video encoding and decoding can be performed using inter-image prediction with motion compensation. Uncompressed digital video may include a series of images, each image having a certain spatial dimension, such as 1920x1080 luminance samples and associated chrominance samples. The image sequence may have a fixed or variable image rate (commonly known as frame rate), for example, 60 images per second or 60Hz. Uncompressed video requires a specific bit rate. For example, a 1080p604:2:0 video (1920x1080 luminance sample resolution at 60Hz frame rate) with 8 bits per sample requires a bandwidth of nearly 1.5G bits / second. One hour of such video requires more than 600G bytes of storage space.

[0005] One purpose of video encoding and decoding is to reduce the redundancy of the input video signal through compression. In some cases, compression can reduce the bandwidth and / or memory requirements by at least two orders of magnitude. Lossless compression, lossy compression, or a combination thereof can be used. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be consistent with the original signal, but the distortion between the original signal and the reconstructed signal is small enough to enable the reconstructed signal to achieve the desired purpose. Lossy compression is widely used in the video field. The amount of distortion allowed depends on the application. For example, users of some consumer live broadcast applications can tolerate greater distortion than users of TV program applications. The achievable compression ratio can reflect that the greater the allowable / tolerable distortion, the higher the compression ratio that can be produced.

[0006] Video encoders and decoders may utilize several broad categories of techniques including, for example, motion compensation, transforms, quantization, and entropy coding.

[0007] Video coding techniques may include techniques known as intra-frame coding. In intra-frame coding, the representation of sample values ​​does not require reference to samples or other data in a previously reconstructed reference image. In some video codecs, an image is spatially subdivided into blocks of samples. When all sample blocks are encoded in intra-frame mode, the image may be an intra-frame image. Intra-frame images and their derivatives (such as independent decoder refresh images) can be used to reset the decoder state and can thus be used as the first image in an encoded video stream and video session, or as a still image. Samples of intra-frame blocks may be transformed, and the transform coefficients may be quantized before entropy coding. Intra-frame prediction may be a technique for minimizing sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after conversion, the smaller the AC coefficient, and the fewer bits required to represent the block with a given quantization step after entropy coding.

[0008] Conventional intra-frame coding (e.g., as known from, for example, MPEG-2 generation coding techniques) does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to, for example, obtain surrounding sample data and / or metadata during the encoding / decoding of spatially adjacent and preceding block data in decoding order. Such techniques are henceforth referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses reference data only from the current image being reconstructed (rather than a reference image).

[0009] There are many different forms of intra-frame prediction. When more than one such technique may be used in a given video coding technique, the technique used may be encoded in the intra-frame prediction mode. In some cases, a mode may have sub-modes and / or parameters, which may be encoded separately or included in the mode codeword. Which codeword is used for a given mode / sub-mode / parameter combination can have an impact on the coding efficiency gain through intra-frame prediction, and so can the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain intra prediction modes were introduced with H.264, refined in H.265, and further refined in new coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Neighboring sample values ​​can be used to form a prediction block, which belongs to samples that are already available. Sample values ​​of neighboring samples are copied to the prediction block according to the direction. The information of the used direction can be encoded in the bitstream, or it can be predicted by itself.

[0011] See also Figure 1A , a subset of nine known prediction directions of the 33 possible prediction directions for H.265 (corresponding to the 33 angular modes for the 35 intra modes) is depicted at the bottom right. The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction of the sample being predicted. For example, arrow (102) indicates that the prediction direction for sample (101) is from one or more samples to the upper right corner, at an angle of 45 degrees to the horizontal. Similarly, arrow (103) indicates that the prediction direction for sample (101) is from one or more samples to the lower left of sample (101), at an angle of 22.5 degrees to the horizontal.

[0012] Still reference Figure 1A , a square block (104) of 4×4 samples is shown in the upper left (indicated by bold dashed lines). The square block (104) includes 16 samples, each sample is labeled with "S" to indicate its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (starting from the top) and the first sample in the X dimension (starting from the left). Similarly, sample S44 in block (104) is the fourth sample in both the Y and X dimensions. Since the size of the block is 4×4 samples, S44 is located in the lower right corner. Reference samples following a similar numbering scheme are also shown. The reference samples are labeled with R, and their Y position (e.g., row index) and X position (column index) relative to the block (104). In H.264 and H.265, the prediction samples are adjacent to the block being reconstructed; therefore, there is no need to use negative values.

[0013] Intra-frame prediction works by copying reference sample values ​​from neighboring samples covered by a signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating that the prediction direction for the block is consistent with arrow (102) - that is, samples are predicted from one or more prediction samples to the upper right corner, at a 45 degree angle to the horizontal plane. In this case, samples S41, S32, S23, and S14 are predicted using the same reference sample R05. Sample S44 is then predicted using reference sample R08.

[0014] In some cases, the values ​​of multiple reference samples may be combined, for example by interpolation, to calculate the reference sample; in particular, when the direction is not divisible by 45 degrees.

[0015] As video coding technology has evolved, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This number was increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of publication. Some experiments have been conducted to identify the most likely directions, and use certain techniques in entropy coding to represent those possible directions with a small number of bits, while bearing the disadvantages of less likely directions. In addition, these directions themselves can sometimes be predicted from adjacent directions used by adjacent decoded blocks.

[0016] Figure 1B A schematic diagram (180) of the 65 intra prediction directions of JEM is shown to illustrate the increasing number of prediction directions over time.

[0017] The mapping method of the intra prediction direction bits representing the direction in the coded video bitstream may vary between different video coding techniques; it may range, for example, from a simple direct mapping of the prediction direction to the intra prediction mode or to the codeword, to complex adaptive schemes involving most possible modes, and similar techniques. However, in all cases, there may be some directions that are statistically less likely to occur in the video content than other directions. Since the goal of video compression is to reduce redundancy, in a well-functioning video coding technique, those less likely directions will be represented by more bits than more likely directions.

[0018] Motion compensation may be a lossy compression technique and may involve a technique where blocks of sample data of a previously reconstructed image or portion thereof (reference image) are used to predict a newly reconstructed image or image portion after being spatially shifted in a direction indicated by a motion vector (hereinafter MV). In some cases, the reference image may be the same as the image currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, the third dimension being an indication of the reference image in use (the latter may indirectly be a temporal dimension).

[0019] In some video compression techniques, the MV applicable to a certain area of ​​sample data can be predicted using other MVs, such as those MVs that are related to sample data of another area that is spatially adjacent to the area currently being reconstructed and that are decoded before the MV. Doing so can substantially reduce the amount of data required to encode the MV, thereby eliminating redundancy and increasing compression strength. MV prediction can work effectively, for example, because when encoding an input video signal derived from a camera (called natural video), there is a statistical likelihood that a larger area than the area to which a single MV can be applied moves in a similar direction, and therefore, in some cases, similar motion vectors derived from MVs of adjacent areas can be used for prediction. This results in the MV found for a given area being similar or identical to the MV predicted from the surrounding MVs, and after entropy coding and decoding, it can be represented with fewer bits than if the MV is directly encoded. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, MV prediction itself may be lossy, for example, due to rounding errors generated when calculating predicted values ​​from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T H.265 Recommendation, "High Efficiency Video Coding", December 2016). Among the various MV prediction mechanisms provided by H.265, this application describes a technique referred to below as "spatial merging".

[0021] Please refer to Figure 2, the current block (201) includes samples that have been discovered by the encoder during the motion search process, and the samples can be predicted based on the previous block of the same size that has generated a spatial offset. In addition, the MV can be derived from metadata associated with one or more reference pictures instead of encoding the MV directly. For example, the MV is derived from metadata of the nearest reference picture (in decoding order) using the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 202 to 206 respectively). In H.265, MV prediction can use the prediction value of the same reference picture that the neighboring blocks are also using. Summary of the invention

[0022] Various aspects of the present application provide methods and apparatus for filtering in video encoding / decoding. In some examples, an apparatus for video filtering includes a processing circuit. The processing circuit determines a first offset value for applying a filter based on a nonlinear mapping based on a first reconstructed sample at a first node along a loop filter chain. The processing circuit then applies the first offset value to an intermediate reconstructed sample at a second node along the loop filter chain to generate a second reconstructed sample.

[0023] In some examples, the filter based on the non-linear mapping is a cross-component sample offset (CCSO) filter, and the intermediate reconstructed samples and the first reconstructed samples have different color components.

[0024] In some examples, the filter based on the non-linear mapping is a local sample offset (LSO) filter, and the intermediate reconstructed samples and the first reconstructed samples have the same color components.

[0025] In some embodiments, the first node and the second node have the same node, such as an input node of a loop filter chain, an output node of a loop filter chain, or an intermediate node of a loop filter chain. In one example, the first reconstructed sample is generated by a processing module before a deblocking filter. In another example, the first reconstructed sample is generated by a deblocking filter. In another example, the first reconstructed sample is generated by a restricted directional enhancement filter. In another example, the first reconstructed sample is generated by a loop recovery filter.

[0026] In some examples, the first node and the second node are different nodes. In one example, the first reconstructed samples are generated by a processing module before a deblocking filter, and the intermediate reconstructed samples are generated by at least one of a deblocking filter, a constrained directional enhancement filter, or a loop restoration filter.

[0027] In some examples, the first reconstructed samples are generated by a deblocking filter and the intermediate reconstructed samples are generated by at least one of a constrained directional enhancement filter or a loop restoration filter.

[0028] In some examples, the first reconstructed samples are generated by a constrained directional enhancement filter and the intermediate reconstructed samples are generated by a loop restoration filter.

[0029] Aspects of the present application also provide a non-transitory computer-readable medium storing instructions, which, when executed by a computer, cause the computer to perform any of the methods for video encoding / decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] More features, properties and various advantages of the subject matter of the present application will become more apparent from the following detailed description and accompanying drawings, in which:

[0031] Figure 1A is a schematic diagram of an exemplary subset of intra prediction modes.

[0032] Figure 1B is a schematic diagram of exemplary intra prediction directions.

[0033] Figure 2 Schematic diagram of the current block and its surrounding spatial merging candidates in an example.

[0034] Figure 3 is a schematic diagram of a simplified block diagram of a communication system (300) of one embodiment.

[0035] Figure 4 is a schematic diagram of a simplified block diagram of a communication system (400) of one embodiment.

[0036] Figure 5 is a schematic diagram of a simplified block diagram of a decoder of an embodiment.

[0037] Figure 6 is a schematic diagram of a simplified block diagram of an encoder of an embodiment.

[0038] Figure 7 A block diagram of an encoder of another embodiment is shown.

[0039] Figure 8 A block diagram of a decoder of another embodiment is shown.

[0040] Fig. 9 An example of the filter shape of an embodiment of the present application is shown.

[0041] FIG. 10A to FIG. 10D An example of sub-sampling positions for calculating gradients according to an embodiment of the present application is shown.

[0042] FIG. 11A to FIG. 11B An example of a virtual boundary filtering process according to an embodiment of the present application is shown.

[0043] FIG. 12A to FIG. 12FAn example of a symmetric filling operation at a virtual boundary according to an embodiment of the present application is shown.

[0044] Fig.13 An example of partitioning of an image according to some embodiments of the present application is shown.

[0045] Fig.14 The quadtree segmentation patterns of some example images are shown.

[0046] Fig.15 The cross component filter of the embodiment of the present application is shown.

[0047] Fig.16 An example of the filter shape of an embodiment of the present application is shown.

[0048] Fig.17 An example of the syntax of the cross-component filter of some embodiments of the present application is shown.

[0049] FIG. 18A to FIG. 18B An example of the relative position of chroma samples with respect to luma samples of embodiments of the present application is shown.

[0050] Fig.19 An example of direction search according to an embodiment of the present application is shown.

[0051] Fig. 20 Examples of subspace projections in some examples are shown.

[0052] Fig.21 A table showing multiple sample adaptive offset (SAO) types according to an embodiment of the present application is shown.

[0053] Fig. 22 Examples of patterns in edge offsets for pixel classification in some examples are shown.

[0054] Fig.23 A table showing pixel classification rules for edge migration in some examples.

[0055] Fig.24 An example of syntax that can be represented by a signal is shown.

[0056] Fig.25 Examples of filter support areas of some embodiments of the present application are shown.

[0057] Fig.26 An example of another filter support region of some embodiments of the present application is shown.

[0058] FIG. 27A to FIG. 27C A table with 81 combinations of embodiments of the present application is shown.

[0059] Fig.28Seven filter shape configurations for three filter taps in one example are shown.

[0060] Fig.29 A block diagram of a loop filter chain in some examples is shown.

[0061] FIG. 30A to FIG. 30D Example diagrams of loop filter chains are shown, including filters based on non-linear mapping at different positions of the loop filter chains.

[0062] FIG. 31A to FIG. 31B Example diagrams of loop filter chains are shown, including filters based on non-linear mapping at different positions of the loop filter chains.

[0063] FIG. 32A to FIG. 32C Example diagrams of loop filter chains are shown, including filters based on non-linear mapping at different positions of the loop filter chains.

[0064] Fig.33 Another example of a loop filter chain that in some examples includes filters based on non-linear mapping is shown.

[0065] Fig.34 is an example of a loop filter chain that includes multiple non-linear mapping based filters in some examples.

[0066] Fig.35 is another example of a loop filter chain that in some examples includes multiple non-linear mapping based filters.

[0067] Fig.36 A flowchart showing an overview of the process of an embodiment of the present application is shown.

[0068] Fig.37 is a schematic diagram of a computer system of an embodiment. DETAILED DESCRIPTION

[0069] Figure 3 3 is a simplified block diagram of a communication system (300) according to an embodiment disclosed in the present application. The communication system (300) includes a plurality of terminal devices, which can communicate with each other through, for example, a network (350). For example, the communication system (300) includes a first terminal device (310) and a second terminal device (320) interconnected through the network (350). Figure 3In an embodiment of the present invention, the first terminal device (310) and the second terminal device (320) perform unidirectional data transmission. For example, the first terminal device (310) may encode video data (e.g., a video image stream captured by the terminal device (310)) for transmission to another second terminal device (320) via a network (350). The encoded video data is transmitted in the form of one or more encoded video code streams. The second terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to restore the video data, and display the video image based on the restored video data. Unidirectional data transmission is more common in applications such as media services.

[0070] In another embodiment, the communication system (300) includes a third terminal device (330) and a fourth terminal device (340) that perform bidirectional transmission of encoded video data, which can occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (330) and the fourth terminal device (340) can encode video data (e.g., a video image stream collected by the terminal device) to transmit to the other terminal device of the third terminal device (330) and the fourth terminal device (340) through a network (350). Each of the third terminal device (330) and the fourth terminal device (340) can also receive the encoded video data transmitted by the other terminal device of the third terminal device (330) and the fourth terminal device (340), and can decode the encoded video data to restore the video data, and can display the video image on an accessible display device based on the restored video data.

[0071] exist Figure 2 In the embodiment of the present invention, the first terminal device (310), the second terminal device (320), the third terminal device (330) and the fourth terminal device (340) may be servers, personal computers and smart phones, but the principles disclosed in the present application may not be limited thereto. The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network (350) represents any number of networks that transmit encoded video data between the first terminal device (310), the second terminal device (320), the third terminal device (330) and the fourth terminal device (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) may exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purpose of this discussion application, unless explained below, the architecture and topology of the network (350) may be irrelevant to the operation disclosed in the present application.

[0072] As an example, Figure 4The video encoder and video decoder are shown in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0073] The streaming system may include an acquisition subsystem (413), which may include a video source (401) such as a digital camera, which creates an uncompressed video image stream (402). In an embodiment, the video image stream (402) includes samples captured by a digital camera. Compared to the encoded video data (404) (or the encoded video code stream), the video image stream (402) is depicted as a thick line to emphasize the high data volume of the video image stream, and the video image stream (402) can be processed by an electronic device (420), which includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video image stream (402), the encoded video data (404) (or the encoded video bitstream (404)) is depicted as a thin line to emphasize the lower amount of data of the encoded video data (404) (or the encoded video bitstream (404)), which can be stored on the streaming server (405) for future use. One or more streaming client subsystems, such as Figure 4 The client subsystem (406) and the client subsystem (408) in the streaming server (405) can access the streaming server (405) to retrieve the copy (407) and the copy (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in the electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and generates an output video image stream (411) that can be presented on a display (412) (e.g., a display screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (404), the video data (407), and the video data (409) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application can be used in the context of the VVC standard.

[0074] It should be noted that the electronic device (420) and the electronic device (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0075] Figure 5 is a block diagram of a video decoder (510) according to an embodiment disclosed in the present application. The video decoder (510) may be provided in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 3 A video decoder (410) of an embodiment.

[0076] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510); in the same or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective consuming entities (not shown). The receiver (531) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, the buffer memory (515) may be provided external to the video decoder (510) (not shown). In other cases, a buffer memory (not shown) is provided outside the video decoder (510) to prevent network jitter, for example, and another buffer memory (515) may be configured inside the video decoder (510) to handle broadcast timing, for example. When the receiver (531) receives data from a storage / forward device with sufficient bandwidth and controllability or from an isochronous network, it may not be necessary to configure the buffer memory (515), or the buffer memory may be made smaller. Of course, in order to use on a service packet network such as the Internet, a buffer memory (515) may also be required, and the buffer memory may be relatively large and have an adaptive size, and may be at least partially implemented in an operating system or a similar element (not shown) outside the video decoder (510).

[0077] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The types of symbols include information used to manage the operation of the video decoder (510) and potential information used to control a display device such as a display device (512) (e.g., a display screen) that is not part of the electronic device (530) but can be coupled to the electronic device (530), such as Figure 5 As shown in . The control information for the display device may be a parameter set fragment (not indicated) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (520) may extract a subgroup parameter set of at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and the like. The parser (420) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0078] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515), thereby creating symbols (521).

[0079] Depending on the type of coded video picture or part of coded video picture (e.g., inter-frame and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. For the sake of brevity, such subgroup control information flow between the parser (520) and the multiple units below is not described.

[0080] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In a practical embodiment operating under commercial constraints, many of these units interact closely with each other and can be integrated with each other. However, for the purpose of describing the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.

[0081] The first unit is a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) receives quantized transform coefficients as symbols (521) from the parser (520) and control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (551) can output a block including sample values, which can be input into an aggregator (555).

[0082] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed image, but may use predictive information from a previously reconstructed portion of a current image. Such predictive information may be provided by an intra-image prediction unit (552). In some cases, the intra-image prediction unit (552) generates surrounding blocks of the same size and shape as the block being reconstructed using reconstructed information extracted from a current image buffer (558). For example, the current image buffer (558) buffers a partially reconstructed current image and / or a fully reconstructed current image. In some cases, the aggregator (555) adds the prediction information generated by the intra-frame prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0083] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (553) may access the reference image memory (557) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (521), these samples may be added to the output of the scaler / inverse transform unit (551) (in this case referred to as residual samples or residual signals) by the aggregator (555) to generate output sample information. The acquisition of the predicted samples by the motion compensated prediction unit (553) from the address in the reference image memory (557) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (553) in the form of the symbols (521), for example, including X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (557) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0084] The output samples of the aggregator (555) may be used by various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques that are controlled by parameters included in the encoded video sequence (also referred to as the encoded video bitstream) and that are available to the loop filter unit (556) as symbols (521) from the parser (520). However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of the encoded image or encoded video sequence, as well as to previously reconstructed and loop filtered sample values.

[0085] The output of the loop filter unit (556) may be a sample stream that may be output to a display device (512) and stored in a reference picture memory (557) for subsequent inter-picture prediction.

[0086] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557), and a new current picture buffer may be reallocated before starting reconstruction of a subsequent coded picture.

[0087] The video decoder (510) may perform decoding operations according to a predetermined video compression technique, such as in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the profile recorded in the video compression technique or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technique or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference image size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.

[0088] In an embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0089] Figure 6 6 is a block diagram of a video encoder (603) according to an embodiment disclosed in the present application. The video encoder (603) is provided in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace Figure 3 A video encoder (603) in an embodiment.

[0090] The video encoder (603) can be used to obtain the video source (601) (not Figure 5 In another embodiment, the video source (601) is a part of the electronic device (620) to receive video samples, and the video source can collect video images to be encoded by the video encoder (603). In another embodiment, the video source (601) is a part of the electronic device (620).

[0091] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (603), wherein the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (601) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate images that are given motion when viewed sequentially. The image itself may be constructed as a spatial pixel array, wherein each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on the samples.

[0092] According to an embodiment, the video encoder (603) may encode and compress the images of the source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing the appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to these units. For the sake of simplicity, the coupling is not marked in the figure. The parameters set by the controller (650) may include rate control related parameters (image skipping, quantizer, lambda value of rate distortion optimization technology, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be used to have other suitable functions that are related to the video encoder (603) optimized for a certain system design.

[0093] In some embodiments, the video encoder (603) operates in an encoding loop. As a simple description, in embodiments, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input image to be encoded and a reference image) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression technology considered in this application, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (634). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference image memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference image synchronicity (and the drift that occurs when synchronicity cannot be maintained, eg due to channel errors) is also used in some related techniques.

[0094] The operation of the "local" decoder (633) can be combined with the above Figure 4 The "remote" decoder described in detail for the video decoder (510) is identical. However, additional brief reference is made to Figure 5 , when symbols are available and the entropy encoder (645) and parser (520) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0095] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the present application focuses on decoder operation. The description of encoder technology can be simplified because encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is only needed in certain areas and is provided below.

[0096] During operation, in some embodiments, the source encoder (630) may perform motion compensated predictive coding. The motion compensated predictive coding predictively encodes an input image with reference to one or more previously encoded images from a video sequence designated as "reference images." In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input image and pixel blocks of a reference image that may be selected as a prediction reference for the input image.

[0097] The local video decoder (633) may decode the encoded video data of the picture that may be designated as the reference picture based on the symbol created by the source encoder (630). The operation of the encoding engine (632) may be a lossy process. When the encoded video data is available at the video decoder ( Figure 6 When the video sequence is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.

[0098] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new image to be encoded, the predictor (635) may search the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new image. The predictor (635) may operate on a pixel block-by-pixel block basis to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it may be determined that the input image may have prediction references taken from a plurality of reference images stored in the reference image memory (634).

[0099] The controller (650) may manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0100] The outputs of all the above functional units may be entropy encoded in an entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0101] The transmitter (640) may buffer one or more encoded video sequences created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) may combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0102] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types:

[0103] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow for different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0104] Predictive pictures (P pictures), which may be pictures that may be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0105] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0106] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8 or 16×16 samples) and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or they may be predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0107] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0108] In an embodiment, the transmitter (640) may transmit additional data along with the encoded video when transmitting the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include time / spatial / SNR enhancement layers, redundant pictures and slices, other forms of redundant data, SEI messages, VUI parameter set fragments, etc.

[0109] The captured video may be taken as a plurality of source images (video images) in a temporal sequence. Intra-image prediction (often simplified to intra-frame prediction) exploits spatial correlations in a given image, while inter-image prediction exploits (temporal or other) correlations between images. In an embodiment, a particular image being encoded / decoded is divided into blocks, and the particular image being encoded / decoded is referred to as the current image. When a block in the current image is similar to a reference block in a reference image that was previously encoded in the video and is still buffered, the block in the current image may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference image, and in the case where multiple reference images are used, the motion vector may have a third dimension that identifies the reference image.

[0110] In some embodiments, bidirectional prediction techniques may be used in inter-frame image prediction. According to the bidirectional prediction technique, two reference images are used, for example, a first reference image and a second reference image, both of which are before the current image in the video in decoding order (but may be in the past and future in display order, respectively). A block in the current image may be encoded by a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. Specifically, the block may be predicted by a combination of the first reference block and the second reference block.

[0111] In addition, merge mode technology can be used in inter-picture prediction to improve coding efficiency.

[0112] According to some embodiments disclosed in the present application, predictions such as inter-frame image prediction and intra-frame image prediction are performed in units of blocks. For example, according to the HEVC standard, an image in a video image sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the image have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Furthermore, each CTU can be split into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In an embodiment, each CU is analyzed to determine the prediction type for the CU, such as an inter-frame prediction type or an intra-frame prediction type. In addition, depending on temporal and / or spatial predictability, the CU is split into one or more prediction units (PUs). Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In an embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Taking the luminance prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luminance values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and the like.

[0113] Figure 7 is a diagram of a video encoder (703) according to another embodiment disclosed in the present application. The video encoder (703) is used to receive a processed block (e.g., a prediction block) of sample values ​​within a current video image in a video image sequence, and encode the processed block into an encoded image that is part of an encoded video sequence. In this embodiment, the video encoder (703) is used to replace Figure 4A video encoder (403) in an embodiment.

[0114] In an HEVC embodiment, a video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) uses, for example, rate-distortion (RD) optimization to determine whether to use intra mode, inter mode, or bidirectional prediction mode to encode the processing block. When encoding the processing block in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded image; and when encoding the processing block in inter mode or bidirectional prediction mode, the video encoder (703) may use inter prediction or bidirectional prediction techniques to encode the processing block into an encoded image, respectively. In some video encoding techniques, the merge mode may be an inter-image prediction submode, in which a motion vector is derived from one or more motion vector predictors without the aid of an encoded motion vector component external to the predictor. In some other video encoding techniques, there may be a motion vector component applicable to the subject block. In an embodiment, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0115] exist Figure 7 In an embodiment of the present invention, the video encoder (703) includes: Figure 7 Shown are an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0116] The inter-frame encoder (730) is used to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in a previous image and a subsequent image), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference image is a decoded reference image decoded based on the encoded video information.

[0117] The intra-frame encoder (722) is used to receive samples of a current block (e.g., a processing block), compare the block with an encoded block in the same image in some cases, generate quantization coefficients after transformation, and in some cases also generate intra-frame prediction information (e.g., intra-frame prediction direction information according to one or more intra-frame coding techniques). In an embodiment, the intra-frame encoder (722) also calculates an intra-frame prediction result (e.g., a predicted block) based on the intra-frame prediction information and a reference block in the same image.

[0118] The general controller (721) is used to determine the general control data and control other components of the video encoder (703) based on the general control data. In an embodiment, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and add the intra prediction information to the bitstream; and when the mode is inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select the inter prediction information and add the inter prediction information to the bitstream.

[0119] The residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra-frame encoder (722) or the inter-frame encoder (730). The residual encoder (724) is used to operate based on the residual data to encode the residual data to generate a transform coefficient. In an embodiment, the residual encoder (724) is used to convert the residual data from the time domain to the frequency domain and generate a transform coefficient. The transform coefficient is then processed by quantization to obtain a quantized transform coefficient. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (722) and the inter-frame encoder (730). For example, the inter-frame encoder (730) can generate a decoded block based on the decoded residual data and the inter-frame prediction information, and the intra-frame encoder (722) can generate a decoded block based on the decoded residual data and the intra-frame prediction information. The decoded blocks are appropriately processed to generate a decoded picture, and in some embodiments, the decoded picture may be buffered in a memory circuit (not shown) and used as a reference picture.

[0120] The entropy encoder (725) is used to format the code stream to produce an encoded block. The entropy encoder (725) generates various information according to a suitable standard such as the HEVC standard. In an embodiment, the entropy encoder (725) is used to obtain general control data, selected prediction information (such as intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the code stream. It should be noted that according to the disclosed subject matter, when the block is encoded in the inter-frame mode or the merge sub-mode of the bidirectional prediction mode, there is no residual information.

[0121] Figure 8FIG. 8 is a diagram of a video decoder (810) according to another embodiment disclosed in the present application. The video decoder (810) is used to receive an encoded image as part of an encoded video sequence, and decode the encoded image to generate a reconstructed image. In an embodiment, the video decoder (810) is used to replace Figure 4 A video decoder (410) of an embodiment.

[0122] exist Figure 8 In one embodiment, the video decoder (810) includes: Figure 7 An entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874) and an intra-frame decoder (872) coupled together are shown in FIG.

[0123] The entropy decoder (871) can be used to reconstruct certain symbols from the encoded image, which represent syntax elements that constitute the encoded image. Such symbols may include, for example, a mode for encoding the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, a merged sub-mode of the latter two, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can identify certain samples or metadata for prediction by the intra-frame decoder (872) or the inter-frame decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients, etc. In an embodiment, when the prediction mode is inter-frame or bidirectional prediction mode, the inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is an intra-frame prediction type, the intra-frame prediction information is provided to the intra-frame decoder (872). The residual information can be inverse quantized and provided to the residual decoder (873).

[0124] The inter-frame decoder (880) is used to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0125] The intra-frame decoder (872) is used to receive intra-frame prediction information and generate a prediction result based on the intra-frame prediction information.

[0126] The residual decoder (873) is used to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require some control information (to obtain the quantizer parameter QP), and this information can be provided by the entropy decoder (871) (the data path is not indicated because this is only low-volume control information).

[0127] The reconstruction module (874) is used to merge the residual output by the residual decoder (873) with the prediction result (which can be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which can be part of the reconstructed image, which in turn can be part of the reconstructed video. It should be noted that other suitable operations such as deblocking operations can be performed to improve visual quality.

[0128] It should be noted that the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using any suitable technology. In an embodiment, the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using one or more processors that execute software instructions. Various aspects of the present application provide filtering techniques for video encoding / decoding.

[0129] Aspects of the present application provide filtering techniques for video encoding / decoding.

[0130] An adaptive loop filter (ALF) with block-based filter adaptation can be applied by the encoder / decoder to reduce artifacts. For the luma component, one of multiple filters (e.g., 25 filters) can be selected for a 4×4 luma block based on the direction and activity of the local gradient, for example.

[0131] The ALF may have any suitable shape and size. Fig. 9 , ALF (910)-(911) has a diamond shape, for example, ALF (910) is a 5×5 diamond shape, and ALF (911) is a 7×7 diamond shape. In ALF (910), elements (920)-(932) form a diamond shape and can be used in the filtering process. Seven values ​​(e.g., C0-C6) can be used for elements (920)-(932). In ALF (911), elements (940)-(964) form a diamond shape and can be used in the filtering process. 13 values ​​(e.g., C0-C12) can be used for elements (940)-(964).

[0132] refer to Fig. 9In some examples, two ALFs (910)-(911) with diamond filter shapes are used. A 5×5 diamond filter (910) can be applied to chroma components (e.g., chroma blocks, chroma CBs), and a 7×7 diamond filter (911) can be applied to luma components (e.g., luma blocks, luma CBs). Other suitable one or more shapes and one or more sizes can be used in the ALFs. For example, a 9×9 diamond filter can be used.

[0133] The filter coefficients at the positions indicated by the values ​​(eg, C0-C6 in (910) or C0-C12 in (920)) may be non-zero. Further, when the ALF includes a clipping function, the clipping values ​​at these positions may be non-zero.

[0134] For block classification of the luminance component, a 4×4 block (or luminance block, luminance CB) can be classified or categorized into one of a plurality of (e.g., 25) classes. The quantized values ​​of the directionality parameter D and the activity value A can be used to classify the luminance component into one of a plurality of (e.g., 25) classes. The classification index C is obtained using equation (1). In order to calculate the directivity parameter D and the quantization value The gradient g in the vertical direction, horizontal direction, and two diagonal directions (e.g., d1 and d2) can be calculated using the following 1-D Laplacian operator: v , g h , g d1 and g d2 . Where indices i and j refer to the coordinates of the top left sample within the 4×4 block, and R(k,l) indicates the reconstructed sample at coordinate (k,l). Each direction (eg, d1 and d2) may refer to two diagonal directions.

[0135] To reduce the complexity of the block classification described above, a sub-sampled 1-D Laplacian calculation may be performed. FIG. 10A to FIG. 10D The calculation methods for the vertical direction ( Fig. 10A ), horizontal direction( Fig. 10B ) and two diagonal directions d1( Fig. 10C ) and d2( Fig. 10D )’s gradient g v , g h , g d1 and g d2 The same subsampling position can be used for gradient calculations in different directions. Fig. 10A The label 'V' shows the calculation of the vertical gradient g v The sub-sampling position of Fig. 10B, the label 'H' shows the number of pixels used to calculate the horizontal gradient g h The sub-sampling position of Fig. 10C , label 'D1' shows the calculation of the diagonal gradient g of d1 d1 The sub-sampling position of Fig. 10D , label 'D2' shows the calculation of the diagonal gradient g of d2 d2 The subsampling position.

[0136] The horizontal gradient g v and the vertical gradient g h The maximum value of and minimum value Set to: The two diagonal gradients g d1 and g d2 The maximum value of and minimum value Set to: Based on the above values ​​and two thresholds t 1 and t 2 The following directivity parameter D can be obtained. Step 1. If (1) And (2) is true, D is set to 0. Step 2. If Then continue to step 3; otherwise, continue to step 4. Step 3. If Then set D to 2; otherwise set D to 1. Step 4. If Then set D to 4; otherwise set D to 3.

[0137] The activity value A can be calculated as: A can be further quantized to the range of 0 to 4 (inclusive), and the quantized value is expressed as

[0138] For chroma components in an image, no block classification is performed, and therefore a single set of ALF coefficients may be applied to each chroma component.

[0139] The geometric transformation may be applied to the filter coefficients and the corresponding filter clipping values ​​(also referred to as clipping values). Before filtering a block (e.g., a 4×4 luminance block), for example, based on the gradient values ​​(e.g., g v , g h , gd1 and / or d2 ), the filter coefficients f(k,l) and the corresponding filter clipping values ​​c(k,l) may be subjected to geometric transformations such as rotation or diagonal and vertical flipping. The geometric transformation applied to the filter coefficients f(k,l) and the corresponding filter clipping values ​​c(k,l) may be equivalent to a geometric transformation of the samples in the region supported by the filter. The geometric transformation may make different blocks to which the ALF is applied more similar by aligning their respective directionality.

[0140] The three geometric transformations (including diagonal flipping, vertical flipping and rotation) can be performed as described in equations (9)-(11) respectively. f D (k,l)=f(l,k),c D (k,l)=c(l,k), Equation (9) f V (k,l)=f(k,Kl-1),c V (k,l)=c(k,Kl-1) Equation (10) f R (k,l)=f(Kl-1,k),c R (k,l)=c(Kl-1,k) Equation (11) Where K is the size of the ALF or filter, and 0≤k,l≤K-1 are the coordinates of the coefficients. For example, position (0,0) is at the upper left corner of the filter f or the clipping value matrix (or clipping matrix) c, and position (K-1,K-1) is at the lower right corner of the filter f or the clipping value matrix (or clipping matrix) c. Depending on the gradient values ​​calculated for the block, these transformations can be performed on the filter coefficients f(k,l) and the clipping values ​​c(k,l). An example of the relationship between this transformation and the four gradients is summarized in Table 1. Table 1: Mapping of gradients computed for a block to transformations Gradient Value Transform <![CDATA[g d2 <g d1 And g h <g v ]]> No transformation <![CDATA[g d2 <g d1 And g v <g h ]]> Diagonal Flip <![CDATA[g d1 <g d2 And g h <g v ]]> Flip Vertically <![CDATA[g d1 <g d2 And g v <g h ]]> Rotation

[0141] In some embodiments, the ALF filter parameters are signaled in an adaptive parameter set (APS) for an image. In the APS, one or more sets of luma filter coefficients and clipping value indices (e.g., up to 25 sets) may be signaled. In one example, one of the one or more sets may include luma filter coefficients and one or more clipping value indices. One or more sets of chroma filter coefficients and clipping value indices (e.g., up to 8 sets) may be signaled. In order to reduce signaling overhead, filter coefficients of different classifications (e.g., with different classification indices) of luma components may be merged. In the slice header, the index of the APS for the current slice may be signaled.

[0142] In an embodiment, a clipping value index (also referred to as a clipping index) can be decoded from the APS. The clipping value index can be used to determine the corresponding clipping value, for example, based on the relationship between the clipping value index and the corresponding clipping value. The relationship can be predefined and stored in the decoder. In one example, the relationship is described by a table, such as a luminance table (e.g., for luminance CB) indexed by the clipping value and the corresponding clipping value, a chrominance table (e.g., for chrominance CB) indexed by the clipping value and the corresponding clipping value. The clipping value may depend on the bit depth B. The bit depth B may refer to an internal bit depth, a bit depth of reconstructed samples in a CB to be filtered, etc. In some examples, equation (12) is used to obtain a table (e.g., a luminance table, a chrominance table). Wherein, AlfClip is the clip value, B is the bit depth (e.g., bitDepth), N (e.g., N=4) is the number of allowed clip values, and (n-1) is the clip value index (also referred to as the clip index or clipIdx). Table 2 shows an example of a table obtained using equation (12), where N=4. In Table 2, the clip index (n-1) can be 0, 1, 2, and 3, and n can be 1, 2, 3, and 4, respectively. Table 2 can be used for luminance blocks or chrominance blocks. Table 2 - AlfClip can depend on bit depth B and clipIdx

[0143] In the slice header of the current slice, one or more APS indexes (e.g., up to 7 APS indexes) may be signaled to indicate multiple luma filter sets that may be used for the current slice. The filtering process may be controlled at one or more appropriate levels (e.g., image level, slice level, and / or CTB level, etc.). In an embodiment, the filtering process may be further controlled at the CTB level. A flag may be signaled to indicate whether the ALF is applied to the luma CTB. The luma CTB may select a filter set from a plurality of fixed filter sets (e.g., 16 fixed filter sets) and one or more filter sets signaled in the APS (also referred to as one or more filter sets signaled). A filter set index may be signaled for the luma CTB to indicate the filter set to be used (e.g., a filter set among a plurality of fixed filter sets and one or more filter sets signaled). Multiple fixed filter sets may be predefined and hard-coded in the encoder and decoder and may be referred to as predefined filter sets.

[0144] For chroma components, an APS index may be signaled in the slice header to indicate the chroma filter set to be used for the current slice. At the CTB level, if more than one chroma filter set exists in the APS, the filter set index may be signaled for each chroma CTB.

[0145] The filter coefficients may be quantized with a norm equal to 128. To reduce multiplication complexity, bitstream conformance may be applied so that the coefficient values ​​at non-center positions may be in the range of -27 to 27-1 (inclusive). In one example, the center position coefficients are not signaled in the bitstream and may be considered equal to 128.

[0146] In some embodiments, the syntax and semantics of the clip index and clip value are defined as follows: alf_luma_clip_idx[sfIdx][j] can be used to indicate the clip index of the clip value to be used before multiplying the jth coefficient of the luminance filter indicated by the signal indicated by sfIdx. The code stream consistency requirements may include that the value of alf_luma_clip_idx[sfIdx][j] for sfIdx=0 to alf_luma_num_filters_signalled_minus1 and j=0 to 11 should be in the range of 0 to 3 (including the end value). The luma filter clipping value AlfClipL[adaptation_parameter_set_id] having the elements AlfClipL[adaptation_parameter_set_id][filtIdx][j] (where filtIdx=0 to NumAlfFilters-1 and j=0 to 11) may be derived as indicated in Table 2 based on bitDepth being set equal to BitDepthY and clipIdx being set equal to alf_luma_clip_idx[alf_luma_coeff_delta_idx[filtIdx]][j]. alf_chroma_clip_idx[altIdx][j] may be used to indicate the clip index of the clip value to be used before multiplying the jth coefficient of the optional chroma filter with index altIdx. Codestream conformance requirements may include that the value of alf_chroma_clip_idx[altIdx][j] for altIdx=0 to alf_chroma_num_alt_filters_minus1, j=0 to 5 should be in the range of 0 to 3 (inclusive). The chroma filter clipping value AlfClipC[adaptation_parameter_set_id][altIdx][j] with elements AlfClipC[adaptation_parameter_set_id][altIdx][j] (where altIdx=0 to alf_chroma_num_alt_filters_minus1, j=0 to 5) can be derived based on bitDepth being set equal to BitDepthC and clipIdx being set equal to alf_chroma_clip_idx[altIdx][j] and as indicated in Table 2.

[0147] In an embodiment, the filtering process may be described as follows. At the decoder side, when ALF is enabled for CTB, the sample R(i, j) within the CU (or CB) may be filtered to generate a filtered sample value R'(i, j), as shown in the following equation (13). In one example, each sample in the CU is filtered. Wherein, f(k, l) represents the decoded filter coefficient, K(x, y) is the clipping function, and c(k, l) represents the decoded clipping parameter (or clipping value). The variables k and l can vary between -L / 2 and L / 2, where L represents the filter length. The clipping function K(x, y) = min(y, max(-y, x)) corresponds to the clipping function Clip3(-y, y, x). By introducing the clipping function K(x, y), the loop filtering method (e.g., ALF) becomes a nonlinear process and can therefore be referred to as a nonlinear ALF.

[0148] In a nonlinear ALF, multiple sets of clipping values ​​may be provided in Table 3. In one example, the luma set includes four clipping values ​​{1024, 181, 32, 6}, and the chroma set includes 4 clipping values ​​{1024, 161, 25, 4}. The four clipping values ​​in the luma set may be selected by approximately equally dividing the full range (e.g., 1024) of the sample values ​​of the luma block (encoded in 10 bits) in the logarithmic domain. The range of the chroma set may be 4 to 1024. Table 3 – Examples of limit values

[0149] The selected clipping value may be encoded in the "alf_data" syntax element as follows: The clipping index corresponding to the selected clipping value, such as shown in Table 3, may be encoded using a suitable coding scheme, such as a Golomb coding scheme. The coding scheme may be the same coding scheme used to encode the filter set index.

[0150] In an embodiment, a virtual boundary filtering process may be used to reduce the line buffer requirements of the ALF. Accordingly, modified block classification and filtering may be used for samples near a CTU boundary (e.g., a horizontal CTU boundary). The virtual boundary (1130) may be achieved by moving the horizontal CTU boundary (1120) by “N”. 样本 " samples and is defined as a line, such as Fig.11A As shown in 样本 Can be a positive integer. In one example, N of the brightness component 样本 is equal to 4, while the chrominance component N 样本 Equals 2.

[0151] refer to Fig.11A , the modified block classification can be applied to the luminance component. In one example, for the 1D Laplacian gradient calculation of the 4×4 block (1110) above the virtual boundary (1130), only the samples above the virtual boundary (1130) are used. Similarly, referring to Fig. 11B , for the 1D Laplacian gradient calculation of the 4×4 block (1111) below the virtual boundary (1131) moved from the CTU boundary (1121), only the samples below the virtual boundary (1131) are used. By considering the reduced number of samples used in the 1D Laplacian gradient calculation, the quantization of the activity value A can be scaled accordingly.

[0152] For the filtering process, symmetric padding operations at virtual boundaries may be used for both luma and chroma components. FIG. 12A to FIG. 12F An example of such modified ALF filtering for luma components at virtual boundaries is shown. When a filtered sample is below a virtual boundary, neighboring samples above the virtual boundary may be filled. When a filtered sample is above a virtual boundary, neighboring samples below the virtual boundary may be filled. Fig. 12A , the adjacent sample C0 can be filled with the sample C2 located below the virtual boundary (1210). Fig. 12B , the adjacent sample C0 can be filled with the sample C2 located above the virtual boundary (1220). Fig. 12C , the adjacent samples C1-C3 can be filled with samples C5-C7 located below the virtual boundary (1230), respectively. Fig.12D , adjacent samples C1-C3 can be filled with samples C5-C7 located above the virtual boundary (1240), respectively. Fig.12E , adjacent samples C4-C8 can be filled with samples C10, C11, C12, C11, and C10 located below the virtual boundary (1250), respectively. Fig.12F, adjacent samples C4-C8 may be filled with samples C10, C11, C12, C11, and C10 located above the virtual boundary (1260), respectively.

[0153] In some examples, when one or more samples and one or more adjacent samples are located to the left (or right) and to the right (or left) of the virtual boundary, the above description may be appropriately modified.

[0154] According to one aspect of the present application, in order to improve coding efficiency, the image can be partitioned based on the filtering process. In some examples, CTU is also referred to as the largest coding unit (LCU). In one example, the CTU or LCU may have a size of 64×64 pixels. In some embodiments, LCU-aligned picture quadtree splitting may be used for filtering-based partitioning. In some examples, a coding unit synchronous picture quadtree-based adaptive loop filter may be used. For example, a luminance image may be partitioned into several multi-level quadtree partitions, each partition boundary aligned with the boundary of the LCU. Each partition has its own filtering process and is therefore referred to as a filter unit (FU).

[0155] In some examples, a 2-channel coded stream can be used. At the first channel of the 2-channel coded stream, the quadtree segmentation mode of the image and the optimal filter for each FU can be determined. In some embodiments, the quadtree segmentation mode of the image and the optimal filter for the FU are determined based on the filtering distortion. The filtering distortion can be estimated by a fast filtering distortion estimation (FFDE) technique during the determination process. The image is partitioned using quadtree segmentation. The reconstructed image can be filtered based on the determined quadtree segmentation mode and the selected filters for all FUs.

[0156] At the second channel of the 2-channel coded stream, CU synchronous ALF on / off control is performed. According to the ALF on / off result, the first filtered image is partially restored using the reconstructed image.

[0157] Specifically, in some examples, a top-down segmentation strategy is adopted to divide the image into multi-level quadtree partitions based on the rate-distortion criterion. Each partition is called a filter unit (FU). The segmentation process aligns the quadtree partitions with the LCU boundaries. The encoding order of the FU follows a "z" scanning order.

[0158] Fig.13 1 shows some examples of partitioning of the present application. Fig.13 In the example of , the image (1300) is divided into 10 FUs, and the encoding order of them is FU0, FU1, FU2, FU3, FU4, FU5, FU6, FU7, FU8 and FU9.

[0159] Fig.14 A quadtree segmentation pattern 1400 of the image 1300 is shown. Fig.14 In some examples, the split flag is used to indicate the image partition mode. For example, "1" indicates that quadtree splitting is performed on the block; and "0" indicates that the block is not further split. In some examples, the minimum size FU has the size of the LCU, and the minimum size FU does not need a split flag. The split flag is as follows Fig.14 The "z" shaped sequence shown in is encoded and transmitted.

[0160] In some examples, the filter for each FU is selected from two filter sets based on a rate-distortion criterion. The first set has 1 / 2-symmetric square filters and diamond filters derived for the current FU. The second set comes from a time-delay filter buffer; the time-delay filter buffer stores filters previously derived for FUs of previous images. The filter with the minimum rate-distortion cost in these two sets can be selected for the current FU. Similarly, if the current FU is not the minimum FU and can be further divided into 4 sub-FUs, the rate-distortion costs of the 4 sub-FUs are calculated. By recursively comparing the rate-distortion costs of the segmented and non-segmented cases, the image quadtree segmentation mode can be determined.

[0161] In some examples, the maximum quadtree split level can be used to limit the maximum number of FUs. In one example, when the maximum quadtree split level is 2, the maximum number of FUs is 16. Further, in determining the quadtree split process, the correlation values ​​used to derive the Wiener coefficients of the 16 FUs (minimum FUs) at the bottom quadtree level can be reused. The remaining FUs can derive their Wiener filters from the correlation of the 16 FUs at the bottom quadtree level. Therefore, in this example, only one frame buffer access is performed to derive the filter coefficients of all FUs.

[0162] After determining the quadtree partitioning mode, in order to further reduce the filtering distortion, CU synchronous ALF on / off control can be performed. By comparing the filtering distortion and non-filtering distortion at each leaf CU, the leaf CU can explicitly turn on / off the ALF in its local area. In some examples, the coding efficiency can be further improved by redesigning the filter coefficients according to the ALF on / off results.

[0163] A cross-component filtering process may apply a cross-component filter, such as a cross-component adaptive loop filter (CC-ALF). The cross-component filter may use a luma sample value of a luma component (e.g., a luma CB) to modify a chroma component (e.g., a chroma CB corresponding to the luma CB). In one example, the luma CB and the chroma CB are located in one CU.

[0164] Fig.15 The cross component filter (e.g., CC-ALF) for generating chrominance components in an embodiment of the present application is shown. In some examples, Fig.15 The filtering process of a first chroma component (e.g., a first chroma CB), a second chroma component (e.g., a second chroma CB), and a luma component (e.g., a luma CB) is shown. The luma component can be filtered using a sample adaptive offset (SAO) filter (1510) to generate an SAO filtered luma component (1541). The SAO filtered luma component (1541) can be further filtered using an ALF luma filter (1516) to become a filtered luma CB (1561) (e.g., "Y").

[0165] The first chroma component may be filtered through an SAO filter (1512) and an ALF chroma filter (1518) to generate a first intermediate component (1552). Further, the SAO filtered luma component (1541) may be filtered by a cross component filter (e.g., CC-ALF) (1521) for the first chroma component to generate a second intermediate component (1542). Subsequently, a filtered first chroma component (1562) (e.g., “Cb”) may be generated based on at least one of the second intermediate component (1542) and the first intermediate component (1552). In one example, the filtered first chroma component (1562) (e.g., “Cb”) may be generated by combining the second intermediate component (1542) and the first intermediate component (1552) using an adder (1522). The cross component adaptive loop filtering process for the first chroma component may include steps performed by the CC-ALF (1521) and steps performed by, for example, the adder (1522).

[0166] The above description may be applicable to the second chroma component. The second chroma component may be filtered using an SAO filter (1514) and an ALF chroma filter (1518) to generate a third intermediate component (1553). Further, the SAO-filtered luminance component (1541) may be filtered by a cross component filter (e.g., CC-ALF) (1531) for the second chroma component to generate a fourth intermediate component (1543). Subsequently, a filtered second chroma component (1563) (e.g., "Cr") may be generated based on at least one of the fourth intermediate component (1543) and the third intermediate component (1553). In one example, the filtered second chroma component (1563) (e.g., "Cr") may be generated by merging the fourth intermediate component (1543) and the third intermediate component (1553) by an adder (1532). In one example, the cross-component adaptive loop filtering process for the second chroma component may include steps performed by the CC-ALF (1531) and steps performed by, for example, an adder (1532).

[0167] The cross-component filter (e.g., CC-ALF (1521), CC-ALF (1531)) can operate by applying a linear filter with any suitable filter shape to the luminance component (or luminance channel) to modify each chrominance component (e.g., first chrominance component, second chrominance component).

[0168] Fig.16An example of a filter (1600) of an embodiment of the present application is shown. The filter (1600) may include non-zero filter coefficients and zero filter coefficients. The filter (1600) has a diamond shape (1620) (represented by a black-filled circle) formed by the filter coefficients (1610). In one example, the non-zero filter coefficients in the filter (1600) are included in the filter coefficients (1610), and the filter coefficients not included in the filter coefficients (1610) are zero. Therefore, the non-zero filter coefficients in the filter (1600) are included in the diamond shape (1620), and the filter coefficients not included in the diamond shape (1620) are zero. In one example, the number of filter coefficients of the filter (1600) is equal to the number of filter coefficients (1610) (in Fig.16 In the example shown in , this number is 18).

[0169] CC-ALF may include any suitable filter coefficients (also referred to as CC-ALF filter coefficients). Fig.15 , CC-ALF (1521) and CC-ALF (1531) may have the same filter shape (eg Fig.16 In one example, the filter coefficient values ​​in CC-ALF (1521) are different from the filter coefficient values ​​in CC-ALF (1531).

[0170] Typically, filter coefficients (e.g., non-zero filter coefficients) in CC-ALF can be transmitted, for example, in APS. In one example, the filter coefficients can be multiplied by a factor (e.g., 2 10 ) for scaling, and rounding may be performed. Application of CC-ALF on variable block sizes may be controlled, and whether CC-ALF is applied may be signaled by a received context-coded flag (e.g., a CC-ALF enable flag) for each sample block. The context-coded flag, e.g., a CC-ALF enable flag, may be signaled at any appropriate level, e.g., a block level. For each chroma component, a block size and a CC-ALF enable flag may be received at a slice level. In some examples, block sizes of 16×16, 32×32, and 64×64 (in chroma samples) may be supported.

[0171] Fig.17 FIG. 2 shows an example of CC-ALF syntax of some embodiments of the present application. Fig.17In the example of , alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is an index indicating whether the cross component Cb filter is used, and if used, the index is the index of the cross component Cb filter. For example, when alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is equal to 0, the cross-component Cb filter is not applied to the Cb color component sample block at the luma position (xCtb, yCtb); when alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is not equal to 0, alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is the index of the filter to be applied. For example, alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]-th cross-component Cb filter is applied to a block of Cb color component samples at luma position (xCtb, yCtb).

[0172] Further, in Fig.17 In the example of , alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is used to indicate whether the cross component Cr filter is used, and if used, the index is the index of the cross component Cb filter. For example, when alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is equal to 0, the cross-component Cr filter is not applied to the Cr color component sample block at the luma position (xCtb, yCtb); when alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is not equal to 0, alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is the index of the cross-component Cr filter. For example, alf_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]-th cross-component Cr filter may be applied to a block of Cr color component samples at luma position (xCtb, yCtb).

[0173] In some examples, a chroma subsampling technique is used, so the number of samples in each of the one or more chroma blocks can be less than the number of samples in the luma block. The chroma subsampling format (also referred to as a chroma subsampling format, for example, specified by chroma_format_idc) can indicate a chroma horizontal subsampling factor (e.g., SubWidthC) and a chroma vertical subsampling factor (e.g., SubHeightC) between each of the one or more chroma blocks and the corresponding luma block. In one example, the chroma subsampling format is 4:2:0, so the chroma horizontal subsampling factor (e.g., SubWidthC) and the chroma vertical subsampling factor (e.g., SubHeightC) are 2, such as FIG. 18A to FIG. 18B In one example, the chroma subsampling format is 4:2:2, so the chroma horizontal subsampling factor (e.g., SubWidthC) is 2, and the chroma vertical subsampling factor (e.g., SubHeightC) is 1. In one example, the chroma subsampling format is 4:4:4, and so the chroma horizontal subsampling factor (e.g., SubWidthC) and the chroma vertical subsampling factor (e.g., SubHeightC) are 1. The chroma sample type (also referred to as the chroma sample position) may indicate the relative position of the chroma samples in the chroma block relative to at least one corresponding luma sample in the luma block.

[0174] FIG. 18A to FIG. 18B FIG. 2 shows an exemplary position of chroma samples relative to luma samples according to an embodiment of the present application. Fig.18A, brightness samples (1801) are located in rows (1811)-(1818). Fig.18A The luma sample (1801) shown in can represent a portion of an image. In one example, a luma block (e.g., luma CB) includes a luma sample (1801). The luma block can correspond to two chroma blocks having a chroma subsampling format of 4:2:0. In one example, each chroma block includes a chroma sample (1803). Each chroma sample (e.g., chroma sample (1803(1)) corresponds to four luma samples (e.g., luma samples (1801(1))-(1801(4)). In one example, the four luma samples are an upper left sample (1801(1)), an upper right sample (1801(2)), a lower left sample (1801(3)), and a lower right sample (1801(4)). The chroma sample (e.g., (1803(1))) is located between the upper left sample (1801(1)) and the lower left sample (1801(3)), and the chroma sample type of the chroma block including the chroma sample (1803) may be referred to as chroma sample type 0. Chroma sample type 0 indicates a relative position 0 corresponding to the left center position between the upper left sample (1801(1)) and the lower left sample (1801(3)). Four luma samples (e.g., (1801(1))-(1801(4))) may be referred to as adjacent luma samples of the chroma sample (1803(1)).

[0175] In one example, each chroma block includes a chroma sample (1804). The above description of the chroma sample (1803) may be applicable to the chroma sample (1804), so its detailed description may be omitted for the sake of brevity. Each chroma sample in the chroma sample (1804) may be located at a center position of four corresponding luma samples, and the chroma sample type of the chroma block having the chroma sample (1804) may be referred to as chroma sample type 1. The chroma sample type 1 indicates a relative position 1 corresponding to the center position of four luma samples (e.g., (1801(1))-(1801(4))). For example, one of the chroma samples (1804) may be located at a center portion of the luma samples (1801(1))-(1801(4)).

[0176] In one example, each chroma block includes a chroma sample (1805). Each chroma sample in the chroma samples (1805) may be located at the same upper left position as the upper left sample in the four corresponding luma samples (1801), and the chroma sample type of the chroma block having the chroma sample (1805) may be referred to as a chroma sample type 2. Accordingly, each chroma sample in the chroma samples (1805) is located at the same position as the upper left sample in the four luma samples (1801) corresponding to the chroma sample. The chroma sample type 2 indicates a relative position 2 corresponding to the upper left position of the four luma samples (1801). For example, one of the chroma samples (1805) may be located at the upper left position of the luma samples (1801(1))-(1801(4)).

[0177] In one example, each chroma block includes a chroma sample (1806). Each of the chroma samples (1806) may be a chroma sample located at a top center position between a corresponding upper left sample and a corresponding upper right sample, and a chroma sample type of a chroma block having the chroma sample (1806) may be referred to as a chroma sample type 3. The chroma sample type 3 indicates a relative position 3 corresponding to a top center position between the upper left sample and the upper right sample. For example, one of the chroma samples (1806) may be located at a top center position of luma samples (1801(1))-(1801(4)).

[0178] In one example, each chroma block includes a chroma sample (1807). Each chroma sample in the chroma samples (1807) may be located at the same lower left position as the lower left sample in the four corresponding luma samples (1801), and the chroma sample type of the chroma block having the chroma sample (1807) may be referred to as a chroma sample type 4. Accordingly, each chroma sample in the chroma samples (1807) is located at the same position as the lower left sample in the four luma samples (1801) corresponding to the chroma sample. The chroma sample type 4 indicates a relative position 4 corresponding to the lower left position of the four luma samples (1801). For example, one of the chroma samples (1807) may be located at the lower left position of the luma samples (1801(1))-(1801(4)).

[0179] In one example, each chroma block includes a chroma sample (1808). Each chroma sample in the chroma samples (1808) is located at a bottom center position between a bottom left sample and a bottom right sample, and the chroma sample type of the chroma block having the chroma sample (1808) may be referred to as a chroma sample type 5. The chroma sample type 5 indicates a relative position 5 corresponding to a bottom center position between a bottom left sample and a bottom right sample in four luma samples (1801). For example, one of the chroma samples (1808) may be located between a bottom left sample and a bottom right sample in luma samples (1801(1))-(1801(4)).

[0180] In general, any suitable chroma sample type may be used for a chroma subsampling format. Chroma sample types 0-5 are examples of chroma sample types described using the chroma subsampling format 4:2:0. Other chroma sample types may be used for the chroma subsampling format 4:2:0. Further, other chroma sample types and / or variants of chroma sample types 0-5 may be used for other chroma subsampling formats, such as 4:2:2, 4:4:4, etc. In one example, a chroma sample type consisting of chroma samples (1805) and (1807) is used for the chroma subsampling format 4:2:2.

[0181] In one example, consider a luma block having alternating rows, such as rows (1811)-(1812), which respectively include the top two samples (e.g., (1801(1))-(1801(2))) of four luma samples (e.g., (1801(1))-(1801(4))) and the bottom two samples (e.g., (1801(3))-(1801(4))) of four luma samples (e.g., (1801(1)-(1801(4))). Accordingly, rows (1811), (1813), (1815), and (1817) may be is referred to as the current row (also referred to as the topfield), and rows (1812), (1814), (1816), and (1818) may be referred to as the next row (also referred to as the bottomfield). Four luminance samples (e.g., (1801(1))-(1801(4))) are located in the current row (e.g., (1811)) and the next row (e.g., (1812)). Relative positions 2-3 are located in the current row, relative positions 0-1 are located between each current row and the corresponding next row, and relative positions 4-5 are located in the next row.

[0182] Chroma samples (1803), (1804), (1805), (1806), (1807), or (1808) are located in rows (1851)-(1854) in each chroma block. The specific positions of rows (1851)-(1854) may depend on the chroma sample type of the chroma samples. For example, for chroma samples (1803)-(1804) having chroma sample types 0-1, respectively, row (1851) is located between rows (1811)-(1812). For chroma samples (1805)-(1806) having chroma sample types 2-3, respectively, row (1851) is located at the same position as the current row (1811). For chroma samples (1807)-(1808) having chroma sample types 4-5, respectively, row (1851) is located at the same position as the next row (1812). The above description is applicable to rows (1852)-(1854), and a detailed description thereof is omitted for the sake of brevity.

[0183] Any suitable scanning method may be used to display, store and / or transmit the above described Fig.18A In one example, a progressive scan method is used.

[0184] like Fig.18B As shown in , interlaced scanning can be used. As described above, the chroma subsampling format is 4:2:0 (e.g., chroma_format_idc is equal to 1). In one example, the variable chroma location type (e.g., ChromaLocType) indicates the current row (e.g., ChromaLocType is chroma_sample_loc_type_top_field) or the next row (e.g., ChromaLocType is chroma_sample_loc_type_bottom_field). The current row (1811), (1813), (1815) and (1817) and the next row (1812), (1814), (1816) and (1818) can be scanned respectively. For example, the current row (1811), (1813), (1815) and (1817) can be scanned first, and then the next row (1812), (1814), (1816) and (1818) can be scanned. The current row may include luma samples (1801) and the next row may include luma samples (1802).

[0185] Similarly, the corresponding chroma blocks can be interlaced. The rows (1851) and (1853) including the chroma samples (1803), (1804), (1805), (1806), (1807), or (1808) without filling can be referred to as the current row (or current chroma row), and the rows (1852) and (1854) including the chroma samples (1803), (1804), (1805), (1806), (1807), or (1808) with gray filling can be referred to as the next row (or next chroma row). In one example, during interlaced scanning, rows (1851) and (1853) are scanned first, and then rows (1852) and (1854) are scanned.

[0186] In some examples, constrained directional enhancement filtering techniques can be used. The use of an in-loop constrained directional enhancement filter (CDEF) can filter out coding artifacts while retaining image details. In one example (e.g., HEVC), the sample adaptive offset (SAO) algorithm can achieve similar purposes by defining signal offsets for different classes of pixels. Unlike SAO, CDEF is a nonlinear spatial filter. In some examples, CDEF can be constrained to be easily vectorized (i.e., implementable with single instruction multiple data (SIMD) operations). It should be noted that other nonlinear filters (e.g., median filters, bilateral filters) cannot be processed in the same way.

[0187] In some cases, the amount of ringing artifacts in the encoded image tends to be roughly proportional to the quantization step size. The amount of detail is a property of the input image, but the smallest detail retained in the quantized image also tends to be proportional to the quantization step size. For a given quantization step size, the magnitude of the ringing is typically smaller than the magnitude of the detail.

[0188] CDEF can be used to identify the direction of each block and then adaptively filter along the identified direction and to a lesser extent along the direction obtained by rotating the identified direction by 45 degrees. In some examples, the encoder can search for the filter strength and can explicitly signal the filter strength, which allows a high degree of control over the blurring.

[0189] Specifically, in some examples, a directional search is performed on the reconstructed pixels immediately after the deblocking filter. Since these pixels are available to the decoder, the decoder can search for the direction, so in one example, the direction does not require signaling. In some examples, the directional search can operate on a specific block size, such as an 8×8 block, which is small enough to fully handle non-straight edges and large enough to reliably estimate the direction when applied to a quantized image. In addition, having a constant direction on an 8×8 area makes it easier to vectorize the filter. In some examples, each block (e.g., 8×8) can be compared with a fully oriented block to determine the difference. A fully oriented block refers to a block in which all pixels on a line in one direction have the same value. In one example, a difference metric between the block and each fully oriented block can be calculated, such as the sum of squared differences (SSD), root mean square (RMS) error. Then, the fully oriented block with the smallest difference (e.g., minimum SSD, minimum RMS, etc.) can be determined, and the direction of the determined fully oriented block can be the direction that best matches the pattern in the block.

[0190] Fig.19 An example of directional search according to an embodiment of the present application is shown. In one example, the block (1910) is an 8×8 block that has been reconstructed and output by a deblocking filter. Fig.19 In the example of , the directional search can determine the direction from the 8 directions shown in (1920) of the block (1910). 8 fully oriented blocks (1930) corresponding to the 8 directions (1920) are formed respectively. A fully oriented block corresponding to a direction refers to a block in which the pixels on the line along the direction have the same value. Further, the difference metric between the block (1910) and each fully oriented block in the fully oriented block (1930) can be calculated, such as SSD, RMS error, etc. Fig.19 In the example of , the RMS error is shown by (1940). As shown in (1943), the RMS error between block (1910) and the fully oriented block (1933) is the smallest, so direction (1923) is the direction that best matches the pattern in block (1910).

[0191] After identifying the direction of the block, a nonlinear low-pass directional filter can be determined. For example, the filter taps of the nonlinear low-pass directional filter can be aligned along the identified direction to reduce ringing while maintaining directional edges or patterns. However, in some examples, directional filtering alone sometimes cannot sufficiently reduce ringing. In one example, additional filter taps are also used for pixels that are not distributed along the identified direction. In order to reduce the risk of blurring, additional filter taps are handled more conservatively. For this reason, CDEF includes primary filter taps and secondary filter taps. In one example, the complete 2-D CDEF filter can be expressed as equation (14): Where D represents the damping parameter, S (p) represents the strength of the primary filter tap, S (s) represents the strength of the secondary filter tap, round(·) represents an operation that rounds ties away from zero, w represents the filter weight, and f(d,S,D) is a constraint function that operates on the difference between the filtered pixel and each of the neighboring pixels. In one example, for small differences, the function f(d,S,D) is equal to D, which can make the filter behave like a linear filter; when the difference is large, the function f(d,S,D) is equal to 0, which can effectively ignore the filter tap.

[0192] In some examples, an in-loop recovery scheme is used in the video codec after deblocking, thereby generally removing noise and improving edge quality in addition to the deblocking operation. In one example, the in-loop recovery scheme is switchable within the frame of each appropriately sized tile. The in-loop recovery scheme is based on a separable symmetric Wiener filter, a dual self-guided filters with subspace projection, and a domain transform recursive filter. Because the content statistics within a frame can vary significantly, the in-loop recovery scheme is integrated into a switchable framework, where different schemes can be triggered in different areas of the frame.

[0193] A separable symmetric Wiener filter can be one of the in-loop restoration schemes. In some examples, each pixel in the degraded frame can be reconstructed as a non-causal filtered version of the pixel within a w×w window around the pixel, where w=2r+1 for an odd integer r. If the 2D filter taps are represented in column vectorized form by a matrix with w 2 × 1 element vector F, then direct LMMSE optimization results in the filter parameters being F = H -1 M is given, where H = E[XX T ] is the autocovariance of x, and w 2 The column vectorized version of the samples in the w×w window around the pixel, and M=E[YX T ] is the cross-correlation of the estimated x with the scalar source sample y. In one example, the encoder can estimate H and M based on the deblocked frame and the realization in the source, and the resulting filter F can be sent to the decoder. However, this will not only 2 The tapping results in a considerable bit rate cost, and the non-separable filtering makes decoding too complicated. In some embodiments, several additional constraints are imposed on the properties of F. For the first constraint, F is constrained to be separable so that the filtering can be implemented as separable horizontal and vertical w-tap convolutions. For the second constraint, each of the horizontal and vertical filters is constrained to be symmetrical. For the third constraint, it is assumed that the sum of the horizontal filter coefficients and the vertical filter coefficients is 1.

[0194] Dual self-guided filtering with subspace projection can be one of the in-loop restoration schemes. Guided filtering is an image filtering technique in which the local linear model is expressed by equation (15): y = F x + G Equation (15) It is used to calculate the filtered output y of the unfiltered sample x, where F and G are determined based on the statistics of the degraded image and the guide image near the filtered pixel. If the guide image is the same as the degraded image, the resulting so-called self-guided filtering has the effect of edge-preserving smoothing. In one example, a specific form of self-guided filtering can be used. The specific form of self-guided filtering depends on two parameters: the radius r and the noise parameter e, and is listed as the following steps: 1. Get the mean μ and variance σ of the pixels in the (2r+1)×(2r+1) window around each pixel 2 This step can be effectively implemented using box filtering based on integral imaging. 2. Calculate for each pixel: f = σ2 / (σ 2 +e); g = (1-f) μ 3. Calculate the F and G of each pixel as the average of the f-value and g-value in a 3×3 window around the pixel for use.

[0195] The specific form of the self-guided filter is controlled by r and e, where higher r means higher spatial variance, and higher e means higher range variance.

[0196] Fig. 20 Examples of subspace projections in some examples are shown. Fig. 20 As shown in 1 , X 2 None of them are close to the source Y, but appropriate multipliers {α,β} can bring them closer if they are moved slightly in the right direction.

[0197] In some examples (e.g., HEVC), a filtering technique called sample adaptive offset (SAO) can be used. In some examples, SAO is applied to the reconstructed signal after the deblocking filter. SAO can use an offset value given in the slice header. In some examples, for luma samples, the encoder can decide whether to apply (enable) SAO to the slice. When SAO is enabled, the current image allows the coding unit to be recursively split into four sub-regions, and each sub-region can select an SAO type from multiple SAO types based on the features in the sub-region.

[0198] Fig.21 A table (2100) of multiple SAO types of an embodiment of the present application is shown. In table (2100), SAO types 0-6 are shown. It should be noted that SAO type 0 is used to indicate that SAO is not applied. Further, each SAO type from SAO type 1 to SAO type 6 includes multiple categories. SAO can classify the reconstructed pixels of a sub-region into categories and reduce distortion by adding offsets to pixels of each category in the sub-region. In some examples, edge characteristics can be used to classify pixels into SAO types 1-4, and pixel intensity can be used to classify pixels into SAO types 5-6.

[0199] Specifically, in one embodiment, in, for example, SAO types 5-6, all pixels of a sub-area can be classified into multiple bands using a band offset (BO). Each of the multiple bands includes pixels of the same brightness interval. In some examples, the brightness range is evenly divided into multiple intervals, such as 32 intervals from zero to the maximum brightness value (e.g., 255 for 8-bit pixels), and each interval is associated with an offset. Further, in one example, the 32 bands are divided into two groups, such as a first group and a second group. The first group includes the central 16 bands (e.g., 16 intervals in the middle of the brightness range), and the second group includes the remaining 16 bands (e.g., 8 intervals at the low side of the brightness range and 8 intervals at the high side of the brightness range). In one example, only the offset of one of the two groups is transmitted. In some embodiments, when using the pixel classification operation in BO, the five most significant bits of each pixel can be used directly as the index of the band.

[0200] Further, in one embodiment, such as SAO types 1-4, edge offset (EO) can be used for pixel classification and offset determination. For example, pixel classification can be determined based on a 1-dimensional 3-pixel pattern and taking edge directionality information into consideration.

[0201] Fig. 22 An example of a 3-pixel pattern of pixel classification used in edge offset in some examples is shown. Fig. 22 In the example, the first mode (2210) (shown by 3 gray pixels) is called the 0 degree mode (the horizontal direction is associated with the 0 degree mode), the second mode (2220) (shown by 3 gray pixels) is called the 90 degree mode (the vertical direction is associated with the 90 degree mode), the third mode (2230) (as shown by 3 gray pixels) is called the 135 degree mode (the 135 degree diagonal direction is associated with the 135 degree mode), and the fourth mode (2240) (as shown by 3 gray pixels) is called the 45 degree mode (the 45 degree diagonal direction is associated with the 45 degree mode). In one example, the edge directionality information of the sub-region can be considered to obtain the Fig. 22 One of the four directional modes shown in is selected. In one example, the selected mode can be sent as side information in the encoded video bitstream. Then, by comparing each pixel with its two neighboring pixels in the direction associated with the directional mode, the pixels in the sub-region can be classified into multiple categories.

[0202] Fig.23 A table (2300) is shown of pixel classification rules for edge offset in some examples. Specifically, pixel c (also in Fig. 22 ) with two adjacent pixels (also shown in Fig. 22 in each mode (shown in gray) and can be compared based on Fig.23 The pixel classification rule shown in classifies pixel c into one of classes 0-4 based on this comparison.

[0203] In some embodiments, the SAO at the decoder end can operate independently of the largest coding unit (LCU) (e.g., CTU), so that line buffers can be saved. In some examples, when the 90 degree, 135 degree, and 45 degree classification modes are selected, the pixels in the top and bottom rows of each LCU are not SAO processed; when the 0 degree, 135 degree, and 45 degree modes are selected, the pixels in the leftmost and rightmost columns of each LCU are not SAO processed.

[0204] Fig.24 An example of syntax that may need to be signaled for a CTU if parameters are not merged from neighboring CTUs is shown (2400). For example, the syntax element sao_type_idx[cldx][rx][ry] may be signaled to indicate the SAO type of the sub-region. The SAO type may be BO (band offset) or EO (edge ​​offset). When the value of sao_type_idx[cldx][rx][ry] is 0, it indicates that SAO is off; values ​​1 to 4 indicate that one of the four EO categories corresponding to 0°, 90°, 135°, and 45° is used; a value of 5 indicates that BO is used. Fig.24 In the example of , each of the BO and EO types has four SAO offset values ​​(sao_offset[cIdx][rx][ry][0] to sao_offset[cIdx][rx][ry][3]) represented by the signal.

[0205] Typically, a filtering process may use reconstructed samples of a first color component as input (e.g., Y or Cb or Cr, or R or G or B) to generate an output, and the output of the filtering process is applied to a second color component, which may be the same color component as the first color component or may be another color component different from the first color component.

[0206] In a related example of cross component filtering (CCF), filter coefficients are derived based on some mathematical equations. The derived filter coefficients are represented by signals from the encoder end to the decoder end, and the derived filter coefficients are used to generate offsets using linear merging. Then, as a filtering process, the generated offsets are added to the reconstructed samples. For example, offsets are generated based on the linear merging of filter coefficients and luminance samples, and the generated offsets are added to the reconstructed chrominance samples. The related example of CCF is based on an assumption that there is a linear mapping relationship between the reconstructed luminance sample values ​​and the incremental values ​​(between the original chrominance samples and the reconstructed chrominance samples). However, the mapping between the reconstructed luminance sample values ​​and the incremental values ​​(between the original sample values ​​and the reconstructed chrominance sample values) does not necessarily follow a linear mapping process, so the encoding and decoding performance of CCF may be limited by the linear mapping relationship assumption.

[0207] In some examples, nonlinear mapping techniques can be used for cross-component filtering and / or same color component filtering without significant signaling overhead. In one example, nonlinear mapping techniques can be used for cross-component filtering to generate cross-component sample offsets. In another example, nonlinear mapping techniques can be used for same color component filtering to generate local sample offsets.

[0208] For convenience, the filtering process using nonlinear mapping technology can be referred to as sample offset by nonlinear mapping (SO-NLM). SO-NLM in the cross-component filtering process can be referred to as cross-component sample offset (CCSO). SO-NLM in the same color component filtering can be referred to as local sample offset (LSO). Filters using nonlinear mapping technology can be referred to as filters based on nonlinear mapping. Filters based on nonlinear mapping can include CCSO filters, LSO filters, etc.

[0209] In one example, CCSO and LSO can be used as loop filters to reduce the distortion of reconstructed samples. CCSO and LSO do not rely on the linear mapping assumption used in the related examples of CCF. For example, CCSO does not rely on the assumption of a linear mapping relationship between luma reconstructed sample values ​​and delta values ​​(between original chroma samples and chroma reconstructed samples). Similarly, LSO does not rely on the assumption of a linear mapping relationship between reconstructed sample values ​​of color components and delta values ​​(between original samples of color components and reconstructed samples of color components).

[0210] In the following description, a SO-NLM filtering process is described that uses reconstructed samples of a first color component as input (e.g., Y or Cb or Cr, or R or G or B) to generate an output, and the output of the filtering process is applied to a second color component. When the second color component is the same color component as the first color component, the description applies to LSO; and when the second color component is different from the first color component, the description applies to CCSO.

[0211] In SO-NLM, a nonlinear mapping is derived at the encoder side. The nonlinear mapping is a mapping between reconstructed samples of a first color component in the filter support region and an offset of a second color component to be added to the filter support region. Nonlinear mapping is used in LSO when the second color component is the same as the first color component and in CCSO when the second color component is different from the first color component. The domain of the nonlinear mapping is determined by the different combinations of processed input reconstructed samples (also referred to as possible combinations of reconstructed sample values).

[0212] The SO-NLM technique may be illustrated using a specific example. In the specific example, reconstructed samples of a first color component that are within a filter support area (also referred to as a "filter support region") are determined. The filter support area is a region within which a filter may be applied, and the filter support region may have any suitable shape.

[0213] Fig.25 An example of a filter support region (2500) according to some embodiments of the present application is shown. The filter support region (2500) includes four reconstructed samples: P0, P1, P2, and P3 of the first color component. Fig.25 In the example of , the four reconstructed samples may form a cross in the vertical and horizontal directions, and the center position of the cross is the position of the sample to be filtered. The sample at the center position and having the same color component as P0-P3 is represented by C. The sample at the center position and having the second color component is represented by F. The second color component may be the same as the first color component of P0-P3, or may be different from the first color component of P0-P3.

[0214] Fig.26 2600 of another filter support region according to some embodiments of the present application is shown. The filter support region (2600) includes four reconstructed samples P0, P1, P2 and P3 of the first color component forming a square shape. Fig.26In the example of , the center position of the square shape is the sample position to be filtered. The sample at the center position and having the same color component as P0-P3 is represented by C. The sample at the center position and having the second color component is represented by F. The second color component may be the same as the first color component of P0-P3, or may be different from the first color component of P0-P3.

[0215] The reconstructed samples are input to the SO-NLM filter and are appropriately processed to form filter taps. In one example, the positions of the reconstructed samples as input to the SO-NLM filter are referred to as filter tap positions. In a specific example, the reconstructed samples are processed in the following two steps.

[0216] In the first step, the incremental values ​​between P0-P3 and C are calculated respectively. For example, m0 represents the incremental value between P0 and C; m1 represents the incremental value between P1 and C; m2 represents the incremental value between P2 and C; and m3 represents the incremental value between P3 and C.

[0217] In the second step, the incremental values ​​m0-m3 are further quantized, and the quantized values ​​are represented as d0, d1, d2, d3. In one example, based on the quantization process, the quantized value can be one of -1, 0, and 1. For example, when m is less than -N (N is a positive value and is called a quantization step), the value m can be quantized to -1; when m is in the range of [-N, N], the value m can be quantized to 0; when m is greater than N, the value m can be quantized to 1. In some examples, the quantization step N can be one of 4, 8, 12, 16, etc.

[0218] In some embodiments, the quantized values ​​d0-d3 are filter taps and can be used to identify a combination in the filter domain. For example, the filter taps d0-d3 can form a combination in the filter domain. Each filter tap can have three quantized values, so when four filter taps are used, the filter domain includes 81 (3×3×3×3) combinations.

[0219] FIG. 27A to FIG. 27CA table (2700) with 81 combinations of an embodiment of the present application is shown. Table (2700) includes 81 rows corresponding to 81 combinations. In each row (which corresponds to a combination), the first column includes the index of the combination; the second column includes the value of the filter tap d0 of the combination; the third column includes the value of the filter tap d1 of the combination; the fourth column includes the value of the filter tap d2 of the combination; the fifth column includes the value of the filter tap d3 of the combination; and the sixth column includes the offset value associated with the combination for nonlinear mapping. In one example, when determining the filter taps d0-d3, the offset value (represented by s) associated with the combination of d0-d3 can be determined according to table (2700). In one example, the offset values ​​s0-s80 are integers, such as 0, 1, -1, 3, -3, 5, -5, -7, etc.

[0220] In some embodiments, the final filtering process of SO-NLM as shown in equation (16) may be applied: f′=clip(f+s) Equation (16) Wherein, f is the reconstructed sample of the second color component to be filtered, and s is an offset value determined, for example using table (2700), according to the filter taps, which are the result of processing the reconstructed samples of the first color component. The sum of the reconstructed samples F and the offset value s is further clipped to a range associated with the bit depth to determine the final filtered samples f' of the second color component.

[0221] It should be noted that, in the case of LSO, the second color component in the above description is the same as the first color component; and, in the case of CCSO, the second color component in the above description may be different from the first color component.

[0222] It should be noted that the above description may be adapted for other embodiments of the present application.

[0223] In some examples, at the encoder end, the encoding device can derive a mapping between reconstructed samples of the first color component in the filter support region and offsets to be added to the reconstructed samples of the second color component. The mapping can be any suitable linear mapping or non-linear mapping. The filtering process can then be applied at the encoder end and / or the decoder end based on the mapping. For example, the mapping is notified to the decoder in an appropriate manner (e.g., the mapping is included in the encoded video bitstream transmitted from the encoder end to the decoder end), and the decoder can then perform the filtering process based on the mapping.

[0224] The performance of filters based on nonlinear mapping (e.g., CCSO filters, LSO filters, etc.) depends on the filter shape configuration. The filter shape configuration (also referred to as filter shape) of a filter may refer to the characteristics of the pattern formed by the filter tap positions. The pattern may be defined by various parameters (e.g., the number of filter taps, the geometry of the filter tap positions, the distance of the filter tap positions to the center of the pattern, etc.). Using a fixed filter shape configuration may limit the performance of filters based on nonlinear mapping.

[0225] like Fig.24 and Fig.25 as well as FIG. 27A to FIG. 27C As shown in , some examples use a 5-tap filter design for a filter shape configuration for a filter based on a nonlinear mapping. The 5-tap filter design may use tap positions at P0, P1, P2, P3, and C. The 5-tap filter design for the filter shape configuration may produce a look up table (LUT) with 81 entries, as shown in FIG. 27A to FIG. 27C As shown in . The LUT of sample offset needs to be signaled from the encoder side to the decoder side, and the signaling of the LUT can contribute most of the signaling overhead and affect the coding efficiency using filters based on nonlinear mapping. According to some aspects of the present application, the number of filter taps can be different from 5. In some examples, the number of filter taps can be reduced, information in the filter support area can still be obtained, and coding efficiency can be improved.

[0226] In some examples, the filter shape configurations in the group for the nonlinear mapping based filters each have 3 filter taps.

[0227] Fig.28Seven filter shape configurations for three filter taps in one example are shown. Specifically, the first filter shape configuration includes 3 filter taps at positions marked as "1" and "c", and position "c" is the center position of position "1"; the second filter shape configuration includes 3 filter taps at positions marked as "2" and position "c", and position "c" is the center position of position "2"; the third filter shape configuration includes 3 filter taps at positions marked as "3" and position "c", and position "c" is the center position of position "3"; the fourth filter shape configuration includes 3 filter taps at positions marked as "4" and position "c", and position "c" is the center position of position "4"; the fifth filter shape configuration includes 3 filter taps at positions marked as "5" and "c", and position "c" is the center position of position "5"; the sixth filter shape configuration includes 3 filter taps at positions marked as "6" and position "c", and position "c" is the center position of position "6"; the seventh filter shape configuration includes 3 filter taps at positions marked as "7" and position "c", and position "c" is the center position of position "7".

[0228] According to some aspects of the present application, the non-linear mapping based filter can be used together with other in-loop filters in a loop filter chain. The position of the non-linear mapping based filter may affect the coding efficiency of the non-linear mapping based filter.

[0229] Fig.29 A block diagram of a loop filter chain (2900) in some examples is shown. The loop filter chain (2900) includes a plurality of filters connected in series in the filter chain. In one example, the loop filter chain (2900) can be used as a loop filter unit (556). The loop filter chain (2900) can be used in an encoding or decoding loop before storing the reconstructed image in a decoded image buffer (e.g., a reference image memory (557)). The loop filter chain (2900) receives input reconstructed samples from a previous processing module and applies filters to the reconstructed samples to generate output reconstructed samples.

[0230] The loop filter chain (2900) may include any suitable filters. Fig.29In the example of , the loop filter chain (2900) includes a deblocking filter (labeled as deblocking), a constrained directional enhancement filter (labeled as CDEF) and an in-loop restoration filter (labeled as LR) connected in the chain. The loop filter chain (2900) has an input node (2901), an output node (2909) and a plurality of intermediate nodes (2902)-(2903). The input node (2901) of the loop filter chain (2900) receives input reconstructed samples from a previous processing module and provides the input reconstructed samples to the deblocking filter. The intermediate node (2902) receives the reconstructed samples from the deblocking filter (after being processed by the deblocking filter) and provides the reconstructed samples to CDEF for further filter processing. The intermediate node (2903) receives the reconstructed samples from CDEF (after being processed by CDEF) and provides the reconstructed samples to the LR filter for further filter processing. The output node (2909) receives the output reconstructed samples from the LR filter (after being processed by the LR filter). The output reconstructed samples may be provided to other processing modules, such as a post-processing module for further processing.

[0231] It should be noted that the following description illustrates the technique of using a filter based on a non-linear mapping based loop filter chain (2900). The technique of using a filter based on a non-linear mapping can be used in other suitable loop filter chains.

[0232] According to one aspect of the present application, a filter based on nonlinear mapping can be connected in series with other filters in a loop filter chain, and the input and output of the filter based on nonlinear mapping can be placed at the same node of the loop filter chain, and there is no other filter between the input and output of the filter based on nonlinear mapping. For example, the filter based on nonlinear mapping receives reconstructed samples at a node of the loop filter chain, and determines a sample offset based on the reconstructed samples at the node of the loop filter chain, and then applies the sample offset to the reconstructed samples at the node of the loop filter chain.

[0233] FIG. 30A to FIG. 30D An example of a loop filter chain is shown, the loop filter chain comprising filters based on non-linear mapping connected in series in the loop filter chain.

[0234] Fig. 30AAn example of a loop filter chain (3000A) in an example is shown. The loop filter chain (3000A) can be used to replace the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3000A), a filter based on nonlinear mapping (labeled as SO-NLM) is applied at an input node. Specifically, the input of the filter based on nonlinear mapping (also referred to as the first reconstructed sample) is the input reconstructed sample at the first node (3011A) (to the loop filter chain (3000A)). Based on the input, the filter based on nonlinear mapping generates a sample offset (SO). The sample offset is merged with the intermediate reconstructed sample at the second node (3012A) to generate an output (also referred to as the second reconstructed sample) at the third node (3013A), and the output is provided to the deblocking filter for further filtering. Fig. 30A In the example of , the first node (3011A) and the second node (3012A) are the same node, and the middle reconstructed sample is the first reconstructed sample.

[0235] Fig. 30B An example of a loop filter chain (3000B) in an example is shown. The loop filter chain (3000B) can be used to replace the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3000B), a filter based on nonlinear mapping (labeled as SO-NLM) is applied at an intermediate node of the loop filter chain (3000B). Specifically, the input of the filter based on nonlinear mapping (also referred to as the first reconstructed sample) is the reconstructed sample (generated by the deblocking filter) at the first node (3011B). Based on the input, the filter based on nonlinear mapping generates a sample offset (SO). The sample offset is merged with the intermediate reconstructed sample at the second node (3012B) to generate an output (also referred to as the second reconstructed sample) at the third node (3013B), which is provided to CDEF for further filtering. Fig. 30B In the example of , the first node (3011B) and the second node (3012B) are the same node, and the middle reconstructed sample is the first reconstructed sample.

[0236] Fig. 30CAn example of a loop filter chain (3000C) in an example is shown. The loop filter chain (3000C) can be used to replace the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3000C), a filter based on nonlinear mapping (labeled as SO-NLM) is applied at an intermediate node of the loop filter chain (3000C). Specifically, the input of the filter based on nonlinear mapping (also referred to as the first reconstructed sample) is the reconstructed sample at the first node (3011C) (generated by CDEF). Based on the input, the filter based on nonlinear mapping generates a sample offset (SO). The sample offset is merged with the intermediate reconstructed sample at the second node (3012C) to generate an output (also referred to as the second reconstructed sample) at the third node (3013C), which is provided to the LR filter for further filtering. Fig. 30C In the example of , the first node (3011C) and the second node (3012C) are the same node, and the middle reconstructed sample is the first reconstructed sample.

[0237] Fig.30D An example of a loop filter chain (3000D) in an example is shown. The loop filter chain (3000D) can be used to replace the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3000D), a filter based on nonlinear mapping (labeled as SO-NLM) is applied at the output node. Specifically, the input of the filter based on nonlinear mapping (also referred to as the first reconstructed sample) is the reconstructed sample at the first node (3011D) (generated by the LR filter). Based on the input, the filter based on nonlinear mapping generates a sample offset (SO). The sample offset is merged with the intermediate reconstructed sample at the second node (3012D) to generate an output (also referred to as the second reconstructed sample) at the third node (3013D), which is the output of the loop filter chain (3000D). In Fig.30D In the example of , the first node (3011D) and the second node (3012D) are the same node, and the middle reconstructed sample is the first reconstructed sample.

[0238] According to one aspect of the present application, a filter based on nonlinear mapping can be connected in parallel with one or more filters in a loop filter chain, and there is at least one filter between the input and output of the filter based on nonlinear mapping. For example, the filter based on nonlinear mapping receives a reconstructed sample at a first node along the loop filter chain, and determines a sample offset based on the reconstructed sample at the first node along the loop filter chain, and then applies the sample offset to the reconstructed sample at the second node along the loop filter chain. The reconstructed sample at the second node along the loop filter chain can be obtained by applying one or more filters to the reconstructed sample at the first node along the loop filter chain.

[0239] In some examples, the input of the filter based on nonlinear mapping is the reconstructed samples after the deblocking filter and before the CDEF, and the output of the filter based on nonlinear mapping is applied to the reconstructed samples after the CDEF and before the LR filter, or after the LR filter.

[0240] Fig.31A An example of a loop filter chain (3100A) is shown, which includes a filter based on nonlinear mapping connected in parallel with CDEF. The loop filter chain (3100A) can be used to replace the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3100A), a filter based on nonlinear mapping (labeled as SO-NLM) is applied between two intermediate nodes. Specifically, the input of the filter based on nonlinear mapping (also referred to as the first reconstructed sample) is the reconstructed sample at the first node (3111A) (generated by the deblocking filter). Based on the input, the filter based on nonlinear mapping generates a sample offset (SO). The sample offset is merged with the intermediate reconstructed sample (generated by CDEF) at the second node (3112A) to generate an output (also referred to as the second reconstructed sample) at the third node (3113A), which is provided to the LR filter for further filtering. Fig.31A In the example of , CDEF is between the first node (3111A) and the second node (3112A). The intermediate reconstructed samples are the output of CDEF.

[0241] Fig.31BAn example of a loop filter chain (3100B) is shown, which includes a filter based on nonlinear mapping connected in parallel with a CDEF and an LR filter. The loop filter chain (3100B) can be used to replace the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3100B), a filter based on nonlinear mapping (labeled as SO-NLM) is applied between an intermediate node and an output node. Specifically, the input of the filter based on nonlinear mapping (also referred to as the first reconstructed sample) is the reconstructed sample (generated by the deblocking filter) at the first node (3111B). Based on the input, the filter based on nonlinear mapping generates a sample offset (SO). The sample offset is merged with the intermediate reconstructed sample (generated by the LR filter) at the second node (3112B) to generate an output (also referred to as the second reconstructed sample) at the third node (3113B), which is the output of the loop filter chain (3100B). At Fig.31B In the example of , CDEF and LR filter are between the first node (3111B) and the second node (3112B). The intermediate reconstructed samples are the output of the LR filter.

[0242] In some examples, the input of the filter based on nonlinear mapping is the reconstructed samples before the deblocking filter, and the output of the filter based on nonlinear mapping is applied to the reconstructed samples after the deblocking filter and before CDEF, or after CDEF and before the LR filter, or after the LR filter.

[0243] Fig.32A An example of a loop filter chain (3200A) is shown, which includes a nonlinear mapping-based filter connected in parallel with a deblocking filter. The loop filter chain (3200A) can be used to replace the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3200A), a nonlinear mapping-based filter (labeled as SO-NLM) is applied between the input node and the intermediate node of the loop filter chain. Specifically, the input of the nonlinear mapping-based filter (also referred to as the first reconstructed sample) is the input reconstructed sample at the first node (3211A) (the input of the loop filter chain (3200A)). Based on the input, the nonlinear mapping-based filter generates a sample offset (SO). The sample offset is merged with the intermediate reconstructed sample (generated by the deblocking filter) at the second node (3212A) to generate an output (also referred to as the second reconstructed sample) at the third node (3213A), which is provided to the CDEF for further filtering. Fig.32A In the example of , the deblocking filter is between the first node (3211A) and the second node (3212A). The intermediate reconstructed samples are the output of the deblocking filter.

[0244] Fig.32B An example of a loop filter chain (3200B) is shown, which includes a filter based on nonlinear mapping connected in parallel with a deblocking filter and a CDEF. The loop filter chain (3200B) can be used to replace the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3200B), a filter based on nonlinear mapping (labeled as SO-NLM) is applied between the input node and the intermediate node of the loop filter chain. Specifically, the input of the filter based on nonlinear mapping (also referred to as the first reconstructed sample) is the input reconstructed sample at the first node (3211B) (the input of the loop filter chain (3200B)). Based on the input, the filter based on nonlinear mapping generates a sample offset (SO). The sample offset is merged with the intermediate reconstructed sample (generated by CDEF) at the second node (3212B) to generate an output (also referred to as the second reconstructed sample) at the third node (3213B), which is provided to the LR for further filtering. Fig.32B In the example of , the deblocking filter and CDEF are between the first node (3211B) and the second node (3212B). The intermediate reconstructed samples are the output of CDEF.

[0245] Fig.32C An example of a loop filter chain (3200C) is shown, which includes a filter based on nonlinear mapping connected in parallel with a deblocking filter, a CDEF and an LR filter. The loop filter chain (3200C) can be used to replace the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3200C), a filter based on nonlinear mapping (labeled as SO-NLM) is applied between the input node and the output node of the loop filter chain. Specifically, the input of the filter based on nonlinear mapping (also referred to as the first reconstructed sample) is the input reconstructed sample at the first node (3211C) (the input of the loop filter chain (3200C)). Based on the input, the filter based on nonlinear mapping generates a sample offset (SO). The sample offset is merged with the intermediate reconstructed sample (generated by the LR filter) at the second node (3212C) to generate an output (also referred to as the second reconstructed sample) at the third node (3213C), which is the output of the loop filter chain (3200C). At Fig.32C In the example of , the deblocking filter, CDEF and LR filter are between the first node (3211C) and the second node (3212C). The intermediate reconstructed samples are the output of the LR filter.

[0246] In some examples, the input of the filter based on nonlinear mapping is the reconstructed samples after CDEF and before the LR filter, and the output of the filter based on nonlinear mapping is applied to the reconstructed samples after the LR filter.

[0247] Fig.33 An example of a loop filter chain (3300) is shown, which includes a filter based on nonlinear mapping connected in parallel with an LR filter. The loop filter chain (3300) can be used to replace the loop filter chain (2900) in an encoding device or a decoding device. In the loop filter chain (3300), a filter based on nonlinear mapping (labeled as SO-NLM) is applied between the intermediate node and the output node of the loop filter chain. Specifically, the input of the filter based on nonlinear mapping (also referred to as the first reconstructed sample) is the reconstructed sample at the first node (3311) (generated by CDEF). Based on the input, the filter based on nonlinear mapping generates a sample offset (SO). The sample offset is merged with the intermediate reconstructed sample (generated by the LR filter) at the second node (3312) to generate an output (also referred to as the second reconstructed sample) at the third node (3313), which is the output of the loop filter chain (3300). At Fig.33 In the example of , the LR filter is between the first node (3311) and the second node (3312). The intermediate reconstructed samples are the output of the LR filter.

[0248] According to another aspect of the present application, multiple nonlinear mapping based filters can be simultaneously applied to multiple positions in the loop filter chain. Each of the multiple nonlinear mapping based filters can be based on FIG. 30A to FIG. 30D , FIG. 31A to FIG. 31B , FIG. 32A to FIG. 32C and Fig.33 Configure any of the examples shown in .

[0249] Fig.34 An example of a loop filter chain (3400) is shown, the loop filter chain comprising a first nonlinear mapping based filter as shown in SO-NLM1 and a second nonlinear mapping based filter as shown in SO-NLM2. The loop filter chain (3400) can be used to replace the loop filter chain (2900) in an encoding device or a decoding device. The first nonlinear mapping based filter is used in conjunction with Fig.31A A similar configuration as shown in the example is connected in parallel with CDEF. The second filter based on nonlinear mapping is connected with Fig.30DA configuration similar to the example shown in is connected in series with other filters in the loop filter chain (3400). In one example, the first nonlinear mapping-based filter and the second nonlinear mapping-based filter are of the same type, such as CCSO or LSO. In another example, the first nonlinear mapping-based filter and the second nonlinear mapping-based filter are of different types, such as one is CCSO and the other is LSO.

[0250] Fig.35 An example of a loop filter chain (3500) is shown, the loop filter chain comprising a first nonlinear mapping based filter as shown in SO-NLM1 and a second nonlinear mapping based filter as shown in SO-NLM2. The loop filter chain (3500) can be used to replace the loop filter chain (2900) in an encoding device or a decoding device. The first nonlinear mapping based filter is used in conjunction with Fig.33 A configuration similar to the example shown in FIG. 1 is connected in parallel with the LR filter. A second filter based on nonlinear mapping is connected in parallel with Fig. 30B A configuration similar to the example shown in is connected in series with other filters in the loop filter chain (3500). In one example, the first nonlinear mapping-based filter and the second nonlinear mapping-based filter are of the same type, such as CCSO or LSO. In another example, the first nonlinear mapping-based filter and the second nonlinear mapping-based filter are of different types, such as one is CCSO and the other is LSO.

[0251] Fig.36 An overview flow chart of the process (3600) of an embodiment of the present application is shown. The process (3600) can be used for video filtering. When the term "block" is used, the block can be interpreted as a prediction block, a coding unit, a luminance block, a chrominance block, etc. In various embodiments, the process (3600) is performed by a processing circuit, such as a processing circuit in a terminal device (310), (320), (330) and (340), a processing circuit that performs the function of a video encoder (403), a processing circuit that performs the function of a video decoder (410), a processing circuit that performs the function of a video decoder (510), and a processing circuit that performs the function of a video encoder (603). In some embodiments, the process (3600) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit executes the process (3600). The process starts at (S3601) and proceeds to (S3610).

[0252] At (S3610), a first offset value for applying a filter based on a non-linear mapping is based on a first reconstructed sample at a first node along the loop filter chain.

[0253] At (S3620), a first offset value is applied to an intermediate reconstructed sample at a second node along the loop filter chain to generate a second reconstructed sample at a third node of the loop filter chain.

[0254] In one example, the filter based on the non-linear mapping is a cross-component sample offset (CCSO) filter, and the intermediate reconstructed sample and the first reconstructed sample have different color components.

[0255] In another example, the filter based on non-linear mapping is a local sample offset (LSO) filter, and the intermediate reconstructed samples and the first reconstructed samples have the same color components.

[0256] In some embodiments, the first node and the second node are the same node, which may be an input node of the loop filter chain or an output node of the loop filter chain or an intermediate node of the loop filter chain.

[0257] In one example, the first reconstructed samples are generated by a processing module before a deblocking filter. In another example, the first reconstructed samples are generated by a deblocking filter. In another example, the first reconstructed samples are generated by a constrained directional enhancement filter. In another example, the first reconstructed samples are generated by a loop recovery filter.

[0258] In some embodiments, the first node and the second node are different nodes. In some examples, the first reconstructed sample is generated by a processing module before the deblocking filter, and the intermediate reconstructed samples are generated by at least one of the deblocking filter, the constrained directional enhancement filter, or the loop recovery filter. In some examples, the first reconstructed sample is generated by the deblocking filter, and the intermediate reconstructed samples are generated by at least one of the constrained directional enhancement filter or the loop recovery filter. In some examples, the first reconstructed sample is generated by the constrained directional enhancement filter, and the intermediate reconstructed samples are generated by the loop recovery filter.

[0259] The process (3600) proceeds to (S3699) and ends.

[0260] It should be noted that in some examples, the filter based on the non-linear mapping is a cross-component sample offset (CCSO) filter, and in some other examples, the filter based on the non-linear mapping is a local sample offset (LSO) filter.

[0261] The process (3600) may be modified as appropriate. Steps in the process (3600) may be modified and / or omitted. One or more additional steps may be added. Any suitable order of implementation may be used.

[0262] The embodiments in this application may be used alone or in any order. Further, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer-readable medium.

[0263] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.37 A computer system (3700) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0264] Computer software may be encoded using any suitable machine code or computer language that may be assembled, compiled, linked or similarly structured to create code comprising instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through interpretation, microcode execution, etc.

[0265] These instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0266] Fig.37 The components for the computer system (3700) shown in the example are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of the computer software implementing the embodiments of the present application. The configuration of the components should not be interpreted as having any dependency or requirement on any one component or combination of components illustrated in the exemplary embodiment of the computer system (3700).

[0267] The computer system (3700) may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove motions), audio input (e.g., voice, tapping), visual input (e.g., gestures), olfactory input (not depicted). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to a person's conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0268] Input human-machine interface devices may include one or more of the following (only one of each is depicted): keyboard (3701), mouse (3702), touchpad (3703), touch screen (3710), data gloves (not shown), joystick (3705), microphone (3706), scanner (3707), camera (3708).

[0269] The computer system (3700) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (3710), a data glove (not shown), or a joystick (3705), although there may also be tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (3709), headphones (not depicted)), visual output devices (e.g., for screens (3710) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, some of which are capable of outputting two-dimensional visual output or more than three-dimensional output by means such as stereo output; virtual reality glasses (not depicted), holographic displays and smoke canisters (not depicted), and printers (not depicted).

[0270] The computer system (3700) may also include human-accessible storage devices and their associated media, such as optical media including media (3721) such as CD / DVD ROM / RW (3720) with CD / DVD, thumb drives (3722), removable hard drives or solid-state drives (3723), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD-based devices such as security software dogs (not depicted), and the like.

[0271] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other volatile signals.

[0272] The computer system (3700) may also include an interface (3754) to one or more communication networks (3755). The network may be, for example, wireless, wired, optical. The network may further be local, wide-area, metropolitan, vehicle-mounted, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks (e.g., Ethernet), wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide-area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicle and industrial networks (including CANBus), and the like. Certain networks typically require an external network interface adapter attached to certain common data ports or peripheral buses (3749) (e.g., a USB port of the computer system (3700)); other networks are typically integrated into the kernel of the computer system (3700) by attaching to a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smart phone computer system). Using any of these networks, the computer system (3700) can communicate with other entities. Such communication may be one-way, receive-only (e.g., broadcast TV), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way (e.g., to other computer systems using a local area digital network or a wide area digital network). Certain protocols and protocol stacks may be used on each of the networks and network interfaces such as those described above.

[0273] The above-mentioned human interface device, human-accessible storage device, and network interface may be attached to the kernel (3740) of the computer system (3700).

[0274] The kernel (3740) may include one or more central processing units (CPUs) (3741), graphics processing units (GPUs) (3742), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (3743), hardware accelerators for certain tasks (3744), graphics adapters (3750), etc. These devices, along with read-only memory (ROM) (3745), random access memory (3746), internal mass storage devices (3747) such as internal non-user accessible hard drives, SSDs, etc., may be connected via a system bus (3748). In some computer systems, the system bus (3748) may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the kernel's system bus (3748) or to the system bus (3748) via a peripheral bus (3749). In one example, a display (3710) may be connected to a graphics adapter (3750). The architecture of the peripheral bus includes PCI, USB, etc.

[0275] The CPU (3741), GPU (3742), FPGA (3743) and accelerator (3744) can execute certain instructions, and the combination of these instructions can constitute the aforementioned computer code. The computer code can be stored in ROM (3745) or RAM (3746). Transition data can also be stored in RAM (3746), while permanent data can be stored in, for example, an internal mass storage device (3747). Fast storage and retrieval of any memory device can be enabled by using a cache memory, which can be closely associated with one or more CPUs (3741), GPUs (3742), mass storage devices (3747), ROM (3745), RAM (3746), etc.

[0276] The computer readable medium may have computer codes for performing various computer-implemented operations. The media and computer codes may be those specially designed and constructed for the purposes of this application, or they may be of a type well known and available to those skilled in the art of computer software.

[0277] As an example and not by way of limitation, a computer system having the architecture (3700) and in particular the kernel (3740) may provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software implemented in one or more tangible computer-readable media. Such computer-readable media may be media associated with a user-accessible mass storage device as described above, as well as certain storage devices of the kernel (3740) having non-volatile properties (e.g., a kernel internal mass storage device (3747) or a ROM (3745)). Software implementing various embodiments of the present application may be stored in such a device and executed by the kernel (3740). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software may cause the kernel (3740) and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (3746) and modifying such data structures according to software-defined processes. In addition or as an alternative, the computer system may provide functionality as a result of logic hardwired or otherwise implemented in circuits (e.g., accelerators (3744)) that may replace software or operate in conjunction with software to perform specific processes or specific portions of specific processes described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits containing logic for execution, or both. The present application includes any suitable combination of hardware and software. Appendix A: Acronyms JEM: joint exploration model VVC: universal video coding BMS: benchmark set MV:motion vector HEVC: High Efficiency Video Coding MPM: Most Probable Mode WAIP: wide-angle intra prediction SEI: Supplementary Enhancement Information VUI:Video Usability Information GOP:Groups of Pictures TU:Transform Unit PU: Prediction Unit CTU: Coding Tree Unit CTB: Coding Tree Block PB: Prediction Block HRD: Hypothetical Reference Decoder SDR: Standard Dynamic Range SNR:Signal Noise Ratio CPU:Central Processing Unit GPU: Graphics Processing Unit CRT: Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM:Read-Only Memory RAM: Random Access Memory ASIC:Application-Specific Integrated Circuit PLD:Programmable Logic Device LAN: Local Area Network GSM:Global System for Mobile communicationGlobal System for Mobile Communication LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Array Field Programmable Gate Array SSD: solid-state drive IC:Integrated Circuit CU:Coding Unit PDPC: Position Dependent Prediction Combination ISP: intra sub-partition SPS: sequence parameter set

[0278] Although the present application has described a number of exemplary embodiments, various changes, permutations, and substitutions of the embodiments are within the scope of the present application. It should therefore be understood that those skilled in the art are capable of designing a variety of systems and methods that, although not explicitly shown or described herein, embody the principles of the present application and are therefore within the spirit and scope of the present application.

Claims

1. A method for filtering in video encoding / decoding, characterized in that: include: determining a first offset value using a cross-component sample offset (CCSO) filter on a first reconstructed luma sample at a first node along a loop filter chain including a plurality of video filters; and The first offset value is applied to reconstructed samples at a second node along the loop filter chain to generate second reconstructed samples, wherein the second node is located after an in-loop constrained directional enhancement filter (CDEF) in the loop filter chain.

2. The method according to claim 1, characterized in that The reconstructed samples and the first reconstructed luma samples have different color components.

3. The method according to claim 1, characterized in that The first node is located before a deblocking filter in the loop filter chain.

4. The method according to claim 1, characterized in that: The first node and the second node are the same node.

5. The method according to claim 4, characterized in that The first node and the second node are one of the following: an output node of the loop filter chain; or The intermediate node of the loop filter chain.

6. The method according to claim 1, characterized in that The first reconstructed luma sample is generated by at least one of: Processing module before deblocking filter; the deblocking filter; the constrained directional enhancement filter; or Loop recovery filter.

7. The method according to claim 1, characterized in that The first node and the second node are different nodes.

8. The method according to claim 7, characterized in that The first reconstructed luma samples are generated by a processing module before a deblocking filter, and the reconstructed samples are generated by at least one of: Constrained Directional Enhancement Filter; or Loop recovery filter.

9. The method according to claim 7, characterized in that: The first reconstructed luma samples are generated by a deblocking filter, and the reconstructed samples are generated by at least one of: Constrained Directional Enhancement Filter; or Loop recovery filter.

10. The method according to claim 7, characterized in that The first reconstructed luma samples are generated by a constrained directional enhancement filter and the reconstructed samples are generated by a loop restoration filter.

11. A method for processing a video code stream, characterized in that: The video code stream is filtered according to the method according to any one of claims 1 to 10.

12. A device for video encoding / decoding, characterized in that: The method comprises a processing circuit for performing the method according to any one of claims 1 to 10.

13. A storage medium storing computer instructions, characterized in that: When the instructions are executed by a computer device, the method according to any one of claims 1 to 10 is implemented.

Images