Cross-component sample offset optimization for early termination of optimal filter parameter search

Abstract

Processing circuitry determines to encode one or more blocks in a current picture using cross component sample offset (CCSO), and begins to search for combinations of values of filter parameters of the CCSO, each combination of values of filter parameters comprising a value for each filter parameter, respectively. The processing circuitry determines that a first rate distortion cost associated with at least a first combination of values satisfies a condition; in response to the first rate distortion cost satisfying a condition, terminating searching for at least a second combination of values of filter parameters of the CCSO; and selecting a value of a filter parameter for performing CCSO to encode one or more blocks in the current picture based on a value combination obtained from the search, the value combination obtained from the search not including a second combination of values.

Description

[0001] Incorporation by reference

[0002] This application claims the benefit of priority to U.S. Patent Application No. 18 / 367,979, filed on September 13, 2023, entitled “CROSS-COMPONENT SAMPLE OFFSET OPTIMIZATION BY EARLY TERMINATION OF OPTIMAL FILTER PARAMETER SEARCH,” which claims the benefit of priority to U.S. Provisional Application No. 63 / 447,868, filed on February 23, 2023, entitled “Cross-Component Sample Offset Optimization by Early Termination of Optimal Filter Parameter Search.” The disclosures of these prior applications are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure describes embodiments generally related to video encoding. Background Art

[0004] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. To the extent that the work of the presently named inventors described in this background section and in various aspects of this specification was performed, it does not indicate that it qualifies as prior art at the time of filing, and it is never explicitly or implicitly admitted that it is prior art to the present disclosure.

[0005] Image / video compression can help transfer image / video files between different devices, storage, and networks in a way that minimizes quality degradation. In some examples, video codec techniques can compress video based on spatial redundancy and temporal redundancy. In one example, a video codec can use a technique called intra-frame prediction, which can compress images based on spatial redundancy. For example, intra-frame prediction can use reference data from the current picture being reconstructed for sample prediction. In another example, a video codec can use a technique called inter-frame prediction, which can compress images based on temporal redundancy. For example, inter-frame prediction can predict samples in the current picture from a picture previously reconstructed using motion compensation. Motion compensation is typically indicated by a motion vector (MV). Summary of the invention

[0006] Aspects of the present disclosure provide methods and apparatus for video encoding / decoding. In some examples, an apparatus for video encoding includes a receiving circuit and a processing circuit. The processing circuit determines that one or more blocks in a current picture are encoded using a cross-component sample offset (CCSO), and begins searching for a combination of values ​​of filter parameters of the CCSO, each combination of values ​​of the filter parameters including values ​​for each filter parameter, respectively. The processing circuit determines that a first rate-distortion cost associated with at least a first combination of values ​​satisfies a condition; in response to the first rate-distortion cost satisfying the condition, terminates the search for at least a second combination of values ​​of the filter parameters of the CCSO; and based on the combination of values ​​obtained from the search, selects the value of the filter parameter for performing the CCSO to encode one or more blocks in the current picture, the combination of values ​​obtained from the search does not include the second combination of values.

[0007] In some embodiments, the filter parameters include at least one of a filter shape, a number of filter taps, a quantization step size, a filter unit size, and a number of frequency bands.

[0008] To determine that the first rate-distortion cost satisfies the condition, in some examples, the processing circuit determines that the first rate-distortion cost is greater than a first threshold cost, where the first threshold cost is a product of a rate-distortion cost of samples not filtered by the CCSO and a first threshold.

[0009] In one example, a first combination of values ​​includes a first number of bands, a first filter shape, and a first quantization step size, and at least a second combination each includes a second number of bands different from the first number of bands, a first filter shape, and a first quantization step size.

[0010] In another example, the first combination includes a first subset combination of a first quantization step size and a value, the first subset combination including a first number of frequency bands and a first filter shape. Each of at least the second combinations includes a second subset combination of the first quantization step size and a value, the second subset combination including at least one of a second number of frequency bands different from the first number of frequency bands and / or a second number of filter shapes different from the first filter shape.

[0011] In some examples, to determine that the first rate-distortion cost satisfies the condition, the processing circuit determines that, for the first subset of filter parameters, a particular combination of values ​​of the second subset of filter parameters achieves a lowest rate-distortion cost for one or more combinations of values ​​of the first subset of filter parameters.

[0012] In some examples, at least a second combination of values ​​includes a combination of values ​​of a second subset of the filter parameters, the combination of values ​​of the second subset of the filter parameters being different from the particular combination of values ​​of the second subset.

[0013] In one example, the second subset of filter parameters includes the number of frequency bands, a specific combination of values ​​of the second subset includes a specific value of the number of frequency bands, and at least a second combination of values ​​includes a value of the number of frequency bands that is different from the specific value of the number of frequency bands.

[0014] In another example, a second subset of filter parameters includes the number of frequency bands and a quantization step size, a specific combination of values ​​of the second subset includes a first specific value for the number of frequency bands and a second specific value for the quantization step size, and at least a second combination of values ​​includes a first value for the number of frequency bands and a second value for the quantization step size, the first value being different from the first specific value for the number of frequency bands and the second value being different from the second specific value for the quantization step size.

[0015] In some examples, to determine that the first rate-distortion cost satisfies the condition, the processing circuit determines that an evaluation of one or more combinations of values ​​of the first subset of filter parameters and a particular combination of values ​​of the second subset of filter parameters has a higher rate-distortion cost than a lowest rate-distortion cost obtained from a combination of values ​​evaluated prior to the evaluation.

[0016] In one example, at least the second combination of values ​​comprises a combination of values ​​of a first subset of filter parameters, the combination of values ​​of the first subset of filter parameters being different from one or more combinations of values ​​of the first subset of filter parameters. For example, the first subset of filter parameters comprises a quantization step size, and the second subset of filter parameters comprises a filter shape.

[0017] In some examples, to determine that a first rate-distortion cost associated with at least a first combination of values ​​satisfies a condition, for the first combination of values ​​of the filter parameters, the processing circuit calculates a cumulative rate-distortion cost for a first plurality of blocks in one or more blocks, the one or more blocks comprising the first plurality of blocks and the one or more additional blocks; compares the cumulative rate-distortion cost to a lowest rate-distortion cost of a combination previously evaluated for the first combination; and in response to the cumulative rate-distortion cost being higher than the lowest rate-distortion cost, skips rate-distortion cost calculation for the one or more additional blocks.

[0018] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions, which, when executed by a computer for video encoding, cause the computer to perform a method for video encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Further features, properties and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which:

[0020] Figure 1 is a schematic illustration of an exemplary block diagram of a communication system (100).

[0021] Figure 2is a schematic illustration of an exemplary block diagram of a decoder.

[0022] Figure 3 is a schematic illustration of an exemplary block diagram of an encoder.

[0023] Figure 4 Diagram showing filter shapes in some examples.

[0024] FIG. 5A to FIG. 5D Examples of sub-sampling positions in some examples are shown.

[0025] Figure 6 A cross-component filter for generating chroma components according to an embodiment of the present disclosure is shown.

[0026] Figure 7 An example of a filter shape according to an embodiment of the present disclosure is shown.

[0027] Figure 8 Syntax examples according to some embodiments of the present disclosure are shown.

[0028] Fig. 9 A table showing multiple sample adaptive offset (SAO) types according to an embodiment of the present disclosure is shown.

[0029] Fig.10 Examples of 3-pixel patterns used for classification of pixels in edge offset type in some examples are shown.

[0030] Fig.11 A table showing pixel classification rules for edge migration in some examples.

[0031] Fig.12 Examples of filter support regions according to some embodiments of the present disclosure are shown.

[0032] Fig.13 A table with 9 combinations according to an embodiment of the present disclosure is shown.

[0033] Fig.14 A diagram of the filter support region illustrating switchable filter shapes in some examples is shown.

[0034] Fig.15 A table listing exemplary rate-distortion costs for various exemplary combinations of filter support and quantization step size is shown.

[0035] Fig.16 A flowchart outlining a process according to some embodiments of the present disclosure is shown.

[0036] Fig.17 is a schematic illustration of a computer system according to an embodiment. DETAILED DESCRIPTION

[0037] Figure 1 A block diagram of a video processing system (100) in some examples is shown. The video processing system (100) is an example of an application of the disclosed subject matter, being a video encoder and a video decoder in a streaming environment. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, streaming services, storing compressed video on digital media including CDs, DVDs, memory sticks, etc., and the like.

[0038] The video processing system (100) includes an acquisition subsystem (113), which may include a video source (101), such as a digital camera, which creates, for example, an uncompressed video picture stream (102). In one example, the video picture stream (102) includes samples recorded by the digital camera. The video picture stream (102), which is depicted as a thick line to emphasize the high amount of data compared to the encoded video data (104) (or the encoded video bitstream), can be processed by an electronic device (120), which includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. The encoded video data (104) (or the encoded video bitstream), which is depicted as a thin line to emphasize the lower amount of data compared to the video picture stream (102), can be stored on a streaming server (105) for future use. One or more streaming client subsystems, such as Figure 1 A client subsystem (106) and a client subsystem (108) in a streaming server (105) may access a copy (107) and a copy (109) of the encoded video data (104). The client subsystem (106) may include, for example, a video decoder (110) in an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and creates an output video picture stream (111) that can be presented on a display (112) (e.g., a display screen) or other presentation device (not depicted). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., a video bitstream) may be encoded according to certain video encoding / compression standards. Examples of such standards include ITU-T Recommendation H.265. In one example, the video coding standard under development is informally referred to as Next Generation Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0039] It should be noted that the electronic device (120) and the electronic device (130) may include other components (not shown). For example, the electronic device (120) may include a video decoder (not shown), and the electronic device (130) may also include a video encoder (not shown). It should be noted that in some examples, pictures in a video sequence may be referred to as frames.

[0040] Figure 2 An exemplary block diagram of a video decoder (210) is shown. The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., a receiving circuit). The video decoder (210) may be used to replace Figure 1 A video decoder (110) in an example of FIG.

[0041] The receiver (231) may receive one or more encoded video sequences to be decoded by the video decoder (210). In one embodiment, the encoded video sequences are received one at a time, wherein the decoding of each encoded video sequence is independent of the decoding of the other encoded video sequences. The encoded video sequence may be received from a channel (201), which may be a hardware / software link leading to a storage device storing the encoded video data. The receiver (231) may receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective consuming entities (not depicted). The receiver (231) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter referred to as "parser (220)"). In some applications, the buffer memory (215) is part of the video decoder (210). In other applications, the buffer memory (215) may be located external to the video decoder (210) (not depicted). In some other applications, a buffer memory (not depicted) may be provided outside the video decoder (210) to, for example, prevent network jitter, and another buffer memory (215) may be provided inside the video decoder (210) to, for example, handle playback timing. When the receiver (231) receives data from a storage / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (215) may not be required, or the buffer memory (215) may be made smaller. For use on a traffic packet network such as the Internet, the buffer memory (215) may be required, and the buffer memory (215) may be relatively large, advantageously may have an adaptive size, and may be implemented at least partially in an operating system or similar element (not depicted) external to the video decoder (210).

[0042] The video decoder (210) may include a parser (220) to reconstruct symbols (221) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (210) and potential information for controlling a presentation device such as a presentation device (212) (e.g., a display screen) that is not an integral part of the electronic device (230) but can be coupled to the electronic device (230), such as Figure 2 As shown. The control information for the rendering device may be in the form of an auxiliary enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not depicted). The parser (220) may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (220) may extract a subgroup parameter set for at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (220) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the coded video sequence.

[0043] The parser (220) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to create symbols (221).

[0044] Depending on the type of coded video picture or part of coded video picture (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of symbol (221) may involve multiple different units. Which units are involved and how they are involved can be controlled by parser (220) through subgroup control information parsed from the coded video sequence. For clarity, such subgroup control information flow between parser (220) and the multiple units below is not depicted.

[0045] In addition to the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, the conceptual subdivision into a number of functional units is appropriate in the following.

[0046] The first unit is a sealer / inverse transform unit (251). The sealer / inverse transform unit (251) receives quantized transform coefficients as symbols (221) from the parser (220) and control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (251) may output a block including sample values, which may be input into an aggregator (255).

[0047] In some cases, the output samples of the scaler / inverse transform unit (251) may belong to an intra-coded block. An intra-coded block is a block that does not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of a current picture. Such prediction information may be provided by an intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) uses surrounding reconstructed information extracted from a current picture buffer (258) to generate a block of the same size and shape as the block being reconstructed. For example, the current picture buffer (258) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (255) adds the prediction information generated by the intra-prediction unit (252) to the output sample information provided by the scaler / inverse transform unit (251) on a per-sample basis.

[0048] In other cases, the output samples of the scaler / inverse transform unit (251) may belong to a block that is inter-coded and potentially motion compensated. In this case, the motion compensated prediction unit (253) may access the reference picture memory (257) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (221) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (251) (in this case, referred to as residual samples or residual signals) by the aggregator (255) to generate output sample information. The extraction of prediction samples by the motion compensated prediction unit (253) from the address in the reference picture memory (257) may be controlled by a motion vector, which may be provided to the motion compensated prediction unit (253) in the form of symbols (221), which may have, for example, an X component, a Y component and a reference picture component. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory (257) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0049] The output samples of the aggregator (255) may be subjected to various loop filtering techniques in a loop filter unit (256). The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and available to the loop filter unit (256) as symbols (221) from the parser (220). The video compression may also be responsive to meta-information obtained during decoding of a coded picture or a previous (in decoding order) portion of the coded video sequence, and to previously reconstructed and loop filtered sample values.

[0050] The output of the loop filter unit (256) may be a sample stream that may be output to a rendering device (212) and stored in a reference picture memory (257) for future inter-picture prediction.

[0051] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) may become part of the reference picture memory (257), and a new current picture buffer may be reallocated before starting to reconstruct a subsequent coded picture.

[0052] The video decoder (210) may perform decoding operations according to a predetermined video compression technology or standard, such as ITU-T H.265 Recommendation. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used, in the sense that the encoded video sequence follows the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence may also be required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further limited by the hypothetical reference decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.

[0053] In one embodiment, the receiver (231) may receive additional (redundant) data when receiving the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0054] Figure 3 An exemplary block diagram of a video encoder (303) is shown. The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit). The video encoder (303) can be used to replace Figure 1 A video encoder (103) in an example of FIG.

[0055] The video encoder (303) can be used to obtain the video source (301) (not Figure 3 In an example, the video source (301) is a part of the electronic device (320) to receive video samples, and the video source (301) can capture video images to be encoded by the video encoder (303). In another example, the video source (301) is a part of the electronic device (320).

[0056] The video source (301) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (303), the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sample structure (e.g., Y CrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (301) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that are given motion when viewed sequentially. The pictures themselves may be organized into a spatial pixel array, where each pixel may include one or more samples depending on the sample structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on the samples.

[0057] According to one embodiment, the video encoder (303) may encode and compress pictures of a source video sequence into an encoded video sequence (343) in real time or under any other time constraints as required. Implementing an appropriate encoding speed is a function of the controller (350). In some embodiments, the controller (350) controls other functional units as described below and is functionally coupled to the other functional units. For clarity, the coupling is not depicted in the figure. The parameters set by the controller (350) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization techniques, etc.), picture size, picture group (GOP) layout, maximum motion vector search range, etc. The controller (350) may be configured to have other suitable functions that are related to the video encoder (303) optimized for a certain system design.

[0058] In some embodiments, the video encoder (303) is configured to operate in an encoding loop. As an oversimplified description, in one example, the encoding loop may include a source encoder (330) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder may also create sample data. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference picture memory (334) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, such as due to channel errors) is also used in some related technologies.

[0059] The operation of the "local" decoder (333) may be similar to that described above in conjunction with Figure 2 The "remote" decoder of the video decoder (210) described in detail is identical. However, additional brief reference is made to Figure 2 , since the symbols are available and the entropy encoder (345) and the parser (220) can losslessly encode / decode the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (210) including the buffer memory (215) and the parser (220) may not be fully implemented in the local decoder (333).

[0060] In one embodiment, decoder techniques other than parsing / entropy decoding present in a decoder are present in a corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on decoder operation. The description of encoder techniques can be simplified because encoder techniques are mutually inverse to the decoder techniques described comprehensively. In some areas, a more detailed description is provided below.

[0061] During operation, in some examples, the source encoder (330) may perform motion compensated predictive coding that predictively encodes an input picture by referencing one or more previously encoded pictures from a video sequence designated as "reference pictures." In this manner, the encoding engine (332) encodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture.

[0062] The local video decoder (333) may decode the encoded video data of the picture that may be designated as the reference picture based on the symbol created by the source encoder (330). The operation of the encoding engine (332) may advantageously be a lossy process. When the encoded video data may be decoded at the video decoder ( Figure 3 When the video encoder (303) is decoded at a remote video decoder (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (333) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture memory (334). In this way, the video encoder (303) may store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.

[0063] The predictor (335) may perform a prediction search for the encoding engine (332). That is, for a new picture to be encoded, the predictor (335) may search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may be used as appropriate prediction references for the new picture. The predictor (335) may operate on a pixel block by pixel block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (334).

[0064] The controller (350) may manage encoding operations of the source encoder (330), including, for example, setting parameters and subgroup parameters for encoding video data.

[0065] The outputs of all the above functional units may be entropy encoded in an entropy encoder (345). The entropy encoder (345) applies lossless compression to the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0066] The transmitter (340) may buffer the encoded video sequence created by the entropy encoder (345) in preparation for transmission over a communication channel (360), which may be a hardware / software link to a storage device that may store the encoded video data. The transmitter (340) may combine the encoded video data from the video encoder (303) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0067] The controller (350) may manage the operation of the video encoder (303). During encoding, the controller (350) may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types:

[0068] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow for different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of those variations of I pictures and their corresponding applications and features.

[0069] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0070] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

[0071] The source picture may typically be spatially subdivided into blocks of samples (e.g. blocks of 4×4, 8×8, 4×8 or 16×16 samples) and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined by the coding allocation applied to the corresponding picture of the block. For example, blocks of an I picture may be non-predictively coded, or blocks of an I picture may be predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0072] The video encoder (303) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (303) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0073] In one embodiment, the transmitter (340) may transmit additional data when transmitting the encoded video. The source encoder (330) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0074] The captured video may be taken as a plurality of source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often shortened to intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In one example, a particular picture being encoded / decoded is partitioned into blocks, and the particular picture being encoded / decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a reference picture that was previously encoded in the video and is still buffered, the block in the current picture may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case where multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0075] In some embodiments, a bidirectional prediction technique may be used for inter-picture prediction. According to the bidirectional prediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture that precede the current picture in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block may be predicted by a combination of the first reference block and the second reference block.

[0076] In addition, merge mode technology can be used for inter-picture prediction to improve coding efficiency.

[0077] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, a picture in a video picture sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Each CTU can be recursively divided into one or more coding units (CUs) in a quadtree. For example, a 64×64 pixel CTU can be divided into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In one example, each CU is analyzed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. The CU is divided into one or more prediction units (PUs) based on temporal and / or spatial predictability. Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In one embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luma values) for pixels, such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0078] It should be noted that the video encoder (103) and the video encoder (303) and the video decoder (110) and the video decoder (210) may be implemented using any suitable technology. In one embodiment, the video encoder (103) and the video encoder (303) and the video decoder (110) and the video decoder (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoder (103) and the video encoder (303) and the video decoder (110) and the video decoder (210) may be implemented using one or more processors that execute software instructions.

[0079] Various aspects of the present disclosure provide techniques for early termination of the search for optimal filter parameters in cross-component sample offsets (CCSOs), including skipping the search for certain combinations of filter parameters. The techniques can be used with filtering techniques in various video standards, such as the Alliance for Open Media Video 1 (AV1) standard, the High Efficiency Video Coding (HEVC) standard, the Next Video Coding (VVC) standard, etc.

[0080] According to some aspects of the present disclosure, filtering techniques are used to implement a loop filter, such as loop filter (256), in a decoder and / or encoder. In some examples (e.g., VVC), the encoder / decoder may apply an adaptive loop filter (ALF) with block-based filter adaptation to reduce artifacts. For the luma component, one of a plurality of filters (e.g., 25 filters) may be selected for a 4×4 luma block, for example, based on the direction and activity of a local gradient. In some examples, each filter includes a set of filter coefficients, and the plurality of filters may correspond to different sets of filter coefficients.

[0081] The ALF may have any suitable shape and size.

[0082] Figure 4 Diagram showing filter shapes in some examples. Figure 4 A first diamond filter shape (410) and a second diamond filter shape (420) are shown. The first diamond filter (410) is referred to as a 5×5 diamond and the second diamond filter shape (420) is referred to as a 7×7 diamond. The first diamond filter shape (410) includes seven values ​​(shown by C0 to C6) assigned to positions in the 5×5 diamond as filter coefficients for the corresponding positions. The second diamond filter shape (420) includes thirteen values ​​(shown by C0 to C12) assigned to positions in the 7×7 diamond as filter coefficients for the corresponding positions. In one example, the first diamond filter shape (410) is applied to chrominance components and the second diamond filter shape (420) is applied to luma components. It should be noted that other suitable filter shapes may be used for the ALF.

[0083] According to one aspect of the present disclosure, block classification may be applied to classify each 4×4 block into a category for filter selection. In some examples, for block classification of luma components, a 4×4 block (or luma block, luma CB) may be classified or categorized into one of a plurality of (e.g., 25) categories. The quantized value of the directivity parameter D and the activity value A may be, for example, according to equation (1). To derive the category index C:

[0084]

[0085] In some examples, to calculate the directivity parameter D and the quantized value For example, according to equations (2) to (5), one-dimensional (1D) Laplace can be used to calculate the corresponding gradients g in the vertical direction, horizontal direction, and two diagonal directions (e.g., d1 and d2), respectively. v , g h , g d1 , g d2 .

[0086]

[0087]

[0088]

[0089]

[0090] Here, the indices i and j refer to the coordinates of the top left sample within the 4×4 block, and R(k, l) indicates the reconstructed sample at coordinates (k, l).

[0091] In order to reduce the complexity of the above block classification, a sub-sample 1D Laplacian calculation may be applied.

[0092] FIG. 5A to FIG. 5D Examples of subsample locations in some examples are shown. FIG. 5A to FIG. 5D In , the same subsample position can be used for gradient calculation in different directions. Figure 5A The sub-sample positions (marked by 'V') used to calculate the vertical gradient gv are shown. Figure 5B The sub-sample positions (marked by 'H') used to calculate the horizontal gradient gh are shown. Figure 5C The sub-sample positions (labeled by 'D1') used to calculate the d1 diagonal gradient gd1 are shown. Figure 5D The sub-sample positions (labeled by 'D2') used to calculate the d2 diagonal gradient gd2 are shown.

[0093] In some examples, the horizontal and vertical gradients g v and g h The maximum value of and minimum value It can be set according to equation (6):

[0094]

[0095] Two diagonal directions g d1 and g d2 The maximum value of the gradient and minimum value It can be set according to equation (7):

[0096]

[0097] In some examples, the directivity parameter D may be derived based on the above value and two thresholds t1 and t2, for example, according to the following steps:

[0098] Step 1: If (1) and (2) is true, D is set to 0.

[0099] Step 2: If Then proceed to step 3; otherwise proceed to step 4.

[0100] Step 3: If Then D is set to 2; otherwise D is set to 1.

[0101] Step 4: If Then D is set to 4; otherwise D is set to 3.

[0102] In some examples, the activity value A may be calculated according to equation (8):

[0103]

[0104] A can be further quantized to a range of 0 to 4 (including the endpoints), and the quantized value is expressed as

[0105] In some examples, for chroma components in a picture, block classification is not applied, so a single set of ALF coefficients may be applied for each chroma component.

[0106] The geometric transformation may be applied to the filter coefficients and the corresponding filter clipping values ​​(also referred to as clipping values). In some examples, before filtering a block (e.g., a 4×4 luma block), the filter coefficients may be filtered based on, for example, a gradient value (e.g., g v , g h , g d1 and / or d2 ), a geometric transformation such as rotation or diagonal flipping and vertical flipping is applied to the filter coefficients f(k, l) and the corresponding filter clipping values ​​c(k, l). The geometric transformation applied to the filter coefficients f(k, l) and the corresponding filter clipping values ​​c(k, l) may be equivalent to applying the geometric transformation to samples in the area supported by the filter. The geometric transformation may make different blocks to which the ALF is applied more similar by aligning their respective directionality.

[0107] In some examples, three geometric transformations may be performed, including diagonal flipping, vertical flipping, and rotation, as described in equations (9) to (11), respectively:

[0108] f D(k,l)=f(l,k),c D (k,l)=c(l,k) Equation (9)

[0109] f v (k,l)=f(k,Kl-1),c V (k,l)=c(k,Kl-1) Equation (10)

[0110] f R (k,l)=f(Kl-1,k),c R (k,l)=c(Kl-1,k) Equation (11)

[0111] Where K is the size of the ALF or filter, and 0≤k, l≤K-1 are the coordinates of the coefficients. For example, position (0, 0) is located at the upper left corner of the filter f or the clipping value matrix (or clipping matrix) c, and position (K-1, K-1) is located at the lower right corner of the filter f or the clipping value matrix (or clipping matrix) c. The transform can be applied to the filter coefficients f(k, l) and the clipping values ​​c(k, l) based on the gradient values ​​calculated for the block. Table 1 summarizes an example of the relationship between the transform and the four gradients.

[0112] Table 1: Mapping of gradients and transformations computed to blocks

[0113] Gradient Value Transform <![CDATA[g d2 <g d1 And g h <g v ]]> No transformation <![CDATA[g d2 <g d1 And g v <g h ]]> Flip diagonally <![CDATA[g d1 <g d2 And g h <g v ]]> Flip vertically <![CDATA[g d1 <g d2 And g v <g h ]]> Rotation

[0114] In some embodiments, the ALF filter parameters are signaled in an adaptive parameter set (APS) of a picture. In the APS, a set of one or more (e.g., up to 25) luminance filter coefficients and a clipping value index may be signaled. In one example, one of the one or more sets may include a luminance filter coefficient and one or more clipping value indexes. A set of one or more (e.g., up to 8) luminance filter coefficients and a clipping value index may be signaled. In order to reduce signaling overhead, filter coefficients of different classifications of luminance components (e.g., with different classification indices) may be merged. In a slice header, an index of the APS for the current slice may be signaled. In some examples, the signaling of the ALF is based on the CTU.

[0115] In one embodiment, a clipping value index (also referred to as a clipping index) may be decoded from the APS. The clipping value index may be used to determine the corresponding clipping value, for example, based on the relationship between the clipping value index and the corresponding clipping value. The relationship may be predefined and stored in the decoder. In one example, the relationship is described by a table, such as a brightness table of the clipping value index and the corresponding clipping value (e.g., for a luma CB), a chrominance table of the clipping value index and the corresponding clipping value (e.g., for a chrominance CB). The clipping value may depend on the bit depth B. The bit depth B may refer to an internal bit depth, a bit depth of a reconstructed sample in a CB to be filtered, etc. In some examples, the table (e.g., a brightness table, a chrominance table) is obtained using equation (12).

[0116]

[0117] Where AlfClip is the clip value, B is the bit depth (e.g., bitDepth), N (e.g., N=4) is the number of allowed clip values, and (n-1) is the clip value index (also referred to as clip index or clipIdx). Table 2 shows an example of a table obtained using equation (12) when N=4. In Table 2, the clip index (n-1) can be 0, 1, 2, 3, and n can be 1, 2, 3, 4. Table 2 can be used for luminance blocks or chrominance blocks.

[0118] Table 2 - AlfClip may depend on bit depth B and clipIdx

[0119]

[0120]

[0121] In the slice header for the current slice, one or more APS indexes (e.g., up to 7 APS indexes) may be signaled to specify a luma filter set that may be used for the current slice. The filtering process may be controlled at one or more suitable levels (e.g., picture level, slice level, CTB level, etc.). In one embodiment, the filtering process may be further controlled at the CTB level. A flag may be signaled to indicate whether the ALF is applied to the luma CTB. The luma CTB may select a filter set from a plurality of fixed filter sets (e.g., 16 fixed filter sets) and a filter set signaled in the APS (also referred to as a signaled filter set). A filter set index may be signaled for the luma CTB to indicate a filter set to be applied (e.g., a filter set in a plurality of fixed filter sets and a filter set in the signaled filter set). A plurality of fixed filter sets may be predefined and hard-coded in the encoder and decoder, and may be referred to as a predefined filter set.

[0122] For chroma components, an APS index may be signaled in the slice header to indicate the chroma filter set to be used for the current slice. At the CTB level, if more than one chroma filter set is present in the APS, the filter set index may be signaled for each chroma CTB.

[0123] The filter coefficients may be quantized using a norm equal to 128. To reduce multiplication complexity, code stream consistency may be applied so that coefficient values ​​at non-center locations may be in the range of -27 to 27-1 (inclusive). In one example, the center position coefficient is not signaled in the code stream and may be considered equal to 128.

[0124] In some embodiments, the syntax and semantics of the clip index and clip value are defined as follows:

[0125] alf_luma_clip_idx[sfIdx][j] may be used to specify the clip index of the clip value to be used before multiplying the jth coefficient of the signaled luma filter indicated by sfIdx. Codestream conformance requirements may include that the value of alf_luma_clip_idx[sfIdx][j] should be in the range of 0 to 3 (inclusive), where sfIdx = 0 to alf_luma_num_filters_signalled_minus1, and j = 0 to 11.

[0126] The luma filter clipping value AlfClipL[adaptation_parameter_set_id] with elements AlfClipL[adaptation_parameter_set_id][filtIdx][j] may be derived from bitDepth set equal to BitDepthY and clipIdx set equal to alf_luma_clip_idx[alf_luma_coeff_delta_idx[filtIdx][j] as specified in Table 2, where filtIdx=0 to NumAlfFilters-1, j=0 to 11.

[0127] alf_chroma_clip_idx[altIdx][j] may be used to specify the clip index of the clip value to be used before multiplying the j-th coefficient of the alternative chroma filter with index altIdx. Codestream conformance requirements may include that the value of alf_chroma_clip_idx[altIdx][j] should be in the range of 0 to 3 (inclusive), where alf_chroma_num_alt_filters_minus1, j=0 to 5.

[0128] The chroma filter clipping value AlfClipC[adaptation_parameter_set_id] with elements AlfClipC[adaptation_parameter_set_id][altIdx][j] may be derived from bitDepth set equal to BitDepthC and clipIdx set equal to alf_chroma_clip_idx[altIdx][j] as specified in Table 2, where altIdx=0 to alf_chroma_num_alt_filters_minus1, j=0 to 5.

[0129] In one embodiment, the filtering process can be described as follows. At the decoder side, when ALF is enabled for CTB, the sample R(i, j) within the CU (or CB) can be filtered to obtain the filtered sample value R'(i, j), as shown below using equation (13). In one example, each sample in the CU is filtered.

[0130] R′(i,j)=R(i,j)+((∑ k≠0 ∑ l≠0 f(k,l)×K(R(i+k,j+l)-R(i,j),c(k,l))+64)>>7) Equation (13)

[0131] Where f(k, l) represents the decoded filter coefficients, K(x, y) is the clipping function, and c(k, l) represents the decoded clipping parameter (or clipping value). The variables k and l can vary between -L / 2 and L / 2, where L represents the filter length. The clipping function K(x, y) = min(y, max(-y, x)), corresponding to the clipping function Clip3(-y, y, x). By including the clipping function K(x, y), the loop filtering method (e.g., ALF) becomes a nonlinear process and can be called a nonlinear ALF.

[0132] In a nonlinear ALF, multiple clip value sets may be used. In one example, the luma set includes four clip values ​​{1024, 181, 32, 6} and the chroma set includes four clip values ​​{1024, 161, 25, 4}. The four clip values ​​in the luma set may be selected by dividing the full range (e.g., 1024) of the sample values ​​of the luma block (encoded in 10 bits) approximately equally in the logarithmic domain. The chroma set may range from 4 to 1024.

[0133] The selected clipping value may be encoded in the "alf_data" syntax element as follows: The clipping index corresponding to the selected clipping value may be encoded using a suitable encoding scheme (eg, a Golomb encoding scheme). The encoding scheme may be the same encoding scheme used to encode the filter set index.

[0134] In some examples, the cross-component filtering process can apply a cross-component filter, such as a cross-component adaptive loop filter (CC-ALF), in a loop filter such as loop filter (256) in a decoder and an encoder. The cross-component filter can use luma sample values ​​of a luma component (e.g., a luma CB) to refine a chroma component (e.g., a chroma CB corresponding to the luma CB). In one example, the luma CB and the chroma CB are collocated in a CU.

[0135] Figure 6 A cross-component filter (e.g., CC-ALF) for generating chrominance components according to an embodiment of the present disclosure is shown. In some examples, Figure 6 The filtering process for a first chroma component (e.g., a first chroma CB), a second chroma component (e.g., a second chroma CB), and a luma component (e.g., a luma CB) is shown. The luma component may be filtered by a sample adaptive offset (SAO) filter (610) to generate an SAO filtered luma component (641). The SAO filtered luma component (641) may be further filtered by an ALF luma filter (616) to become a filtered luma CB (661) (e.g., 'Y').

[0136] The first chroma component may be filtered by an SAO filter (612) and an ALF chroma filter (618) to generate a first intermediate component (652). In addition, the SAO filtered luma component (641) may be filtered by a cross-component filter (e.g., CC-ALF) (621) for the first chroma component to generate a second intermediate component (642). Subsequently, a filtered first chroma component (662) (e.g., 'Cb') may be generated based on at least one of the second intermediate component (642) and the first intermediate component (652). In one example, the filtered first chroma component (662) (e.g., 'Cb') may be generated by combining the second intermediate component (642) and the first intermediate component (652) with an adder (622). The cross-component adaptive loop filtering process for the first chroma component may include steps performed by the CC-ALF (621) and steps performed by, for example, the adder (622).

[0137] The above description may be applicable to the second chroma component. The second chroma component may be filtered by the SAO filter (614) and the ALF chroma filter (618) to generate a third intermediate component (653). In addition, the SAO filtered luma component (641) may be filtered by a cross-component filter (e.g., CC-ALF) (631) for the second chroma component to generate a fourth intermediate component (643). Subsequently, a filtered second chroma component (663) (e.g., 'Cr') may be generated based on at least one of the fourth intermediate component (643) and the third intermediate component (653). In one example, the filtered second chroma component (663) (e.g., 'Cr') may be generated by combining the fourth intermediate component (643) and the third intermediate component (653) with an adder (632). In one example, the cross-component adaptive loop filtering process for the second chroma component may include steps performed by the CC-ALF (631) and steps performed by, for example, the adder (632).

[0138] The cross-component filter (e.g., CC-ALF (621), CC-ALF (631)) may operate by applying a linear filter having any suitable filter shape to the luma component (or luma channel) to refine each chroma component (e.g., a first chroma component, a second chroma component).

[0139] Figure 7 An example of a filter shape (700) according to an embodiment of the present disclosure is shown. The filter shape (700) has a diamond shape, and each black dot indicates the position of an assigned filter coefficient. The filter coefficients of the filter shape (700) may include non-zero filter coefficients and zero filter coefficients.

[0140] CC-ALF may include any suitable filter coefficients (also referred to as CC-ALF filter coefficients). Figure 6 , CC-ALF (621) and CC-ALF (631) may have the same filter shape (eg, Figure 7 In one example, the values ​​of the filter coefficients in CC-ALF (621) are different from the values ​​of the filter coefficients in CC-ALF (631).

[0141] Typically, the filter coefficients in CC-ALF can be transmitted, for example, in APS. In one example, the filter coefficients can be multiplied by a factor (e.g., 2 10) and may be rounded for a fixed point representation. The application of CC-ALF may be controlled on a variable block size and signaled by a context coded flag (e.g., a CC-ALF enable flag) received for each sample block. The context coded flag (e.g., a CC-ALF enable flag) may be signaled at any suitable level (e.g., a block level). For each chroma component, the block size and the CC-ALF enable flag may be received together at the slice level. In some examples, block sizes 16×16, 32×32, 64×64 may be supported (in chroma samples).

[0142] Figure 8 FIG. 2 shows an example of CC-ALF syntax according to some embodiments of the present disclosure. Figure 8 In the example of , alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is an index indicating whether a cross-component Cb filter is used, and if used, the index of the cross-component Cb filter. For example, when alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is equal to 0, the cross-component Cb filter is not applied to the Cb color component sample block at the luma position (xCtb, yCtb); when alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is not equal to 0, alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is the index of the filter to be applied. For example, the alf_ctb_cross_component_cb_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]th cross-component Cb filter is applied to the block of Cb color component samples at luma position (xCtb, yCtb).

[0143] In addition, Figure 8In the example of , alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is used to indicate whether to use a cross-component Cr filter, and the index of whether to use the cross-component Cr filter. For example, when alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is equal to 0, the cross-component Cr filter is not applied to the Cr color component sample block at the luma position (xCtb, yCtb); when alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is not equal to 0, alf_ctb_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY] is the index of the cross-component Cr filter. For example, the alf_cross_component_cr_idc[xCtb>>CtbLog2SizeY][yCtb>>CtbLog2SizeY]th cross-component Cr filter may be applied to the Cr color component sample block at luma location (xCtb, yCtb).

[0144] In some examples (e.g., HEVC), a filtering technique called sample adaptive offset (SAO) may be used. In some examples, SAO is applied to the reconstructed signal after the deblocking filter. The deblocking filter may be used for a loop filter or a post filter. In some examples, a loop filter such as a loop filter (256) is located in the encoding loop, while the post filter is located outside the encoding loop (e.g., operating on a display buffer). SAO may use an offset value given in a slice header. In some examples, for luma samples, the encoder may decide whether to apply (enable) SAO on the slice. When SAO is enabled, the current picture allows the coding unit to be recursively divided into four sub-regions, and each sub-region may select an SAO type from a plurality of SAO types based on features in the sub-region. In some examples, the sub-region corresponds to a CTB. In some examples, the sub-region may be a block smaller than the CTB. In some examples, the features include band features and edge features. Band features are detected based on pixel intensity. Edge features are detected based on edge pattern directions in the sub-region. In some examples, the feature of the sub-region is to classify the sub-region as one of the SAO types. For example, edge attributes may be used to classify sub-regions into four SAO types, and pixel intensity may be used to classify pixels into two SAO types. Each SAO type may include multiple categories corresponding to respective offsets.

[0145] In some examples, for each sub-region, the encoder may try each SAO type and determine the best SAO type based on rate-distortion performance. Classification of sample classes in the sub-region is performed by the encoder and the decoder. Thus, in some examples, the encoder transmits the best SAO type and an offset of the class associated with the best SAO type for each sub-region to the decoder.

[0146] Fig. 9 A table (900) of multiple SAO types according to an embodiment of the present disclosure is shown. In the table (900), SAO type 0 to SAO type 6 are shown. It should be noted that SAO type 0 is used to indicate that SAO is not applied. In addition, each SAO type of SAO type 1 to SAO type 6 includes multiple categories. SAO can classify the reconstructed pixels of a sub-region into categories and reduce distortion by adding an offset to the pixels of each category in the sub-region.

[0147] Specifically, in one embodiment, for SAO type 5 and SAO type 6, which are also referred to as band offset (BO) types in some examples, pixels of a sub-region (e.g., CTU) are classified into multiple bands according to pixel intensity, and corresponding band offsets can be applied to the multiple bands. Each of the multiple bands includes pixels at the same intensity interval. In some examples, the intensity range is equally divided into multiple intervals, such as 32 intervals from zero to a maximum intensity value (e.g., 255 for 8-bit pixels), and each interval is associated with an offset. In addition, in one example, the 32 bands are divided into two groups, such as a first group and a second group. The first group includes the center 16 bands (e.g., 16 intervals located in the middle of the intensity range), and the second group includes the remaining 16 bands (e.g., 8 intervals located on the low side of the intensity range and 8 intervals located on the high side of the intensity range). In one example, only the offset of one of the two groups is transmitted. In some examples, the offset to be transmitted is determined based on rate-distortion performance. In some embodiments, when a BO type pixel classification operation is used, the five most significant bits of each pixel may be used directly as a band index.

[0148] Additionally, in one embodiment, for example, SAO Type 1 to SAO Type 4, edge offset (EO) types may be used to classify pixels and determine offsets. For example, pixel classification may be determined based on a one-dimensional 3-pixel pattern that takes into account edge direction information. In some examples,

[0149] Fig.10 An example of a 3-pixel pattern for classifying pixels in the edge offset type is shown in some examples. Fig.10In the example, the first pattern (1010) (as shown by 3 gray pixels) is for the 1D 0-degree pattern (horizontal) (SAO type 1), the second pattern (1020) (as shown by 3 gray pixels) is for the 1D 90-degree pattern (vertical) (SAO type 2), the third pattern (1030) (as shown by 3 gray pixels) is for the 1D 135-degree pattern (SAO type 3), and the fourth pattern (1040) (as shown by 3 gray pixels) is for the 1D 45-degree pattern (SAO type 4). In one example, one of the four directional patterns shown in Fig.10 can be selected considering the SAO type of the sub-region. Then, based on the pattern associated with the SAO type of the sub-region, each pixel (e.g., c) in the sub-region can be classified into multiple categories by comparing it with its two adjacent pixels (a and b).

[0150] In some examples, each pixel is classified into 4 categories. In one example, for pixel c, when c < a && c < b, pixel c is classified into category 1; when (c < a && c == b) || (c == a && c < b), pixel c is classified into category 2; when (c > a && c == b) || (c == a && c > b), pixel c is classified into category 3; and when c > a && c > b, pixel c is classified into category 4.

[0151] Fig.11 Table (1100) showing the pixel classification rules for edge offset in some examples is shown. Specifically, pixel c (also shown in each pattern of Fig.10 is compared with two adjacent pixels (a and b, also shown in gray in each pattern of Fig.10 ), and based on the comparison, pixel c can be classified into one of categories 0 to 4 according to the pixel classification rules shown in Fig.11 .

[0152] In some embodiments, SAO on the decoder side can operate independently of the largest coding unit (LCU) (e.g., CTU), so that the line buffer can be saved. In some examples, when the 90-degree classification pattern, 135-degree classification pattern, and 45-degree classification pattern are selected, the pixels in the top row and bottom row of each LCU are not subject to SAO processing; when the 0-degree pattern, 135-degree pattern, and 45-degree pattern are selected, the pixels in the leftmost column and rightmost column of each LCU are not subject to SAO processing.

[0153] Generally, the cross-component filtering process can use the reconstructed samples of the first color component as input (e.g., Y or Cb or Cr, or R or G or B) to generate an output, and the output of the filtering process is applied to a second color component different from the first color component.

[0154] In some examples, filter coefficients for cross component filtering (CCF) are derived based on some mathematical equations. The derived filter coefficients are signaled from the encoder side to the decoder side, and the derived filter coefficients are used to generate offsets using linear combinations. The generated offsets are then added to the reconstructed samples as a filtering process. For example, the offsets are generated based on a linear combination of the filter coefficients and the luma samples, and the generated offsets are added to the reconstructed chroma samples.

[0155] In some examples, a non-linear mapping technique may be used for cross-component filtering to generate a cross-component sample offset (CCSO), which may be referred to as CCSO.

[0156] In one example, CCSO can be used as a loop filter to reduce distortion of reconstructed samples.It should be noted that CCSO does not rely on the assumption of a linear mapping relationship between luma reconstructed sample values ​​and delta values ​​between original chroma samples and chroma reconstructed samples.

[0157] In some examples, for CCSO, a nonlinear mapping is derived on the encoder side. The nonlinear mapping is between the reconstructed samples of the first color component in the filter support region and the offset of the second color component to be added to the filter support region. The second color component is different from the first color component. The domain of the nonlinear mapping is determined by different combinations of processed input reconstructed samples (also referred to as possible combinations of reconstructed sample values).

[0158] A specific example may be used to illustrate the technique of CCSO. In the specific example, reconstructed samples from a first color component located in a filter support region (also referred to as a "filter support region") are determined. The filter support region is the region in which the filter can be applied, and the filter support region may have any suitable shape.

[0159] Fig.12 An example of a filter support region (1200) according to some embodiments of the present disclosure is shown. The filter support region (1200) includes three reconstructed samples: P0, P1, and C of a first color component, C represents a center sample, and P0 and P1 represent adjacent samples of C in the horizontal direction. P0 and P1 may be neighbors of C, or may not be neighbors of C. The filter support region (1200) also includes a sample F located at the center of a second color component. Samples C and F are juxtaposed and have different color components. The reconstructed samples P0, P1, and C are input to CCSO.

[0160] In some examples, reconstructed samples (eg, P0, P1, and C in a specific example) are input to the CCSO, which processes the input to form filter taps. In a specific example, the reconstructed samples are processed in two steps as follows.

[0161] In the first step, the delta values ​​between P0 and P1 and C are calculated respectively. For example, m0 represents the delta value between P0 and C; m1 represents the delta value between P1 and C.

[0162] In the second step, the delta values ​​m0 and m1 are further quantized, and the quantized values ​​are represented as d0 and d1. In one example, based on the quantization process, the quantized value can be one of -1, 0, and 1. For example, when m is less than -N, the value m can be quantized to -1 (N is a positive value, called the quantization step); when m is in the range of [-N, N], the value m can be quantized to 0; and when m is greater than N, the value m can be quantized to 1. In some examples, the quantization step N can be one of 4, 8, 12, 16, and so on.

[0163] In some embodiments, the quantized values ​​d0 and d1 are filter taps and can be used to identify a combination in the filter domain. For example, the filter taps d0 and d1 can form a combination in the filter domain. Each filter tap can have three quantized values, so when two filter taps are used, the filter domain includes 9 (3×3) combinations.

[0164] Fig.13 A table (1300) with 9 combinations according to an embodiment of the present disclosure is shown. Table (1300) includes 9 rows corresponding to the 9 combinations. In each row corresponding to a combination, the first column includes the index of the combination; the second column includes the value of the filter tap d0 for the combination; the third column includes the value of the filter tap d1 for the combination; and the fourth column includes an offset value associated with the combination for nonlinear mapping. In one example, when determining the filter taps d0 and d1, the offset value (denoted by s) associated with the combination of d0 and d1 can be determined according to table (1300). In one example, the offset values ​​s0 to s8 are integers, such as 0, 1, -1, 3, -3, 5, -5, -7, etc.

[0165] In some examples, the positions of adjacent samples P0 and P1 may be selected from a plurality of candidate positions, and the filter shapes may be switched.

[0166] Fig.14A diagram of a filter support region (1400) illustrating switchable filter shapes in some examples is shown. The filter support region (1400) includes a center sample C and 6 adjacent sample pairs 1 to 6. Each adjacent sample pair and the center sample C can form a 3-tap filter shape. For example, adjacent sample pair 1 and the center sample C form a first filter shape; adjacent sample pair 2 and the center sample C form a second filter shape; adjacent sample pair 3 and the center sample C form a third filter shape; adjacent sample pair 4 and the center sample C form a fourth filter shape; adjacent sample pair 5 and the center sample C form a fifth filter shape; adjacent sample pair 6 and the center sample C form a sixth filter shape. In some examples, the filter shape can be switched at the frame level. For example, a signal at the frame level may indicate a filter shape selected from the 6 filter shapes.

[0167] In some embodiments, a final filtering process of CCSO may be applied, as shown in equation (14):

[0168] f′=clip(F+s) Equation (14)

[0169] Wherein, F is the reconstructed sample of the second color component to be filtered, s is the offset value determined according to the filter tap, and the filter tap is the processing result of processing the reconstructed sample of the first color component, for example, using the table (1300). The sum of the reconstructed sample F and the offset value s is further clipped to a range associated with the bit depth to determine the final filtered sample f' of the second color component.

[0170] According to one aspect of the present disclosure, at the encoder side, the encoding device may derive a mapping between a reconstructed sample of a first color component in a filter support region and an offset to be added to a reconstructed sample of a second color component. The mapping may be any suitable linear or nonlinear mapping. Then, the filtering process may be applied on the encoder side and / or the decoder side based on the mapping. For example, the mapping is appropriately notified to the decoder (e.g., the mapping is included in the encoded video code stream transmitted from the encoder side to the decoder side), and then the decoder may perform the filtering process based on the mapping.

[0171] According to one aspect of the present disclosure, band features and edge features in SAO can be appropriately applied to CCSO. Band features and edge features are used jointly for offset derivation, where the edge features are derived using the delta value between P0 and P1 and C. In one example, a fixed number of bands (1, 2, 4, or 8) is used, which is signaled in the picture header. The number of pixel values ​​(also called intensity intervals) in each band is also fixed. In some examples, the encoder and decoder classify the reconstructed samples of the first color component (e.g., brightness) in a sub-region (e.g., CTU) into multiple bands according to pixel intensity and fixed intensity intervals. For each band, the encoder may determine the offset based on the average difference between the original sample and the reconstructed sample of the second color component. Then, the encoder may appropriately provide the offset of the band to the decoder. Then, the decoder may apply the offset to the reconstructed sample of the second color component according to the band classification (based on the first color component).

[0172] In some examples, signaling of CCSO can be performed at the frame level and the block level. At the frame level, the following syntax elements are signaled: a 1-bit flag indicating whether CCSO is applied in the frame; a 3-bit syntax element (ext_filter_support) indicating the selection of the CCSO filter shape; a 2-bit index indicating the selection of the quantization step size; and nine 3-bit offset values ​​used in the lookup table. At the 128×128 chroma block level, a flag is signaled to indicate whether the CCSO filter is enabled.

[0173] According to one aspect of the present disclosure, in some CCSO implementations, filter parameter search is computationally expensive and contributes a major share of encoder complexity. The present disclosure provides techniques for optimizing CCSO by terminating filter parameter search early (e.g., terminating some encoder calculations early). These techniques can speed up CCSO.

[0174] In the following description, CCSO is defined as a filtering process using reconstructed samples of a first color component as input (eg, Y or Cb or Cr), and the output is applied to a second color component that is a different color component from the first color component.

[0175] In the following description, the term 'block' may be interpreted as a prediction block, a coding block, or a coding unit, i.e., a CU. The term 'block' may also be used to refer to a transform block. Here, the term 'block' may also be used to refer to a CTU block or a CCSO super block (SB). The term 'filter block' or 'filter unit' may also be used to refer to a CTU block or a CCSO SB, which is associated with a flag indicating whether CCSO filtering is applied on the block (or CCSO SB).

[0176] In the description below, the term 'CCSO parameters' or 'filter parameters' may also be used to refer to CCSO filter parameters, examples of CCSO parameters include but are not limited to quantization step size, filter shape (or filter support), number of filter taps, number of frequency bands (or number of frequency bands), filter unit size, etc.

[0177] In the following description, the terms 'value' and 'index of value' may be used interchangeably. It should be noted that CCSO may be applied to luma blocks and / or chroma blocks.

[0178] According to one aspect of the present disclosure, the encoder's search for optimal values ​​of CCSO filter parameters, such as filter support (e.g., filter shape), quantization step size, and frequency band (frequency band properties, such as the number of frequency bands, the bandwidth of the frequency band), can be terminated early (optimized) based on the rate-distortion cost associated with the candidate CCSO filter parameters that the encoder has evaluated.

[0179] In some examples, the CCSO has multiple adjustable parameters that can change the performance of the CCSO. Each adjustable parameter provides some parameter candidates for evaluation. For example, for different pictures, different filter shapes (e.g., Fig.14 6 filter shapes in the example, etc.) can affect the rate-distortion cost of CCSO differently. In one example, 6 filter shapes can be filter shape candidates, 4 quantization step sizes (e.g., 16, 8, 4, 32) can be quantization step size candidates, and 4 frequency band numbers (e.g., 1, 2, 4, 8) can be frequency band candidates, then CCSO has 96 parameter candidate combinations. The encoder can search for parameter candidate combinations that can achieve, for example, the lowest rate-distortion cost. In some examples, the encoder does not evaluate all combinations, but can stop searching when the rate-distortion cost associated with the evaluated combination meets certain requirements, and the encoder can skip some combinations. In some examples, one or more adjustable parameters (rather than all adjustable parameters) can be referred to as an adjustable parameter subset.

[0180] The exemplary encoder disclosed herein searches for optimal values ​​of CCSO filter parameters (e.g., filter support, quantization step size, and frequency band), and may terminate and / or optimize the search early based on the rate-distortion cost associated with the candidate CCSO filter parameters that the encoder has evaluated. Specifically, the disclosed encoder may begin searching for a combination of values ​​of the filter parameters of the CCSO. Each combination of values ​​of the filter parameters includes values ​​for each filter parameter, respectively. Then, the disclosed encoder determines that a first rate-distortion cost associated with at least a first combination of values ​​satisfies a condition. In response to the first rate-distortion cost satisfying the condition, the encoder may terminate the search for at least a second combination of values ​​of the filter parameters of the CCSO early. Then, the encoder may select the values ​​of the filter parameters for performing CCSO to encode one or more blocks in the current picture based on the combination of values ​​obtained from the search. Here, the combination of values ​​obtained from the search does not include the second combination of values.

[0181] In one embodiment, when the rate-distortion cost of a picture filtered using one or more of a plurality of (more than one) options of a combination of CCSO filter parameters or a subset thereof is greater than the product of the rate-distortion cost of the unfiltered picture (i.e., rdcost_unfiltered) and a threshold (i.e., rdcost_unfiltered*thr1), further search for optimal values ​​of the CCSO filter parameters or a subset thereof is skipped and the given option is selected. In this case, the condition satisfied by the first rate-distortion cost is greater than the product of the rate-distortion cost of the unfiltered picture and a predetermined threshold thr1, e.g., rdcost_unfiltered*thr1. Exemplary predetermined threshold thr1 includes, but is not limited to, a value greater than or equal to 1.0.

[0182] In some examples, when the rate-distortion cost of a picture filtered by CCSO using one or more of the multiple options of a combination of adjustable filter parameters is greater than the rate-distortion cost of an unfiltered picture, further searching is skipped. The encoder may decide the value of the adjustable parameter based on the evaluated option (combination).

[0183] In one example, early termination is applied to only one of the CCSO filter parameters, namely the number of bands. For each combination of filter support and quantization step size, when the rate-distortion cost of the image filtered using a number of bands equal to B (the value of B includes but is not limited to 1, 2, 4, 8, ...) is greater than the product of the rate-distortion cost of the unfiltered image (i.e., rdcost_unfiltered) and the threshold (i.e., rdcost_unfiltered*thr2), the check of the values ​​of the remaining options for the number of bands other than B is skipped. Exemplary values ​​of thr2 include but are not limited to values ​​greater than or equal to 1.0.

[0184] In one example, the number of bands is evaluated according to a sequence (e.g., a sequence of 1, 2, 4, 8). To evaluate (subset) combinations of filter shapes and quantization step sizes, the number of bands may be evaluated according to a sequence. When the rate-distortion cost of the filtered picture with the number of bands is greater than the product of the rate-distortion cost of the unfiltered picture and a threshold, the evaluation of the remaining options for the number of bands in the sequence is skipped. For example, when the rate-distortion cost of 2 bands with the (subset) combination of filter shape and quantization step size is greater than the product of the rate-distortion cost of the unfiltered picture and a threshold, the 4-band and 8-band options are skipped.

[0185] In another example, early termination is applied to the encoder's selection of filter support and number of bands. For each value of the quantization step size, if the rate-distortion cost of a picture filtered using one of multiple combinations of filter support and number of bands and a given quantization step size is greater than the rate-distortion cost of the unfiltered picture (i.e., rdcost_unfiltered) multiplied by a threshold (i.e., rdcost_unfiltered*thr3), then the encoder's search for other combinations of filter support and bands is skipped (terminated). Exemplary values ​​of thr3 include, but are not limited to, values ​​greater than or equal to 1.0.

[0186] In one example, the filter shape and the number of bands are evaluated according to a sequence (e.g., a sequence generally along a direction of increasing rate-distortion). To evaluate each quantization step size, the filter shape and the number of bands may be evaluated according to the sequence. When the rate-distortion cost of the filtered picture with the filter shape and the number of bands is greater than the product of the rate-distortion cost of the unfiltered picture and a threshold, the evaluation of the remaining (subset) combinations of the filter shape and the number of bands in the sequence is skipped.

[0187] In one embodiment, for a certain combination value of a subset A of CCSO filter parameters (the number of values ​​in set A includes but is not limited to 1, 2, 3, ...), if the remaining subset B of filter parameters (the number of values ​​in set B includes but is not limited to 1, 2, 3, ...) produces the minimum rate distortion cost for a certain value set (e.g., M (the number of values ​​in set M is equal to set B)), then (by default) the value set M is selected for other combination values ​​of the subset A filter parameters without the need for the encoder to search. In other words, if a certain value of the subset B parameters produces the minimum rate distortion cost for the initial value of the subset A filter parameters, then for other combination values ​​of the subset A filter parameters, the encoder completely skips the search for the optimal value of the subset B parameters.

[0188] In some examples, the adjustable parameters are divided into subsets A and subsets B. Subset A may form, for example, P (subset) combinations, and subset B may form Q (subset) combinations. Evaluation of the adjustable parameters may be performed by selecting a value set (combination from the P combinations) and a value set (combination from the Q combinations) for subset A to form the combination being evaluated. In one example, during the evaluation, for each initial combination in the P (subset) combinations, the lowest rate-distortion cost is achieved by a certain value set M of subset B (e.g., a certain combination in the Q combinations), and then the evaluation of the remaining combinations in the P combinations of subset A is performed using a certain value set M of subset B, and other combinations of subset B may be skipped.

[0189] In one example, if the number of bands equal to A produces the lowest rate-distortion penalty for one of multiple possible combinations of filter support and quantization step size, then the number of bands equal to A is selected by default and the encoder does not perform any search for other combinations of filter support and quantization step size.

[0190] In one example, subset A includes quantization step size and filter shape, and subset B includes number of bands. When the number of bands is equal to S, and the lowest rate-distortion cost can be achieved for one or more initial combinations of filter shape and quantization step size, the number of bands equal to S is selected to evaluate other combinations of filter shape and quantization step size.

[0191] In one example, if a certain value set (e.g., S) from multiple combined values ​​of frequency band and quantization step size produces the lowest rate-distortion cost for a given value of filter support (e.g., X), then the value set S (for frequency band and quantization step size) is selected by default for filter support values ​​other than X, without the encoder having to perform any search for optimal frequency band and quantization step size values.

[0192] In one example, subset A includes filter shapes, while subset B includes number of bands and quantization step size. When a certain set of values ​​(e.g., S1 for number of bands and S2 for quantization step size) can achieve the lowest rate-distortion cost for X values ​​of filter shape, then S1 for number of bands and S2 for quantization step size are used to evaluate other values ​​of filter shape (in addition to the X values).

[0193] In one embodiment, the rate-distortion cost of the initial value of the subset A of filter parameters (the number of values ​​in set A includes but is not limited to 1, 2, 3, ...) and the minimum value among the rate-distortion costs of N (exemplary values ​​of N include but are not limited to 1, 2, 3, ...) checked values ​​of all combinations of the remaining subset B of filter parameters (e.g., set B may include values ​​1, 2, 3, ...) are recorded, for example, X. If the rate-distortion cost of the initial value of the subset A of filter parameters and the rate-distortion cost of the subsequently checked values ​​of the subset B filter parameters are greater than X, then checking other values ​​of the subset A filter parameters is skipped.

[0194] In some examples, the minimum value of the rate-distortion cost is recorded, tracked, and updated during the evaluation process. The adjustable parameters are divided into subsets A and subsets B. Subset A can form, for example, P (subset) combinations, and subset B can form Q (subset) combinations. The evaluation of the adjustable parameters can be performed by selecting a value set (a combination from the P combinations) for subset A and selecting a value set (a combination from the Q combinations) for subset B to form the combination being evaluated. In one example, for each combination of the Q combinations of subset B, the P combinations of subset A are evaluated. During the evaluation, for the combinations of subset B, when the rate-distortion cost of the initial combination of the P combinations of subset A is greater than the minimum value of the rate-distortion cost, the remaining combinations of the P combinations of subset A are skipped. The evaluation can continue to the next combination of subset B.

[0195] In one example, subset A includes quantization step sizes and subset B includes filter support. If the rate-distortion cost for a certain filter support and initial quantization step size value is greater than the minimum cost encountered so far, the encoder's search for the optimal quantization step size for that value of filter support is skipped.

[0196] Fig.15 A table (1500) is shown listing exemplary rate-distortion costs for various exemplary combinations of filter support and quantization step size. It is assumed that the rate-distortion costs listed in table (1500) are calculated using the listed filter support and quantization step size and the optimal frequency band for each combination of filter support and quantization step size. As can be seen from table (1500), for filter support values ​​3, 4, 5, the encoder's search for quantization step size is skipped because the rate-distortion cost for filter support 3, 4, 5 and quantization step size 16 is greater than the lowest rate-distortion cost for quantization step size 16.

[0197] Specifically, in one example, an optimal value for the number of frequency bands is determined, and the optimal value for the number of frequency bands is used to evaluate the filter shape (e.g., subset B) and the quantization step size (subset A). During the evaluation, for each filter shape, the quantization step sizes of 16, 8, 4, and 32 are evaluated in sequence, and the lowest rate-distortion cost is recorded. The filter shapes are evaluated in order from 0 to 5. In one example, when evaluating filter shape 3, the rate-distortion cost of filter shape 3 and quantization step size 16 is first calculated, such as 7490726759 shown in Table (1500), which is greater than the lowest rate-distortion cost of 7487391623 at that time. Then, the evaluation of quantization step sizes 8, 4, and 32 using filter shape 3 is skipped. The evaluation continues to filter shape 4. Similarly, when evaluating filter shape 4, the rate-distortion cost of filter shape 4 and quantization step size 16 is first calculated, such as 7495963893 shown in table (1500), which is greater than the lowest rate-distortion cost 7487391623 at that time. Then, the evaluation of quantization step sizes 8, 4, and 32 using filter shape 4 is skipped. The evaluation continues to filter shape 5. Similarly, when evaluating filter shape 5, the rate-distortion cost of filter shape 5 and quantization step size 16 is first calculated, such as 7497231888 shown in table (1500), which is greater than the lowest rate-distortion cost 7487391623 at that time. Then, the evaluation of quantization step sizes 8, 4, and 32 using filter shape 5 is skipped.

[0198] In some embodiments, if for the encoder's evaluation of the rate-distortion cost for a candidate combination of filter parameters, the cumulative rate-distortion cost of the CCSO super-blocks (SBs) visited so far is greater than the previous best rate-distortion cost, then for the remaining CCSO SBs for the given filter parameter combination, the encoder search, i.e., the rate-distortion cost calculation, is skipped / terminated early. In some examples, the super-block may correspond to a CTB.

[0199] For example, in one example, to evaluate the rate-distortion cost of the current combination of adjustable filter parameters, the encoder calculates the rate-distortion cost by SB in sequence (e.g., in scan order). After the encoder calculates the rate-distortion cost of the SB, the encoder accumulates the rate-distortion cost with the rate-distortion cost of the previously accumulated SB. When the accumulated rate-distortion cost is higher than the lowest rate-distortion cost of the previous combination, the encoder stops calculating the rate-distortion cost of the remaining SBs in the picture, for example. The evaluation may continue to the next combination.

[0200] Fig.16A flow chart outlining a process (1600) according to an embodiment of the present disclosure is shown. The process (1600) may be used for a video encoder. In various embodiments, the process (1600) is performed by a processing circuit, such as a processing circuit that performs the functions of a video encoder (103), a processing circuit that performs the functions of a video encoder (303), etc. In some embodiments, the process (1600) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the process (1600). The process starts at (S1601) and proceeds to (S1610).

[0201] At (S1610), it is determined to encode one or more blocks in a current picture using cross-component sample offsets (CCSO).

[0202] At (S1620), an evaluation process is started, which searches for value combinations of filter parameters of the CCSO. The filter parameters have multiple optional values ​​respectively, and the value combinations of the filter parameters include values ​​respectively assigned to the filter parameters. For example, a search for value combinations of filter parameters of the CCSO is started, and each combination of values ​​of the filter parameters includes values ​​respectively used for each filter parameter.

[0203] According to one aspect of the present disclosure, the encoder's search for optimal values ​​of CCSO filter parameters, such as filter support (e.g., filter shape), quantization step size, and frequency band (frequency band properties, such as the number of frequency bands, the bandwidth of the frequency band), can be terminated early (optimized) based on the rate-distortion cost associated with the candidate CCSO filter parameters evaluated by the encoder (e.g., the operation in the following steps).

[0204] At (S1630), it is determined that a first rate-distortion cost associated with at least a first combination of values ​​satisfies a condition.

[0205] At (S1640), in response to the first rate-distortion cost satisfying the condition, searching for at least a second combination of values ​​of the filter parameters is terminated.

[0206] At (S1650), based on the combination of values ​​obtained from the search, values ​​of filter parameters for performing CCSO to encode one or more blocks in the current picture are selected, the combination of values ​​obtained from the search excluding the second combination of values.

[0207] In some examples, the filter parameters include at least one of a filter shape, a number of filter taps, a quantization step size, a filter unit size, and a number of frequency bands.

[0208] In some embodiments, when the rate-distortion cost of a picture filtered using one or more of a plurality (more than one) of a combination of CCSO filter parameters or a subset thereof is greater than the product of the rate-distortion cost of the unfiltered picture (i.e., rdcost_unfiltered) and a threshold (i.e., rdcost_unfiltered*thr1), further search for optimal values ​​of CCSO filter parameters or a subset thereof is skipped and the given option is selected. Exemplary values ​​of thr1 include, but are not limited to, values ​​greater than or equal to 1.0.

[0209] In some examples, to determine whether the first rate-distortion cost meets the requirement, it is determined that the first rate-distortion cost is greater than a first threshold cost, where the first threshold cost is a product of a rate-distortion cost of samples not filtered by CCSO and a first threshold.

[0210] In one example, early termination is applied to only one of the CCSO filter parameters, namely the number of bands. For each combination of filter support and quantization step size, when the rate-distortion cost of the image filtered using a number of bands equal to B (the value of B includes but is not limited to 1, 2, 4, 8, ...) is greater than the product of the rate-distortion cost of the unfiltered image (i.e., rdcost_unfiltered) and the threshold (i.e., rdcost_unfiltered*thr2), the check of the values ​​of the remaining options for the number of bands other than B is skipped. Exemplary values ​​of thr2 include but are not limited to values ​​greater than or equal to 1.0.

[0211] Specifically, in one example, a first combination of values ​​includes a first number of frequency bands, a first filter shape, and a first quantization step size, and at least each of the second combinations includes a first filter shape, a first quantization step size, and a second number of frequency bands different from the first number of frequency bands.

[0212] In another example, early termination is applied to the encoder's selection of filter support and number of bands. For each value of the quantization step size, if the rate-distortion cost of a picture filtered using one of multiple combinations of filter support and number of bands and a given quantization step size is greater than the rate-distortion cost of the unfiltered picture (i.e., rdcost_unfiltered) multiplied by a threshold (i.e., rdcost_unfiltered*thr3), then the encoder's search for other combinations of filter support and bands is skipped (terminated). Exemplary values ​​of thr3 include, but are not limited to, values ​​greater than or equal to 1.0.

[0213] Specifically, in one example, the first combination includes a first subset combination of a first quantization step size and a value, the first subset combination including a first number of frequency bands and a first filter shape. Each of at least the second combinations includes a second subset combination of a first quantization step size and a value, the second subset combination including at least one of a second number of frequency bands different from the first number of frequency bands and / or a second number of filter shapes different from the first filter shape.

[0214] In some embodiments, for a certain combination value of a subset A of CCSO filter parameters (the number of values ​​in set A includes but is not limited to 1, 2, 3, ...), if the remaining subset B of filter parameters (the number of values ​​in set B includes but is not limited to 1, 2, 3, ...) produces the minimum rate distortion cost for a certain value set (e.g., M (the number of values ​​in set M is equal to set B)), then (by default) the value set M is selected for other combination values ​​of the subset A filter parameters without the need for the encoder to search. In other words, if a certain value of the subset B parameters produces the minimum rate distortion cost for the initial value of the subset A filter parameters, then for other combination values ​​of the subset A filter parameters, the encoder completely skips the search for the optimal value of the subset B parameters.

[0215] Specifically, in some examples, to determine that the first rate-distortion cost satisfies the requirement, it is determined that for the first subset of filter parameters, a specific combination of values ​​of the second subset of filter parameters achieves a minimum rate-distortion cost for one or more combinations of values ​​of the first subset of filter parameters. Then, at least a second combination of values ​​includes a combination of values ​​of the second subset of filter parameters, and the combination of values ​​of the second subset of filter parameters is different from the specific combination of values ​​of the second subset.

[0216] In one example, if the number of bands equal to A produces the lowest rate-distortion penalty for one of multiple possible combinations of filter support and quantization step size, then the number of bands equal to A is selected by default and the encoder does not perform any search for other combinations of filter support and quantization step size.

[0217] Specifically, in one example, a second subset of filter parameters includes the number of frequency bands, a specific combination of values ​​of the second subset includes a specific value of the number of frequency bands, and at least a second combination of values ​​includes a value of the number of frequency bands that is different from the specific value of the number of frequency bands.

[0218] In one example, if a certain value set (e.g., S) from multiple combined values ​​of frequency band and quantization step size produces the lowest rate-distortion cost for a given value of filter support (e.g., X), then the value set S (for frequency band and quantization step size) is selected by default for filter support values ​​other than X, without the encoder having to perform any search for optimal frequency band and quantization step size values.

[0219] Specifically, in one example, the second subset of filter parameters includes the number of frequency bands and the quantization step size. A specific combination of values ​​of the second subset includes a first specific value of the number of frequency bands and a second specific value of the quantization step size, and at least a second combination of values ​​includes a first value of the number of frequency bands and a second value of the quantization step size, the first value being different from the first specific value of the number of frequency bands, and the second value being different from the second specific value of the quantization step size.

[0220] In some embodiments, the rate-distortion cost of the initial value of the subset A of filter parameters (the number of values ​​in set A includes but is not limited to 1, 2, 3, ...) and the rate-distortion cost of all combinations of N (exemplary values ​​of N include but are not limited to 1, 2, 3, ...) checked values ​​of the remaining subset B of filter parameters (e.g., set B may include values ​​1, 2, 3, ...) are recorded, for example, a minimum value X. If the rate-distortion cost of the initial value of the subset A of filter parameters and the rate-distortion cost of the subsequently checked values ​​of the subset B filter parameters are greater than X, then checking other values ​​of the subset A filter parameters is skipped.

[0221] Specifically, in some examples, to determine that a first rate-distortion cost satisfies a requirement, it is determined that a current evaluation of one or more combinations of values ​​of a first subset of filter parameters and a particular combination of values ​​of a second subset of filter parameters has a higher rate-distortion cost than a lowest rate-distortion cost obtained from a combination of values ​​evaluated prior to the current evaluation.

[0222] In one example, at least the second combination of values ​​comprises a combination of values ​​of the first subset of filter parameters, the combination of values ​​of the first subset of filter parameters being different from one or more combinations of values ​​of the first subset of filter parameters. For example, the first subset of filter parameters comprises a quantization step size, and the second subset of filter parameters comprises a filter shape, e.g., see Fig.15 Description.

[0223] In some embodiments, if, for the encoder's evaluation of the rate-distortion cost of a candidate combination of filter parameters, the cumulative rate-distortion cost of the CCSO super-blocks (SBs) visited so far is already greater than the previous best rate-distortion cost, then for the remaining CCSO SBs for the given filter parameter combination, the encoder search, i.e., the rate-distortion cost calculation, is skipped / terminated early.

[0224] Specifically, in some examples, to determine that a first rate-distortion cost associated with at least a first combination of values ​​satisfies a requirement, for the first combination of values ​​of the filter parameter, a cumulative rate-distortion cost is calculated for a first plurality of blocks in the one or more blocks, the one or more blocks comprising the first plurality of blocks and the one or more additional blocks, the cumulative rate-distortion cost is compared with a lowest rate-distortion cost of a combination previously evaluated for the first combination, and in response to the cumulative rate-distortion cost being higher than the lowest rate-distortion cost, the rate-distortion cost calculation for the one or more additional blocks is skipped.

[0225] Then, the process proceeds to (S1699) and terminates.

[0226] The process (1600) may be adapted as appropriate. Steps in the process (1600) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0227] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.17 A computer system (1700) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0228] Computer software may be encoded using any suitable machine code or computer language that may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through interpretation, microcode execution, etc.

[0229] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0230] Fig.17 The components of the computer system (1700) shown are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components shown in the exemplary embodiment of the computer system (1700).

[0231] The computer system (1700) may include certain human interface input devices. Such human interface input devices may be responsive to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not depicted). Human interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, captured images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0232] The human-machine interface input device may include one or more of the following (only one of each is shown): keyboard (1701), mouse (1702), touchpad (1703), touch screen (1710), data gloves (not shown), joystick (1705), microphone (1706), scanner (1707), camera (1708).

[0233] The computer system (1700) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen (1710), a data glove (not shown), or a joystick (1705), but may also be a tactile feedback device that is not an input device), audio output devices (e.g., speakers (1709), headphones (not depicted)), visual output devices (e.g., screens (1710) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, some of which are capable of outputting two-dimensional visual outputs or outputs in more than three dimensions through devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).

[0234] The computer system (1700) may also include human-machine accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (1720) with CD / DVD and other media (1721), thumb drives (1722), removable hard drives or solid-state drives (1723), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD-based devices such as security software dogs (not depicted), etc.

[0235] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0236] The computer system (1700) may also include an interface (1754) to one or more communication networks (1755). The network may be, for example, a wireless network, a wired network, an optical network. The network may further be a local network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a delay-tolerant network, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CANBus, etc. Some networks typically require an external network interface adapter attached to some general data port or peripheral bus (1749) (e.g., a USB port of the computer system (1700)); as described below, other network interfaces are typically integrated into the kernel of the computer system (1700) by attaching to the system bus (e.g., connected to an Ethernet interface in a PC computer system or connected to a cellular network interface in a smartphone computer system). The computer system (1700) can use any of these networks to communicate with other entities. Such communications may be one-way receive only (e.g., broadcast television), one-way send only (e.g., CANBus connected to certain CANBus devices), or bidirectional, for example, using a LAN or WAN digital network to connect to other computer systems. As described above, certain protocols and protocol stacks may be used on each of those networks and network interfaces.

[0237] The above-mentioned human-machine interface device, human-machine accessible storage device and network interface may be attached to the kernel (1740) of the computer system (1700).

[0238] The kernel (1740) may include one or more central processing units (CPUs) (1741), graphics processing units (GPUs) (1742), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1743), hardware accelerators (1744) for certain tasks, graphics adapters (1750), and the like. These devices, as well as read-only memory (ROM) (1745), random access memory (1746), internal mass storage (1747) such as internal non-user accessible hard drives, SSDs, and the like, may be connected via a system bus (1748). In some computer systems, the system bus (1748) may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to the kernel's system bus (1748) or to the kernel's system bus (1748) via a peripheral bus (1749). In one example, a touch screen (1710) may be connected to a graphics adapter (1750). The architecture of the peripheral bus includes PCI, USB, and the like.

[0239] The CPU (1741), GPU (1742), FPGA (1743) and accelerator (1744) can execute certain instructions, which can be combined to form the above-mentioned computer code. The computer code can be stored in ROM (1745) or RAM (1746). Transitional data can also be stored in RAM (1746), while permanent data can be stored, for example, in internal mass storage (1747). Fast storage and retrieval to any storage device can be performed by using a cache, which can be closely associated with the following: one or more CPUs (1741), GPUs (1742), mass storage (1747), ROM (1745), RAM (1746), etc.

[0240] The computer readable medium may have thereon computer codes for performing various computer-implemented operations. The medium and computer codes may be those specially designed and constructed for the purposes of the present disclosure, or the medium and computer codes may be of a type well known and available to those skilled in the art of computer software.

[0241] As an example, and not by way of limitation, a computer system (1700) having an architecture, particularly a kernel (1740), may provide functionality due to one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage as described above, as well as certain non-temporary kernel (1740) memories, such as kernel internal mass storage (1747) or ROM (1745). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the kernel (1740). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software may cause the kernel (1740), particularly the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (1746) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality due to logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1744)) that may replace software or operate in conjunction with software to perform specific processes or specific portions of specific processes described herein. Where appropriate, references to portions of software may include logic and vice versa. Where appropriate, references to portions of computer-readable media may include circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0242] As used in this disclosure, "at least one of" is intended to include any one or combination of the listed elements. For example, references to at least one of A, B, and C; at least one of A, B, and C; at least one of A, B, and / or C; and at least one of A through C are intended to include only A, only B, only C, or any combination thereof.

[0243] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various replacement equivalents that fall within the scope of the present disclosure. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and therefore fall within the spirit and scope of the present disclosure.

Claims

1. A method for performing video processing in an encoder, comprising: Determining to encode one or more blocks in a current picture using cross-component sample offsets (CCSO); Initiate a search for combinations of values ​​of filter parameters of the CCSO, each combination of values ​​of filter parameters comprising a value for each filter parameter; determining that a first rate-distortion cost associated with at least a first combination of values ​​satisfies a condition; terminating searching for at least a second combination of values ​​of the filter parameters of the CCSO in response to the first rate-distortion penalty satisfying the condition; as well as Based on the combination of values ​​obtained from the search, values ​​of the filter parameters are selected for performing the CCSO to encode the one or more blocks in the current picture, wherein the combination of values ​​obtained from the search does not include the second combination of values.

2. The method according to claim 1, wherein: The filter parameters include at least one of a filter shape, a number of filter taps, a quantization step size, a filter unit size, and a number of frequency bands.

3. The method according to claim 1, wherein: Determining that the first rate-distortion cost satisfies the condition further includes: It is determined that the first rate-distortion cost is greater than a first threshold cost, the first threshold cost being a product of a rate-distortion cost of samples not filtered by the CCSO and a first threshold.

4. The method according to claim 3, wherein: The first combination of values ​​comprises a first number of frequency bands, a first filter shape and a first quantization step size, and at least each of the second combinations comprises a second number of frequency bands, the first filter shape and the first quantization step size; the second number of frequency bands is different from the first number of frequency bands.

5. The method according to claim 3, wherein: The first combination comprises a first subset combination of a first quantization step size and a value; the first subset combination comprises a first number of frequency bands and a first filter shape; at least each of the second combinations comprises the first quantization step size, and a second subset combination of values; and the second subset combination comprises at least one of a second number of frequency bands different from the first number of frequency bands and / or a second number of filter shapes different from the first filter shape.

6. The method according to claim 1, wherein: Determining that the first rate-distortion cost satisfies the condition further includes: For the first subset of filter parameters, determining that a particular combination of values ​​of the second subset of filter parameters achieves a lowest rate-distortion penalty for one or more combinations of values ​​of the first subset of filter parameters.

7. The method according to claim 6, wherein: At least said second combination of values ​​comprises a combination of values ​​of said second subset of said filter parameters that is different from said specific combination of values ​​of said second subset.

8. The method according to claim 7, wherein: The second subset of the filter parameters comprises the number of frequency bands, the specific combination of values ​​of the second subset comprises a specific value of the number of frequency bands, and at least the second combination of values ​​comprises a value of the number of frequency bands that is different from the specific value of the number of frequency bands.

9. The method according to claim 7, wherein: The second subset of the filter parameters comprises the number of frequency bands and a quantization step size, the specific combination of values ​​of the second subset comprises a first specific value of the number of frequency bands and a second specific value of the quantization step size, and at least the second combination of values ​​comprises a first value of the number of frequency bands and a second value of the quantization step size, the first value being different from the first specific value of the number of frequency bands and the second value being different from the second specific value of the quantization step size.

10. The method according to claim 1, wherein: Determining that the first rate-distortion cost satisfies the condition further includes: Determining an evaluation of one or more combinations of values ​​of the first subset of filter parameters and a particular combination of values ​​of the second subset of filter parameters has a higher rate-distortion cost than a lowest rate-distortion cost obtained from a combination of values ​​evaluated prior to the evaluation.

11. The method according to claim 10, wherein: At least the second combination of values ​​comprises a combination of values ​​of the first subset of the filter parameters that is different from the one or more combinations of values ​​of the first subset of the filter parameters.

12. The method according to claim 11, wherein: The first subset of the filter parameters comprises a quantization step size and the second subset of the filter parameters comprises a filter shape.

13. The method according to claim 1, wherein: Determining that the first rate-distortion cost associated with at least the first combination of values ​​satisfies the condition further comprises: For said first combination of values ​​of said filter parameters, calculating a cumulative rate-distortion cost for a first plurality of blocks of the one or more blocks, the one or more blocks comprising the first plurality of blocks and one or more additional blocks; comparing the accumulated rate-distortion cost to a lowest rate-distortion cost of a combination previously evaluated for the first combination; and In response to the accumulated rate-distortion cost being higher than the lowest rate-distortion cost, rate-distortion cost calculation for the one or more additional blocks is skipped.

14. An apparatus for video encoding, comprising a processing circuit, the processing circuit being configured to: Determining to use cross-component sample offsets (CCSO) to encode one or more blocks in a current picture; Initiate a search for combinations of values ​​of filter parameters of the CCSO, each combination of values ​​of filter parameters comprising a value for each filter parameter; determining that a first rate-distortion cost associated with at least a first combination of values ​​satisfies a condition; terminating searching for at least a second combination of values ​​of the filter parameters of the CCSO in response to the first rate-distortion penalty satisfying the condition; as well as Based on the combination of values ​​obtained from the search, values ​​of the filter parameters are selected for performing the CCSO to encode the one or more blocks in the current picture, wherein the combination of values ​​obtained from the search does not include the second combination of values.

15. The device according to claim 14, wherein: The filter parameters include at least one of a filter shape, a number of filter taps, a quantization step size, a filter unit size, and a number of frequency bands.

16. The device according to claim 14, wherein: The processing circuit is configured to: It is determined that the first rate-distortion cost is greater than a first threshold cost, the first threshold cost being a product of a rate-distortion cost of samples not filtered by the CCSO and a first threshold.

17. The device according to claim 16, wherein: The first combination of values ​​comprises a first number of frequency bands, a first filter shape and a first quantization step size, and at least each of the second combinations comprises a second number of frequency bands, the first filter shape and the first quantization step size, the second number of frequency bands being different from the first number of frequency bands.

18. The device according to claim 16, wherein: The first combination includes a first subset combination of a first quantization step size and a value; the first subset combination includes a first number of frequency bands and a first filter shape; at least each of the second combinations includes the first quantization step size, and a second subset combination of values; the second subset combination includes at least one of a second number of frequency bands different from the first number of frequency bands and / or a second number of filter shapes different from the first filter shape.

19. The device according to claim 14, wherein: The processing circuit is configured to: For the first subset of filter parameters, determining that a particular combination of values ​​of the second subset of filter parameters achieves a lowest rate-distortion penalty for one or more combinations of values ​​of the first subset of filter parameters.

20. The device according to claim 19, wherein At least said second combination of values ​​comprises a combination of values ​​of said second subset of said filter parameters that is different from said specific combination of values ​​of said second subset.

Images