Enhanced cross-component sample offset filtering system for video encoding and decoding

Abstract

The present disclosure relates to an enhanced cross-component sample offset filtering system for video encoding and decoding. A cross-component based filtering system for video encoders and decoders is disclosed. The filtering system can include a filter having an input for a filter offset and an input for samples reconstructed from encoded video data representing a native component of a source video on which the filter operates. The offset can be generated at least in part from a sample classifier that classifies samples reconstructed from encoded video data representing a color component of the source video that is orthogonal to the native component according to sample intensity.

Description

[0001] Cross Reference to Related Applications

[0002] This patent application claims the benefit under 35 U.S.C. § 119(e) of provisional U.S. Patent Application No. 63 / 277,326, filed November 9, 2021, the contents of which are incorporated herein in their entirety. TECHNICAL FIELD

[0003] The present disclosure relates to techniques for video encoding and decoding for bandwidth-limited communication channels, and in particular to techniques for loop filtering for such applications. BACKGROUND

[0004] Video encoding / decoding applications generally exploit spatial and / or temporal redundancies in video data to generate an encoded representation of the video data that has reduced bandwidth compared to source video data from which the video data was generated. These techniques generally apply prediction algorithms that predict video content from earlier encoded content, determine differences between the actual video content and its predicted content, and then encode the residuals representing these differences. In this way, video encoding devices and video decoding devices employ algorithms to maintain a synchronized representation of the prediction data.

[0005] Video encoding / decoding techniques are generally “lossy” processes, meaning that the recovered video data generated by decoding the encoded video data will exhibit errors compared to the source video data from which the video data was generated. Since the video decoder does not have access to the source video data, the prediction algorithms operate in the domain of the recovered video data.

[0006] Decoding algorithms can apply filtering to the recovered video data to improve the quality of the recovered video data before employing the recovered video data for prediction. The filtering algorithms can be applied in the “loop” of prediction, meaning that the filtering algorithms are applied by the video encoder and the video decoder in the same way, so that they remain synchronized. Filtering algorithms that affect the quality of the recovered video data are naturally undesirable. Thus, the video encoder and decoder are designed to dynamically adjust the filtering operations to improve the overall quality of the video. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 Block diagram of a video encoding system according to embodiments of the present disclosure.

[0008] Figure 2 Block diagram of a video encoder according to embodiments of the present disclosure.

[0009] Figure 3 Block diagram of a video decoder according to embodiments of the present disclosure.

[0010] Figure 4 A block diagram of a cross-component sample offset filter according to embodiments of the disclosure.

[0011] Figure 5 An exemplary sample array suitable for use with embodiments of the disclosure is shown.

[0012] Figure 6 An exemplary sample array suitable for use with embodiments of the disclosure is shown.

[0013] Figure 7 An exemplary set of candidate directional blocks suitable for use with embodiments of the disclosure is shown.

[0014] Figure 8 An exemplary set of filter taps suitable for use with embodiments of the disclosure is shown.

[0015] Figure 9 An exemplary set of filter taps suitable for use with embodiments of the disclosure is shown. DETAILED DESCRIPTION

[0016] Embodiments of the disclosure provide a cross-component based filtering system for video encoders and decoders. The filtering system can include a filter having an input for a filter offset and an input for samples reconstructed from encoded video data representing native components of a source video on which the filter operates. The offset can be generated at least in part from a sample classifier that classifies samples reconstructed from encoded video data representing color components of the source video that are orthogonal to the native components according to sample intensity.

[0017] Figure 1 A simplified block diagram of a video encoding system 100 according to embodiments of the disclosure is shown. System 100 can include a plurality of terminals 110-120 interconnected via a network 130. Certain terminals can encode video data for transmission to another terminal via network 130. Other terminals can receive other terminals' encoded video data from network 130, decode the encoded video data, and generally consume the video data recovered therefrom by displaying the decoded video.

[0018] The video encoding system 100 can be used in a variety of applications. In a first application, the pair of terminals 110, 120 can support real-time two-way exchange of encoded video to establish a video conferencing session between them. In another application, the terminal 110 can encode a pre-produced video (e.g., a television or movie program) and store the encoded video for delivery to a downloading client 120 or frequently to multiple downloading clients 120. Thus, the video being encoded can be real-time or pre-produced, and it can be distributed in a one-to-one or one-to-many distribution model. For the purposes of this discussion, the type of video and the video distribution scheme are immaterial unless otherwise indicated.

[0019] In Figure 1 The terminals 110-120 are shown as smartphones in the figure, but the principles of the application are not limited thereto. Embodiments of the disclosure can be applied to set-top boxes, televisions, computers (both desktop and laptop computers), tablet computers, computer servers, media players, and / or dedicated video conferencing and entertainment devices, among others.

[0020] The network 130 represents any number of networks that transport encoded video data between the terminals 110-120, including, for example, wired and / or wireless communication networks. The communication network 130 can exchange data in circuit- switched channels and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network 130 are immaterial to the operation of the disclosure unless otherwise indicated.

[0021] Figure 2 A simplified block diagram of an encoding system 200 in an embodiment of the disclosure is shown. The system 200 can encode input frames of a video sequence by compression techniques that reduce the bit rate size of the frames. The system can include a segmentation unit 210 and one or more encoding units 220-240. The segmentation unit 210 can first deconstruct the content of the input frames according to their components in a color space, such as by deconstructing the components into a luminance (often simply referred to as "luma") component and chrominance components (often two chrominance components are present, labeled Cr and Cb). The segmentation unit can output the component data to its respective color encoder 220-240. In Figure 2 In the example shown, the luminance color data is output to a luminance encoder 220, the Cr color data is output to a Cr encoder 230, and the Cb color data is output to a Cb encoder 240. Although Figure 2 The application to the Y / Cb / Cr color space is shown, but the principles of the disclosure are not limited thereto; the principles of the disclosure can be applied to other color spaces such as RGB, XYZ, etc. in addition to what is described below.

[0022] The partition unit 210 can also parse the data of each color component into a spatial array, referred to as "pixel blocks." Many encoding protocols employ various types of partitioning schemes, which can be recursive; the partition unit 210 can work in conjunction with such schemes. Thus, the color component data can be partitioned according to a quadtree partitioning scheme, which can result in different sized encoding units to be processed by the encoders 220-240. Alternatively, the color component data can be partitioned into macroblocks, and further partitioned into blocks for encoding by the encoders 220-240. Often, it will occur that the pixel blocks of a color plane frame can have different block sizes within the frame. Moreover, the pixel blocks generated for one encoder (i.e., the luma encoder 220) can have different sizes than the pixel blocks generated for the other encoders (the Cr encoder 230 and / or the Cb encoder 240). Typically, the pixel block size and arrangement within a color plane is selected according to content analysis of that plane.

[0023] The encoders 220-240 can perform encoding operations on their respective color plane data to reduce their bandwidth. Typically, the encoders 220-240 exploit temporal and / or spatial redundancy within their input data. For example, the encoders 220-240 can perform motion-compensated predictive encoding, in which an input pixel block is encoded according to any of a variety of predictive encoding modes, such as:

[0024] • intra-frame encoding, in which an input pixel block is differentially encoded relative to previously encoded / decoded data of a common frame;

[0025] • single-predicted inter-frame encoding, in which an input pixel block is differentially encoded relative to data of a previously encoded / decoded frame; and

[0026] • multi-hypothesis motion-compensated predictive encoding, in which an input pixel block is predictively encoded using decoded data from two or more sources via temporal or spatial prediction.

[0027] The predictive encoding modes can be used in cooperation with other encoding techniques, such as transform-skip encoding, RRU encoding, scaling of the predictive source, palette encoding, and so forth.

[0028] Figure 2Exemplary functional units of a pixel block encoder that can be applied in the luma encoder 220, the Cr encoder 230, and the Cb encoder 240 are shown. The pixel block encoder can include a pixel block encoder 250, a pixel block decoder 260, a loop filter 270, a reference picture cache 280, and a predictor 290. The pixel block encoder 250 can encode an input pixel block with reference to a predicted pixel block supplied by the predictor 290 and output encoded pixel block data derived therefrom. The pixel block decoder 260 can reverse the encoding operations applied by the pixel block encoder 250 to derive decoded pixel block data therefrom. The loop filter 270 can apply selected filtering operations to the decoded pixel block data, which can be reconstructed into a recovered frame and stored in the reference picture cache 280. The predictor 290 can supply prediction data as prediction data to the pixel block encoder 250.

[0029] The pixel block encoder 250 can include a subtractor 252, a transform unit 254, a quantizer 256, and an entropy encoder 258. The pixel block encoder 250 can accept a pixel block of input data at the subtractor 252. The subtractor 252 can receive a predicted pixel block from the predictor 290 and generate therefrom an array of pixel residuals representing the difference between the input pixel block and the predicted pixel block. The transform unit 254 can apply a transform to the sample data output from the subtractor 252 to convert the data from the pixel domain to the transform coefficient domain. The quantizer 256 can perform quantization on the transform coefficients output by the transform unit 254. The quantizer 256 can be a uniform quantizer or a non-uniform quantizer. The entropy encoder 258 can reduce the bandwidth of the output of the quantizer by, for example, encoding the output with variable length code words or using a context adaptive binary arithmetic encoder.

[0030] The transform unit 254 can operate in a variety of transform modes. For example, the transform unit 254 can apply a discrete cosine transform (DCT), a discrete sine transform (DST), a Walsh-Hadamard transform, a Haar transform, a Daubechies wavelet transform, etc. In one aspect, a controller (not shown) can select an encoding mode M to be applied by the transform unit 254, can configure the transform unit 254 accordingly, and can signal the encoding mode M in the encoded video data, either explicitly or implicitly.

[0031] The quantizer 256 can operate according to a quantization parameter QP supplied by a controller (not shown). In one aspect, the quantization parameter QP can be applied as a multi-valued quantization parameter, e.g., that can vary over different coefficient positions within a transform domain pixel block. Thus, the quantization parameter QP can be provided as an array of quantization parameters.

[0032] The entropy encoder 258, as its name implies, can perform entropy encoding of the data output from the quantizer 256. For example, the entropy encoder 258 can perform run-length encoding, Huffman encoding, Columbus encoding, context adaptive binary arithmetic coding, and the like.

[0033] The pixel block decoder 260 can reverse the encoding operations of the pixel block encoder 250. For example, the pixel block decoder 260 can include an inverse quantizer 262, an inverse transform unit 264, and a summer 266. The pixel block decoder 260 can obtain its input data from the output of the quantizer 256. Although permitted, the pixel block decoder 260 need not perform entropy decoding of the entropy encoded data, as entropy encoding is a lossless event. The inverse quantizer 262 can reverse the operations of the quantizer 256 of the pixel block encoder 250. The inverse quantizer 262 can perform uniform or non-uniform dequantization, as specified by a decoded signal QP. Similarly, the inverse transform unit 264 can reverse the operations of the transform unit 254. The inverse quantizer 262 and the inverse transform unit 264 can use the same quantization parameter QP and transform mode M as their counterparts in the pixel block encoder 250. Quantization operations generally truncate data in various ways, and thus, the data recovered by the inverse quantizer 262 can have encoding errors when compared to the data presented to the quantizer 256 in the pixel block encoder 250.

[0034] The summer 266 can reverse the operations performed by the subtracter 252. It can receive the same predicted pixel block from the predictor 290 as was used by the subtracter 252 in generating the residual signal. The summer 266 can add the predicted pixel block to the reconstructed residual values output by the inverse transform unit 264, and can output the reconstructed pixel block data.

[0035] As described, the loop filter 270 can perform various filtering operations 272-276 on the recovered pixel block data. The loop filter can have a cross-component sample offset ("CCSO") filter 274, as discussed below. In one embodiment, the CCSO filter 274 can be part of a multi-filter filtering system 270, which can include other filters 272, 276. Such other filters can include, for example, a deblocking filter, a sample adaptive offset ("SAO") filter, a constrained direction enhancement filter ("CDEF"), an adaptive loop filter ("ALF"), a cross-component adaptive loop filter ("CCALF"), and / or a loop restoration ("LR") filter.

[0036] The CCSO filter 274 presented herein can work in conjunction with other filtering units within the loop filter 270. For example, the CCSO filter 274 can accept input data from the deblocking filter and output filtered data to the CDEF. Alternatively, the CCSO filter 274 can accept input data from the CDEF and output filtered data to the LR filter. In another implementation, the CCSO can be placed in parallel to the CDEF; in this application, the input to the CCSO can be the reconstructed samples prior to the CDEF, and the output of the CCSO can be applied to the reconstructed samples obtained from the CDEF. In another implementation, the CCSO filter 274 can accept input data from the deblocking filter and output filtered data to the ALF.

[0037] As discussed, the reference picture cache 280 can store filtered frame data for use in subsequent prediction of other pixel blocks. For different prediction modes, different types of prediction data can be available to the predictor 290. For example, for an input pixel block, intra prediction obtains a prediction reference from decoded data of the same frame in which the input pixel block is positioned. Thus, the reference picture cache 280 can store decoded pixel block data for each frame at the time of encoding. For the same input pixel block, inter prediction can employ a prediction reference from a previously encoded and decoded frame, which is designated as a reference frame. Thus, the reference picture cache 280 can store these decoded reference frames.

[0038] As discussed, the predictor 290 can supply a prediction block to the pixel block encoder 250 for use in generating a residual. The predictor 290 can include an inter predictor, an intra predictor, and a mode decision unit (not shown). The inter predictor can receive pixel block data representing a new pixel block to be encoded, and can search the reference picture cache 280 for reference frame data from pixel block data of a reference frame for use in encoding the input pixel block. The inter predictor can select prediction reference data that provides the closest match to the input pixel block being encoded. The inter predictor can generate prediction reference metadata, such as a prediction block size and a motion vector, to identify which portion(s) of which reference frame(s) are selected as a prediction source for the input pixel block.

[0039] The intra predictor can support intra (I) mode encoding. The intra predictor can search from pixel block data from the same frame as the pixel block being encoded, which provides the closest match to the input pixel block. The intra predictor can also generate a prediction mode indicator to identify which portion of the frame is selected as a prediction source for the input pixel block.

[0040] The mode decision unit (MDU) selects a hypothetical final coding mode from the outputs of the inter-frame predictor and the intra-frame predictor. The MDU outputs prediction data and coding parameters (e.g., selection of a reference frame, motion vectors, etc.) for the selected prediction mode. Typically, as described above, given a target bit rate, the MDU will select the mode that achieves the lowest distortion when decoding the video. Exceptions may occur when the coding mode is selected to satisfy other strategies followed by the coding system 200, such as satisfying specific channel behavior or supporting random access or data refresh strategies.

[0041] As discussed herein, embodiments of this disclosure propose a CCSO filtering system to improve the quality of input reconstructed samples. The filtering system can jointly utilize both local band and edge information, resulting in finer sample classification than previous CCSO filtering operations that only operate on edge information. For each band classification, the associated LUT can have an entry corresponding to the edge classification. Enhanced input sample classification can lead to better coding efficiency compared to the coarse-edge-only approach in CCSO. Furthermore, the filtering system improves luminance component filtering and separately improves chrominance component filtering.

[0042] Figure 3 This is a block diagram of a decoding system 300 according to an embodiment of the present disclosure. The decoding system 300 can decode data that has been encoded by an encoder (e.g., Figure 2 The decoding system 300 decodes frames of a video sequence encoded by the encoder shown. The decoding system 300 may include one or more encoding units 310-330, which are reversible from frames encoded by the encoder shown. Figure 2 The encoding operations are performed by encoding units 220-240. Therefore, continue. Figure 2 For example, the decoding system 300 may have a luminance decoder 310, a chrominance decoder 320, and a chrominance decoder 330. The decoding system 300 may also have a reconstruction 340 that generates a restored frame based on the color component data output by decoders 310-330. Figure 2 Similarly, although Figure 3 The decoding system 300 illustrates the application of the Y / Cb / Cr color space, but the principles of this disclosure can also be applied to other color spaces such as RGB, XYZ, etc.

[0043] Figure 3 An exemplary functional unit of a pixel block decoder that can be applied in a luminance decoder, a Cr decoder, and a Cb decoder 310-340 is shown. The decoder may be a pixel block decoder 350, a loop filter 360, a reference image cache 370, and a predictor 380 that operate under the control of a controller (not shown).

[0044] The predictor 380 can receive prediction metadata identifying a prediction mode and a prediction reference for an encoded pixel block, and can output prediction data from the reference picture cache 370 to the pixel block decoder 350. The pixel block decoder 350 can generate a recovered pixel block from the encoded pixel block data and the prediction data supplied by the predictor 380. The in-loop filter 360 can filter the reconstructed pixel block data that can be output from the decoder 350. For frames designated to be used as reference frames, the output of the in-loop filter can also be stored in the reference picture cache 370.

[0045] The pixel block decoder 350 can include an entropy decoder 352, an inverse quantizer 354, an inverse transform unit 356, and an adder 358. The entropy decoder 352 can perform entropy decoding to reverse the process performed by the entropy encoder 258 (see Figure 2 ). The inverse quantizer 324 can reverse the operation of the quantizer 256 of the pixel block encoder 250 (see Figure 2 ). Similarly, the inverse transform unit 326 can reverse the operation of the transform unit 252 (see Figure 2 ). They can use a quantization parameter QP and a transform mode M provided in the encoded video data stream. Since quantization can truncate data, the pixel block recovered by the inverse quantizer 324 will likely have encoding errors compared to the input pixel block presented to the pixel block encoder 250 of the encoder (see Figure 2

[0046] The adder 358 can reverse the operation performed by the subtractor 252 (see Figure 2 ). It can receive a predicted pixel block from the predictor 380, as determined by the prediction reference in the encoded video data stream. The adder 358 can add the predicted pixel block to the reconstructed residual values output by the inverse transform unit 356, and can output reconstructed pixel block data.

[0047] As described, the in-loop filter 360 can perform various filtering operations 362-366 on the recovered pixel block data, in synchronization with the in-loop filter 270 of Figure 2 . The in-loop filter can have a CCSO filter 364 that utilizes a pixel intensity classification of the reconstructed samples generated by the decoders 310-330. Thus, the decoders 310-330 can exchange data RS between them that represents the reconstructed samples generated by the decoders 310-330.

[0048] In one embodiment, the CCSO filter 364 can be part of a multi-filter filtering system 360 that can include other filters 362, 366. Such other filters can include, for example, a deblocking filter, an SAO filter, a CDEF, an ALF, and / or an LR filter.

[0049] ​The CCSO 364 presented herein can work in conjunction with other filtering units within the loop filter 360. For example, the CCSO 364 can accept input data from the deblocking filter and output filtered data to the CDEF. Alternatively, the CCSO 364 can accept input data from the CDEF and output filtered data to the LR filter. In another implementation, the CCSO can be placed in parallel to the CDEF; in this application, the input to the CCSO can be the reconstructed samples prior to the CDEF and the output of the CCSO can be applied to the reconstructed samples obtained from the CDEF. In another implementation, the CCSO 364 can accept input data from the deblocking filter and output filtered data to the ALF.

[0050] The reference picture cache 370 can store filtered frame data for use in subsequent prediction of other pixel blocks. The reference picture cache 370 can store decoded frames at the time of encoding for use in intra-prediction. The reference picture cache 370 can also store decoded reference frames.

[0051] As discussed, the predictor 380 can supply predicted blocks to the pixel block decoder 350. The predictor 380 can retrieve prediction data from the reference picture cache 370 as determined by prediction reference indicators supplied in the encoded video data stream.

[0052] Figure 4 A block diagram of a CCSO 400 according to embodiments of the present disclosure. The CCSO 400 can be applied to encoders and decoders as shown above Figure 2 and Figure 3 The CCSO 400 can include an intensity classifier 410, an index calculator 420, an offset calculator 430, and a filter 440. The CCSO 400 can receive a set of reconstructed samples and generate corrected samples therefrom. The intensity classifier 410 can classify the reconstructed samples and output an identifier of the determined classification to the offset calculator 430. The index calculator 420 can generate offset indices d0and d1from selected input samples that are input to the offset calculator 430. The offset calculator 430 can generate an offset from its input data, which can be output to the filter 440. The filter 440 can filter the reconstructed samples using the offset supplied by the offset calculator.

[0053] As shown, the filter 440 can filter the reconstructed samples r natThe operation is performed on reconstructed samples that are intrinsic to the color plane in which CCSO 400 operates. Thus, when CCSO 400 is applied to Cb chroma decoding, filter 440 can apply filtering to reconstructed Cb samples. Similarly, when CCSO 400 is applied to luma or Cr chroma decoding, filter 400 can apply its filtering to reconstructed luma or Cr chroma samples, respectively. Intensity classifier 410 can operate on reconstructed samples r orth The operation is performed on reconstructed samples that are intrinsic to the color plane in which CCSO 400 operates. Thus, when CCSO 400 is applied to Cb chroma decoding, filter 440 can apply filtering to reconstructed Cb samples. Similarly, when CCSO 400 is applied to luma or Cr chroma decoding, filter 400 can apply its filtering to reconstructed luma or Cr chroma samples, respectively. Intensity classifier 410 can operate on reconstructed samples r

[0054] In one embodiment, CCSO 400 can select filter 440 settings separately for each reconstructed sample r orth ( Figure 5 ) input from the orthogonal color plane. CCSO 400 can select filter settings based on an analysis of reconstructed samples present in spatial array 500 with respect to reconstructed sample r orth . In other embodiments, array 500 can include reconstructed samples from the same color plane (r orth ) as well as from the native color plane (r nat ) in which filter 440 is applied.

[0055] In another embodiment, intensity classifier 410 can classify reconstructed sample r orth ( Figure 5 ) according to its intensity. In a system in which sample values are represented numerically within a predetermined range R, range R can be partitioned into N intensity bands, where each band represents a fraction of overall range R. Intensity classifier 410 can generate a class ID that identifies the band to which reconstructed sample r orth belongs.

[0056] Intensity bands can be defined in a variety of ways. As one example, range R of sample values can be partitioned into N uniform-sized bands that collectively cover the entire range R of sample values. Value N can be predetermined within the system. Alternatively, N can be dynamically defined during system operation and signaled between encoder and decoder at predetermined intervals (e.g., in an APS, in a slice header, in a tile header, in a picture parameter set, in a sequence parameter set, in a video parameter set, or when N changes).

[0057] As another example, the range R of sample values can be partitioned into N bands of non-uniform size that collectively cover the entire range R of sample values. In this example, the number of bands and their sizes can be predetermined within the system. Alternatively, the number and sizes of the bands can be dynamically defined during operation of the system and signaled between the encoder and the decoder at predetermined intervals (e.g., in an APS, in a slice header, in a tile header, in a picture parameter set, in a sequence parameter set, in a video parameter set, or as they change). For example, a frame header can include a flag that, in one state (e.g., true), indicates the use of a set of uniformly sized sample intensity bands and, in another state (e.g., false), indicates the use of a set of non-uniformly sized sample intensity bands.

[0058] In another implementation, the range R of sample values can be partitioned into N segments from the minimum intensity value to the maximum intensity value, where each group of M adjacent segments is grouped together to form a band of CCSO 400. In this implementation, N and M can be positive integers, where N > M.

[0059] In another implementation, the band information of a reconstructed sample r orth may be determined by deriving an index band_idx as follows:

[0060] band_idx = r orth >> (bitDepth - maxBandLog2)

[0061] where bitDepth corresponds to the bit depth of the reconstructed sample in the applicable color plane, and maxBandLog2 is the log2 of the maximum allowed number of bands.

[0062] In one implementation, the band information of a reconstructed sample r orth may be computed at the sample level. For example, the intensity value of each reconstructed sample r orth may be used to check which band the current reconstructed sample belongs to.

[0063] In another implementation, the band information of a reconstructed sample r orth may be computed at the block level. For example, the average intensity value of the pixel block to which a reconstructed sample r orth belongs can be used to assign the reconstructed sample r orth to a sample intensity band. The block can be a super block (SB), a coding tree unit (CTU), a coding block, a partition unit, a transform unit, a prediction unit, a filter unit, etc.

[0064] In one implementation, the band information of a reconstructed sample r orth may be derived from the band information of a reconstructed sample r orthits own value calculation, which value represents the sample content in a single color plane. In another implementation, the band information of a reconstructed sample r orth may be calculated not only from its own value calculation, but also from the values of the reconstructed samples r orth of other color components related to the reconstructed sample r nat . Typically, due to the chroma format used by the system (e.g., 4:4:4 vs. 4:2:0, etc.), there can not be a perfect registration between all luma samples and all Cb and Cr chroma samples. In such cases, it can be useful to downscale the reconstructed luma samples to match the reconstructed Cb and Cr chroma samples. Alternatively, it can be useful to upscale the reconstructed Cb and Cr chroma samples to match the reconstructed luma samples.

[0065] In one implementation, the reconstructed samples r orth obtained prior to the application of the other loop filter can be used to calculate the band information. Alternatively, the reconstructed samples r orth obtained after the other loop filter can be used to calculate the band information.

[0066] As yet another example, CCSO 400 can employ both uniform-sized and non-uniform-sized sample intensity bands during operation. The encoder Figure 2 ) can signal the use of a certain type of sample intensity band (e.g., whether a uniform-sized band or a non-uniform-sized band is used) in the channel data, such as in an APS, in a slice header, in a tile header, in a picture parameter set, in a sequence parameter set, in a video parameter set.

[0067] In one implementation, intensity classifier 410 operates to generate its class IDs at a granularity that can vary for different reconstructed samples. For example, the classification can operate on a per-pixel block basis, on a per-slice basis, or on a per-frame basis. The granularity can change at predetermined intervals in the encoding. The encoder can signal its selection of features that the intensity classifier 410 is using in a high-level syntax (HLS) structure, such as an APS, a slice header, a tile header, a frame header, a PPS, an SPS, a VPS, etc.

[0068] As discussed, index calculator 420 can generate indices d0, d1 for offset calculator 430 from the analysis of reconstructed samples r orth in the orthogonal color planes. nat

[0069] Figure 6 One such implementation is shown in FIG. 6, where reconstructed samples r orth 610 in the orthogonal color planes​nat In the vicinity, the filtered reconstructed samples are shown with neighboring samples 615-690. Index calculator 420 can operate according to a plurality of 3-tap filters, where filter inputs (referred to as p0 and pi, respectively) are selected according to different filter directions. In one aspect, six filter combinations can be used, such as described in Table 1 below:

[0070]

[0071] Table 1

[0072] It is contemplated that system designers can customize the number of different directions and inputs thereto to suit their respective application needs.

[0073] In one aspect, an encoder can select the direction to apply during encoding. The direction can be changed at a predetermined level of the encoding hierarchy, such as at the Cb / Cr color component level. The encoder can signal its selected direction in the encoded bitstream in an appropriate flag. For example, a variable filter sup can be exchanged, which indicates the selected filter for each corresponding color component. For Figure 6 For the configuration shown, the filter sup syntax element can be signaled using 3 bits.

[0074] For each filter direction, CCSO 400 can compute a pair of delta values m0, m1, denoted as m j = r orth - p j where j = 0, 1. Thereafter, index calculator 420 can generate quantized values d0, di from the delta values m0, m1, as follows:

[0075] di = -1, if m < -thr;

[0076] di = 0, if -thr <= m <= thr; and

[0077] di = 1, if m > thr.

[0078] where thr is the quantization step size, which can be a predetermined value set by the encoder. For example, thr can take on values of 8, 16, 32, or 64. Further, the thr value can be dynamically set during operation, and can be signaled from the encoder to the decoder in the encoded bitstream.

[0079] In one embodiment, offset calculator 430 can be implemented as a multi-dimensional lookup table (LUT) that is indexed by the class ID provided by intensity classifier 410 and the index output by index calculator 420. Prior to runtime operation of CCSO 400, the LUT can be pre-populated with offset values that are appropriate for the samples being processed. During runtime operation, CCSO 400 can read out the offset values from the LUT and supply them to filter 440.

[0080] The LUT can have sub-tables corresponding to different class IDs generated by intensity classifier 410. In one embodiment, intensity classifier 410 can classify the input samples according to the sample intensity band information and based on edge information. In this regard, the class ID can have a first component (i) that represents the band index of the sample and a second component (j) that represents the edge index of the sample. These (i,j) pairs can correspond to the associated offset values s ij used by filter 440. For example, when there are N bands and N LUTs, each band can have an associated LUT with a different set of entries. An example value of N can be 8, 16, and 32.

[0081] In one embodiment, the LUTs can be derived dynamically. For example, the LUTs can be signaled at the sequence level, frame level, tile level, CTU / SB level, or pixel block level of the coding hierarchy.

[0082] In another embodiment, offset calculator 430 can have N LUTs in CCSO 400, each associated with a corresponding pixel intensity band. Each LUT can have M entries, each storing an offset value. In this embodiment, the encoder can signal the M*N offset values signaled in the HLS (e.g., as part of the APS, slice header, tile header, frame header, PPS, SPS, VPS, etc.).

[0083] In another embodiment, offset calculator 430 can form a parametric model of the offset defined by parameters. For example, samples in the luma and chroma components of CCSO 400 can be used to derive a linear relationship s = A*y + B*c, where A and B are coefficients to be derived, y refers to the luma sample value, c refers to the Cb or Cr sample value under evaluation, and s is the output offset value used in the filtering process of CCSO 400. During operation, the encoder can estimate the values of A and B that minimize distortion during application of CCSO 400, which can be signaled to the decoder. Alternatively, the values of A and B can be implicitly derived from the neighborhood of the reconstructed sample r orth with respect to the sample being processed, which can reduce the signaling overhead within the system.

[0084] Another example of using a parametric model can employ reconstructed samples in all three color planes in a non-linear relationship s = f0(y i )+ f1(cb i )+ f2(cr i ), where f0, f1, and f2 are functions that map input values to output values, y i refers to luma sample values, cb i / cr i refers to Cb / Cr sample values, and s is an output offset value used in the filtering process of CCSO 400. During operation, an encoder can estimate the functions f0, f1, and f2 that minimize distortion during application of CCSO 400 from a set of candidate functions F. The encoder can identify its selection of f0, f1, and f2 functions to a decoder in channel signaling. Alternatively, the f0, f1, and f2 functions can be implicitly derived from a neighborhood of reconstructed samples with respect to the sample being processed, which can reduce signaling overhead within the system.

[0085] When the parametric models are dynamically derived, they can be signaled at the sequence level, frame level, tile level, CTU / SB level, or pixel block level of the coding hierarchy.

[0086] In one embodiment, when the offset values in a LUT or parametric model are dynamically formed during run-time operation, the encoder does not need to signal the entire LUT or parametric model. In some implementations, the encoder signals some of the offset values of the LUT and lets the decoder derive the remaining offset values from the signaled offset values, or it can be sufficient that the remaining offset values are fixed values. For example, consider a system that employs a nine-entry lookup table corresponding to a given class ID, the encoder can signal a subset of offset values s0, s1, s2, s3, and s4 for a selected combination, and the decoder can form the complete table by inverting the offset values to the following values:

[0087]

[0088] As another example, the encoder can signal a subset of offset values s0, s1, s2, s3, and s4 for a selected combination, and the decoder can form the complete table by inverting the offset values to the following values:

[0089]

[0090] It is contemplated that during implementation, system designers can customize the derived values to suit their individual application needs.

[0091] In another embodiment, a single LUT indexed by the class ID output by the sample classifier 410 and the indices d0, d1 output by the index calculator 420 can be formed. For example, in an implementation where the class ID is generated as a single-bit binary value, the LUT takes the form:

[0092]

[0093] It is contemplated that during implementation, the system designer can customize the number of class IDs and the offset values to suit their individual application needs.

[0094] In another embodiment, for the CCSO 400, a filter scaling factor x is used. The scaling refers to scaling the offset values produced by the CCSO 400 by x times. The value of x can be signaled at the sequence level, frame level, tile level, CTU / SB level, or coding block level. In this case, the offset values derived by the offset calculator 430 can be scaled by the x factor received over a channel.

[0095] In one embodiment, the class ID can represent a sample intensity band, and the offset calculator 430 can have a LUT corresponding to each available band, where each LUT stores values sijassociated with its respective band i and entry j (entry j corresponds to a respective combination of d0and d1). ij In this embodiment, the input sample can be classified using band information, output as a band index, denoted i, and it can also be classified using edge information, output as an edge index, denoted j. Each (i, j) pair has an associated offset value sijused in the filtering process of the CCSO 400.

[0096] In another embodiment, the current color representation (e.g., matrix coefficients that can be associated with a video) can be used to adjust the offset values of the CCSO 400 based on the values of luminance and / or Cb and Cr. For example, certain relationships and constraints of the Cb and Cr components can be derived based on the color representation. Such constraints can be taken into account to adjust the base offset values stored in the LUT according to the values of the signal, thereby avoiding the use of some offsets in certain regions.

[0097] The filter 440 can apply offset filtering to the reconstructed samples using the offset values obtained from the offset calculator 430. The filter 440 can generate filtered reconstructed samples, which represent the output of the CCSO 400. The CCSO 400 can have a predetermined number F of taps.

[0098] In one embodiment, CCSO 400 can have a number of taps F that can be signaled by an encoder in the channel. Similarly, CCSO 400 can have a number of candidate filter shapes that can be selected according to signaling provided in the channel. The filter shape can be selected and signaled at sequence level, frame level, tile level, CTU / SB level, or coding block level. Alternatively, the number of taps and / or filter shape can be selected according to the class ID output by intensity classifier 410.

[0099] In one embodiment, filter 440 can apply a filter direction based on the current luma and chroma relationship. The relationship includes, but is not limited to, the chroma sub-sampling format and the location of the chroma sample relative to the luma sample.

[0100] In another embodiment, filter 440 can apply F filter taps according to the current luma and chroma relationship. The relationship includes, but is not limited to, the chroma sub-sampling format and the location of the chroma sample relative to the luma sample.

[0101] In one embodiment, the selection of the number of taps, filter shape, and / or offset values of CCSO 400 can be entropy coded using a universal variable length code (“UVLC”). Alternatively, one or more of these values can be signaled using fixed length coding.

[0102] During the run-time operation of the encoder and decoder, the CCSO can operate in a synchronized state. Although the operational settings of CCSO 400 can change dynamically during operation, the operational settings present at CCSO 400 when a given pixel block is encoded should also be present at CCSO 400 when the same pixel block is decoded. In one embodiment, the encoder and decoder can exchange signaling as appropriate to synchronize the state of their CCSO.

[0103] In one embodiment, the number of bands for each color component can be signaled. For example, the encoder can provide a value max_band_log2 that represents the log2 of the maximum allowed number of bands for each color component. It can be convenient to signal max_band_log2 using 2 bits, which would allow the number of bands to be identified as one of 1, 2, 4, and 8.

[0104] In another embodiment, the selection of the filter to be used for each color component can be signaled. As discussed above with respect to Figure 6 CCSO 400 can support several different filters. The encoder can exchange a variable filter_sup that indicates the selected filter for each corresponding color component. It can be convenient to signal filter_sup using 3 bits.

[0105] In another embodiment, an identifier of a sample intensity threshold can be signaled. For example, the encoder can supply a thr_idx value that represents an index of a threshold value for each color component. In one application, it can be convenient to use 2 bits to signal thr_idx, which identifies a sample intensity threshold of 8, 16, 32, or 64.

[0106] In another embodiment, a filter offset value can be signaled. In one application, it can be convenient to use 3 bits to signal a filter offset value, which represents a selection of a predetermined offset value from the set {-10, -7, -3, 0, 1, 3, 7}.

[0107] In one embodiment, a flag can be provided to signal when CCSO is enabled or disabled. For example, it can be convenient to provide a 1-bit ccso_enable signal for each color component, which indicates whether CCSO filtering is enabled for the color component. In another aspect, it can be convenient to provide a 1-bit ccso_blk_ctr signal for each 128x128 superblock, which indicates whether CCSO filtering is enabled for the superblock.

[0108] In one embodiment, the offset values formed by offset calculator 430 can be formed by the encoder through a training operation that derives the offset values from a training sequence based on an estimate of the values that minimize distortion when CCSO 400 is applied. The derived offset values can be communicated to the decoder by a variety of techniques. Figure 1 For example, the derived CCSO values can be explicitly communicated at the sequence level, frame level, or tile level in a communication hierarchy. In one implementation, the encoder can communicate a single set of information signaled per sequence or per frame or per tile, with the Y / Cb / Cr components sharing the signaled information.

[0109] In one embodiment, the training can derive a linear relationship s = A*y + B*c, where A and B are coefficients to be derived, y is a reference to a luma sample value, c is a reference to a Cb or Cr sample value, and s is an output offset value used in the filtering process of CCSO 400. The coefficient values can be signaled from the encoder to the decoder, which can use the coefficient values to derive the offset values for its CCSO 400.

[0110] In another implementation, the samples in all three color components of CCSO 400 can use a non-linear relationship s = f0(y) + f1(cb) + f2(cr), where f0 / f1 / f2 are derived functions that map input values to output values, y refers to luma sample values, cb / cr refer to Cb / Cr sample values, and s is the output offset value used in the filtering process of CCSO 400. Function identifiers can be signaled from the encoder to the decoder, which can use these function identifiers to derive the offset values for its CCSO 400.

[0111] In another implementation, the current color representation (e.g., matrix coefficients that can be associated with the video) can be used to adjust the offset values of CCSO 400 based on the values of y and / or Cb and Cr. For example, certain relationships and constraints of the Cb and Cr components are derived based on the color representation. Such constraints can be considered to adjust the offset values of CCSO 400 according to the values of the signal, so as to avoid using some offsets in certain regions.

[0112] In one implementation, when training the LUTs for CCSO 400 at the encoder, the initial block-level on / off status of CCSO 400 can be carried over from the status of other in-loop filters of the current frame or previous encoded frames to achieve faster encoder convergence. For example, the status of SAO filter, ALF, CCALF, LR filter, or CDEF can determine the status of CCSO 400 during the training.

[0113] In one implementation, the encoder can signal the number (L) of LUTs that are in effect in CCSO 400. In one example, L is a fixed positive integer value. In another example, L can change during operation, and the change of L can be signaled at sequence level, frame level, or tile level or CTU / SB level or coding block level.

[0114] In one implementation, CCSO 400 can have L LUTs, each associated with a pixel intensity band of CCSO 400. Each LUT can have M entries, each of which represents an offset value. In this implementation, there will be M*L offset values, which can be signaled in HLS (APS, slice or tile header, frame header, PPS, SPS, VPS). For a LUT, the LUT entries can have the same offset value for multiple of the pixel intensity bands, in which case the LUT can have less than M entries. As discussed, the LUTs can be one or more parametric models defined by parameters.

[0115] In another implementation, the following equation can be used when deriving the entries of the CCSOLUT given an input sample: m = r1 - ((a * p0 + b * p1) » k), where k is a positive integer. In an aspect, the values of a, b, and k can be signaled in the HLS (APS, slice or tile header, frame header, PPS, SPS, VPS). Alternatively, these values can be predefined between the encoder or the decoder.

[0116] In another implementation, multiple sets of CCSO information can be signaled at the first level of the protocol hierarchy (e.g., sequence level). At a lower level of the protocol hierarchy (e.g., frame level, slice level, or tile level), the encoder can signal an index of the set of CCSO information to be used.

[0117] In another implementation, when the filter shape of the CCSO 400 is entropy coded (e.g., signaled using UVLC), the signaling of other information of the CCSO 400 can be adjusted based on the filter shape. In one example, the UVLC code 0 of the filter shape can mean that the CCSO 400 is not used (e.g., off); in this case, there is no need to signal the other information of the CCSO 400.

[0118] In another implementation, the CCSO LUT can be signaled once per group (N frames or N blocks). In the first frame (or block) of the group, the LUT can be signaled. For the remaining frames or blocks of the group, if the encoder changes the CCSO LUT, the encoder can signal the data to change the LUT, e.g., by completely replacing the LUT or by signaling the changes to the previously used LUT.

[0119] In another implementation, the block-level status (e.g., on or off) of the CCSO 400 can be inherited from the block-level status of other in-loop filters 272, 276 (e.g., SAO filter, ALF, CCALF, LR filter, or CDEF) in the in-loop filter system 270.

[0120] In another implementation, the on / off status of the CCSO 400 can be derived from other coding information of the block or / and its neighboring blocks, such as the transform type, prediction mode, number of coefficients, etc. In this way, the signaling overhead of the CCSO process can be reduced.

[0121] In another implementation, the encoder can signal an array of CCSO offset values, and for each combination, the encoder can signal an index of the offset value s that represents the location of the offset value s in the CCSO array. Such offset values and / or indices can be signaled at the sequence level, frame level, tile level, or block level. In another implementation, for each combination, the offset value s is signaled at the sequence level, frame level, tile level, or block level.

[0122] It is proposed herein that the technique of pixel intensity classification in classifying / grouping input samples can also be applied to other loop filtering methods. In one implementation, the intensity classification technique can be used to constrain the CDEF edge direction search process. Typically, the CDEF direction search operates on reconstructed pixels just after the deblocking filter. Since the decoder can use these pixels, the direction does not need to be signaled. The search is performed on 8x8 blocks, and for each block, the direction that best matches the pattern in the block is determined by minimizing the sum of squared differences (SSD) between the quantized block and the nearest directional block. The identified direction is then used to select the filter shape and filter coefficients. In this application, the class IDs generated by the intensity classifier 410 can also be employed to select the filter shape and filter coefficients for the CDEF filter. Using such a technique, the CDEF can expand the set of candidate directional blocks from eight to a larger number, such as 16, 32, or 64 candidate blocks, that incorporate both the direction and the class ID.

[0123] The CDEF works by identifying the direction of each block, then performing adaptive filtering along the identified direction, and a lesser degree of filtering along a direction that is rotated 45 degrees from the identified direction. The CDEF direction search is typically performed on reconstructed pixels just after the deblocking filter is applied, on 8x8 blocks, and finds the direction that best matches the pattern in the block by minimizing the sum of squared differences (SSD) between the quantized block and the nearest directional block. Figure 7 An exemplary set of eight candidate directional blocks is shown. In one implementation, the pixel intensity classification method can be employed to select the filter shape and filter coefficients of the CDEF filter. The filter shape and filter coefficients can be selected based on the pixel intensity analysis discussed above ( Figure 4 ). In this way, the CDEF can use the pixel intensity classification analysis described herein to select among its candidate modes, either working alone or in combination with the CDEF filter's conventional SSD technique.

[0124] CDEFs can use a nonlinear low-pass filter that is designed to remove encoding artifacts without blurring sharp edges. This is achieved by selecting filter tap positions based on the identified direction, but also by preventing excessive blurring when applying the filter across an edge. The latter is achieved using a nonlinear low-pass filter that does not emphasize pixels that differ too much from the pixel being filtered. CDEFs define a primary tap and a secondary tap. The primary tap follows the direction d and the weights shown in the example of Figure 8 For the primary tap, the weights alternate every other intensity, so the weights for intensities 1, 3, 5, etc. are different from the weights for intensities 2, 4, 6, etc. The secondary tap forms a cross that is oriented at 45 degrees to the direction d, with weights as shown in the example of Figure 9

[0125] In another implementation, a pixel intensity classification technique can be used in the ALF block classification process. The normal ALF process classifies an input block based on the quantized values of the directionality D and activity of the block. In one aspect, the pixel intensity classification analysis discussed above ( Figure 4 ) can be used or combined with D and used to identify the class of the current block. For example, the pixel intensity analysis can provide an estimate of the pixel intensity range, denoted P. The classification index C can be derived from D, and P. Thereafter, the ALF process can be applied using the classification index.

[0126] The foregoing discussion has described the operation of various aspects of the disclosure in the context of video encoders and decoders. These components are often provided as electronic devices. Video decoders and / or controllers can be embedded in integrated circuits, such as application specific integrated circuits, field programmable gate arrays, and / or digital signal processors. Alternatively, they can be embedded in computer programs that are executed on camera devices, personal computers, notebook computers, tablet computers, smart phones, or computer servers. Such computer programs are often stored in physical storage media such as electronic, magnetic, and / or optical-based storage devices, where they are read into processors and executed. Decoders are often packaged in consumer electronics devices, such as smart phones, tablet computers, game systems, DVD players, portable media players, etc.; and, they can also be packaged in consumer software applications, such as video games, media players, media editors, etc. And, of course, these components can be provided as hybrid systems that allocate functionality as desired between special-purpose hardware components and programmed general-purpose processors.

[0127] Video encoders and decoders can exchange video over a channel in a variety of ways. They can communicate with each other via communication and / or computer networks, as Figure 1 ​In other applications, the video encoder can output the video data to a storage device, such as a magnetic or optical storage medium or an electronic storage medium, which can be provided to the decoder at a later time. In such applications, the decoder can retrieve the encoded video data from the storage device and decode it.

[0128] Numerous embodiments of the present application are specifically illustrated and / or described herein. However, it will be understood that modifications and variations of the present application are covered by the above teachings and within the purview of the appended claims without departing from the spirit and intended scope of the present application.

Claims

1. A filtering system, comprising: A filter having an input for a filter offset and an input for samples reconstructed from coded video data representing a first color component of a source video to which the filter operates in a multi-color color space, and an output for the filtered samples. An intensity classifier having input to samples reconstructed from coded video data representing a source video in the multi-color color space that is different from the first color component, and having output to data representing the intensity classification of the samples input to the intensity classifier. An offset calculator having a first input to the intensity classification data from the intensity classifier and an output for the filter offset input to the filter.

2. The filtering system of claim 1, wherein the intensity classification data from the intensity classifier represents the classification of a reconstructed sample into one of a plurality of sample intensity bands.

3. The filtering system according to claim 2, wherein the sample intensity band is defined by signaling, and the signaling is provided by the video encoder that generates the encoded video data.

4. The filtering system according to claim 3, wherein the signaling identifies the number of sample intensity bands.

5. The filtering system of claim 3, wherein the signaling identifies the sample intensity bands as uniformly spaced.

6. The filtering system according to claim 3, wherein the signaling identifies breakpoints between the sample intensity bands.

7. The filtering system of claim 1, wherein the intensity classification data from the intensity classifier represents the classification of a reconstructed sample block into one of a plurality of sample intensity bands.

8. The filtering system according to claim 1, wherein, The reconstructed samples are received from the output of the second filter of the video coding system, and The intensity classification data is derived from the filtering parameters of the second filter.

9. The filtering system of claim 1, wherein the offset calculator is an N-way lookup table having a path corresponding to each of a plurality of candidate intensity categories.

10. The filtering system of claim 9, further comprising an index calculator having an input for a reconstructed sample representing the second color component and an output for an index identifier, wherein the offset calculator has a second input for the index identifier output from the index calculator.

11. The filtering system of claim 9, wherein at least one entry of the lookup table is provided by the video encoder that generates the encoded video data.

12. The filtering system of claim 1, wherein the offset calculator is a computer that calculates the filter offset according to a parameter model, and the parameters of the parameter model are provided by a video encoder that generates the encoded video data.

13. A filtering method, comprising: The intensity of the first input sample is classified. The first input sample is reconstructed from coded video data representing a first color component of the source video in a multi-color color space. This first color component is different from the second color component to be filtered in the multi-color color space. The filter offset is estimated based on the classification strength of the first input sample, and The second input sample is filtered based on the estimated filter offset, the second input sample being reconstructed from the coded video data representing the second color component of the source video.

14. The filtering method according to claim 13, wherein the classification separately classifies each reconstructed sample into one of a plurality of sample intensity bands.

15. The filtering method of claim 14, wherein the sample intensity band is defined by signaling provided by a video encoder that generates the encoded video data.

16. The filtering method of claim 15, wherein the signaling identifies the number of sample intensity bands.

17. The filtering method of claim 15, wherein the signaling identifies the sample intensity bands as uniformly spaced.

18. The filtering method according to claim 15, wherein the signaling identifies breakpoints between the sample intensity bands.

19. The filtering method according to claim 13, wherein the classification classifies the reconstructed sample into one of a plurality of sample intensity bands on a pixel-by-pixel basis.

20. The filtering method of claim 13, wherein the estimation includes reading an offset value from a lookup table using the classification strength as an index.

21. The filtering method according to claim 20, wherein the lookup table is an N-way lookup table, the N-way lookup table having a path corresponding to each of the plurality of candidate intensity classifications.

22. The filtering method of claim 20, wherein at least one entry of the lookup table is provided in signaling by the video encoder that generates the encoded video data.

23. A computer-readable storage medium storing a computer program, said computer program including instructions that, when executed by one or more processors, cause said one or more processors to perform the filtering method according to any one of claims 13 to 22.

24. A computer program product comprising a computer program, the computer program including instructions that, when executed by one or more processors, cause the one or more processors to perform the filtering method according to any one of claims 13 to 22.

Images