Adaptive quantizer design for video coding

Abstract

To provide improved video encoding and decoding methods including techniques for adaptively deriving quantization step sizes using quantization step size table templates.SOLUTION: In the quantization selection system 400, the video encoder and decoder may quantize the video according to multiple quantization step size table templates, where each table 410.0, 410.1,., 410. n provides its own relationship between index values (q _ idx) and quantization parameter adjustments (q _ Step). During encoding and decoding operations, the encoder and decoder can adaptively select from the quantization step size values in the tables 410.0, 420.1,., 420. n to suit the immediate encoding needs.SELECTED DRAWING: Figure 4

Description

[Background technology]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Patent Application No. 17 / 234,596, filed April 19, 2021, which claims U.S. Provisional Patent Application No. 63 / 042,451, filed June 22, 2020, which further claims the benefit under 35 U.S.C. § 119(e) of U.S. Provisional Patent Application No. 63 / 147,163, filed February 8, 2021, the contents of which are incorporated herein by reference in their entireties.

[0002] The techniques proposed herein generally relate to the compression and decompression of image and video data, and in particular to quantizer designs in video coding. The techniques are not limited to ITU-T / MPEG video codecs such as the High-Efficiency video coding (HEVC) or Versatile video coding (VVC) standard, or AOMedia video codecs such as the AV1 or AV2 standard. In describing the techniques proposed herein, examples based on the current AV1 and draft VVC specifications are used, but all proposed techniques can be applied to any other quantizer designs and to different color components (luma / chroma / alpha / depth), bit depths, etc.

[0003] The proposed techniques can also be applied to the field of neural network model compression, i.e., compression of neural networks for multimedia content description and analysis, which is being explored in MPEG activities. In model coding systems, neural network parameters (e.g., model architecture, weights, biases, and other hyperparameters) are compressed instead of video content. Some existing techniques apply quantizer design as one of their sub-methods to reduce the amount of parameter information that needs to be entropy coded, and all the techniques proposed herein can also be applied to such quantizer design.

[0004] Introduction In the AV1 specification, the quantization index (q_idx) has a data range of 0 to 255 (or a total of 256). It is observed that AV1 has a finer number of step sizes (256) than MPEG video coding standards such as MPEG4-AVC, HEVC, and VVC, which have fewer quantization indices, e.g., 52 and 64 for AVC / HEVC and VVC, respectively, for 8-bit content with an increment of six additional quantization indices for each increment of bit depth precision for the color components. However, it is observed that the mapping of these q_idx to quantization step sizes (q_step) in AV1 has a lower slope compared to AVC / HEVC or VVC, as shown in Figure 7. For example, in VVC, q_step grows exponentially (~pow(q_idx / 6,2), i.e., doubles in size every six indices). As a result, q_step is relatively very large at higher q_idx values, resulting in a much lower bitrate. In contrast, the AV1 step size varies approximately linearly at lower q_idx and increases exponentially at higher q_idx, resulting in a relatively smaller q_step at higher q_idx. It is also worth noting that AV1 has separate mappings from q_idx to q_step for AC and DC coefficients, as shown in Figure 7.

[0005] AV1 specification and quantization method design. The syntax element "base_q_idx" is coded in the sequence / frame header to specify the base frame q_idx. base_q_idx is fixed-length coded with 8 bits. base_q_idx serves as the base offset for all AC coefficients. For coefficients other than luma AC coefficients, an additional delta offset can be signaled in the bitstream, as described below. Final_q_idx is calculated as follows:

number

[0006] Additionally, the syntax element "diff_uv_delta" specifies whether the same delta is used for the chroma U and V components. This q_idx is an index into a lookup table of quantization step sizes.

[0007] These techniques pose several problems.

[0008] Reaching Lower Bitrates: As explained above, in VVC, q_step increases exponentially to very large values ​​at higher q_idx, resulting in much lower bitrates. In contrast, the AV1 step size varies approximately linearly at lower q_idx and then increases exponentially at higher q_idx, resulting in a relatively smaller q_step at higher q_idx. This is believed to be one of the reasons why AV1 does not achieve low bitrates for some sequences, even when operating at high q_idx. Also, the DC step size has a lower slope compared to AC, maintaining greater precision in the DC coefficient. Unlike MPEG standards such as VVC and HEVC, which provide flexible signaling of scaling matrices that can extend the step size per coefficient position, the AV1 quantization matrix is ​​fixed and does not have the ability to further increase the DC value. This also negatively impacts the ability to reach extremely low bitrates, since DC is the most important component in these bitrates. It is argued that this observation also depends on the frame content and configuration (intra / inter, predicted, etc.).

[0009] Flexible and content-adaptive variation of quantization step size: It can be observed that coarser or larger quantization step sizes may result in significant loss of precision for some content characteristics (flat regions, fine details, etc.), while coarser quantization may be required to achieve lower rates for some other characteristics in the content (complex textures, water, etc.). [Brief explanation of the drawings]

[0010] [Figure 1] 1 illustrates a simplified block diagram of a video distribution system according to one aspect of the present disclosure. [Figure 2] FIG. 1 is a functional block diagram of an encoding system according to one aspect of the present disclosure. [Figure 3] FIG. 1 is a functional block diagram of a decoding system according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a functional block diagram of a quantization selection system according to an embodiment of the present disclosure. [Figure 5-1] FIG. 10 is a functional block diagram of a quantization selection system according to another embodiment of the present disclosure. [Figure 5-2] FIG. 10 is a functional block diagram of a quantization selection system according to another embodiment of the present disclosure. [Figure 6] 1 illustrates a method according to an embodiment of the present disclosure. [Figure 7] The quantizer values ​​supported by several video coding protocols are compared. [Figure 8] 1 illustrates a method according to an embodiment of the present disclosure having a quantization step mode. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiments of the present disclosure provide techniques for adaptively deriving quantization step sizes using quantization step size table templates. These techniques can be used in video encoding and decoding devices, as described herein. The described quantization techniques provide finer control over quantization, particularly at higher degrees of quantization, with more flexible quantization step sizes. This can result in improved overall compression quality, especially at lower bit rates. In some embodiments, other coding parameters, such as in-loop filtering parameters, can be derived based on the more flexible quantization step sizes. Figure 1

[0012] FIG. 1 illustrates a simplified block diagram of a video distribution system 100 according to one embodiment of the present disclosure. The system 100 may include multiple terminals 110, 120 interconnected via a network 130. The terminals 110, 120 can encode video data for transmission to a destination via the network 130. Thus, a first terminal 110 can capture video data locally, encode the video data, and transmit the encoded video data to a destination terminal 120 via the network 130. The receiving terminal 120 can receive the encoded video data, decode it, and render it locally, e.g., on the display of the terminal 120. When the terminals are engaged in a two-way exchange of video data, the terminal 120 can capture video data locally, encode it, and transmit the encoded video data to the destination terminal 110 via the network 130. The receiving terminal 110 can receive the encoded video data transmitted from the terminal 120, decode it, and render it locally, e.g., on its own display. The process described can work for both frame and field-frame encoding, but for simplicity, this discussion describes the technique in the context of full frames.

[0013] Video encoding system 100 can be used in a variety of applications. In a first application, terminals 110, 120 can support real-time, two-way exchange of encoded video and establish video conferencing sessions between them. In another application, terminal 110 can encode pre-generated video (e.g., television or movie programs) and store the encoded video for distribution to one or, more likely, many downloading clients (e.g., terminal 120). Thus, the encoded video can be live or pre-generated, and terminal 110 can function as a media server that distributes the encoded video according to a one-to-one or one-to-many distribution model. For the purposes of this discussion, the type of video and video distribution scheme is not important unless otherwise specified.

[0014] 1, terminals 110, 120 are shown as a smartphone and a tablet computer, respectively, but the principles of this disclosure are not so limited. Aspects of this disclosure also apply to various types of computers (desktop, laptop, and tablet computers), computer servers, media players, dedicated video conferencing equipment, and / or dedicated video encoding equipment.

[0015] Network 130 represents any number of networks that convey encoded video data between terminals 110 and 120, including, for example, wired and / or wireless communication networks. Communication network 130 may exchange data over circuit-switched or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of the network are not important to the operation of the present disclosure unless otherwise specified. Figure 2

[0016] 2 is a functional block diagram of an encoding system 200 according to one embodiment of the present disclosure. The system 200 may include a pixel block encoder 210, a pixel block decoder 220, a frame buffer 230, an in-loop filter system 240, a reference picture buffer 250, a predictor 260, a controller 270, and a syntax unit 280. The encoding system 200 may operate on predetermined units of a video frame, conveniently referred to as "pixel blocks," and differentially encode the pixel blocks according to prediction techniques. Thus, a frame of video to be encoded may be parsed into pixel blocks, which the pixel block encoder 210 processes pixel block by pixel block. The pixel block encoder 210 may present the encoded pixel block data to a syntax unit 280, which formats the encoded pixel blocks into a transmission syntax that conforms to a governing encoding protocol.

[0017] The pixel block decoder 220 can decode the coded pixel block data and generate decoded pixel block data therefrom. The frame buffer 230 can generate reconstructed frame data from the decoded pixel block data. The in-loop filter 240 can perform one or more filtering operations on the reconstructed frame. For example, the in-loop filter 240 can perform deblocking filtering, sample adaptive offset (SAO) filtering, adaptive loop filtering (ALF), maximum likelihood (ML)-based filtering schemes, de-ringing, band removal, sharpening, resolution scaling, etc. The reference picture buffer 260 can store the filtered frame so that it can be used as a source of prediction for later received pixel blocks.

[0018] The pixel block encoder 210 may include a subtractor 212, a transform unit 214, a quantizer 216, and an entropy encoder 218. The pixel block encoder 210 may accept pixel blocks of input data at the subtractor 212. The subtractor 212 may receive a predicted pixel block from the predictor 260 and generate therefrom an array of pixel residuals representing the difference between the input pixel block and the predicted pixel block. The transform unit 214 may apply a transform to the sample data output from the subtractor 212 to convert the data from the pixel domain to the domain of transform coefficients. In some scenarios (e.g., when operating with a high dynamic range), the input may be reshaped or an adaptive scheme may be applied to adjust the content transfer characteristics before the transform unit 214 and / or the subtractor 212. Such adaptation may be either simple scaling based on a remapping function or more sophisticated pixel manipulation techniques. The quantizer 216 may perform quantization of the transform coefficients output by the transform unit 214 according to a quantization parameter qp. The quantizer 216 can apply either a uniform or a non-uniform quantization parameter, which can vary across predetermined positions in the block of coefficients output from the transform unit 214. The entropy encoder 218 can reduce the bandwidth of the output of the coefficient quantizer by encoding the output, for example, with variable length codewords or using a context-adaptive binary arithmetic encoder.

[0019] Transform unit 214 may operate in various transform modes determined by controller 270. For example, transform unit 214 may apply a discrete cosine transform (DCT), a discrete sine transform (DST), a Walsh-Hadamard transform, a Haar transform, a Daubechies wavelet transform, etc. In one aspect, controller 270 may select the coding mode to be applied by transform unit 215, configure transform unit 215 accordingly, and / or signal, either explicitly or implicitly, the coding mode M of the encoded video data.

[0020] Quantizer 216 may operate according to a quantization parameter qp determined by controller 270. Techniques for generating the quantization parameter are described below. Controller 270 may provide data representing its quantization parameter selection to syntax unit 280.

[0021] The entropy encoder 218, as its name implies, may perform entropy encoding of the data output from the quantizer 216. For example, the entropy encoder 218 may perform run-length encoding, Huffman encoding, Golomb encoding, context-adaptive binary arithmetic encoding, etc.

[0022] The pixel block decoder 220 can reverse the encoding operation of the pixel block encoder 210. For example, the pixel block decoder 220 can include an inverse quantizer 222, an inverse transform unit 224, and an adder 226. In some scenarios (e.g., when operating in a high dynamic range), after the inverse transform unit 224 and / or the adder 226, the input can be inversely reshaped or remapped, typically according to the function applied in the encoder and the content characteristics. The pixel block decoder 220 can take its input data from the output of the quantizer 216. Although allowed, the pixel block decoder 220 does not need to perform entropy decoding of entropy-encoded data because entropy encoding is a lossless event. The inverse quantizer 222 can reverse the operation of the quantizer 216 of the pixel block encoder 210. The inverse quantizer 222 can perform uniform or non-uniform inverse quantization as specified by the quantization parameter data qp. Similarly, inverse transform unit 224 may reverse the operation of transform unit 214. Inverse quantizer 222 and inverse transform unit 224 may use the same quantization parameters qp and transform modes as their counterparts in pixel block encoder 210. The quantization operation may truncate data at various points, and therefore the data recovered by inverse quantizer 222 may have coding errors when compared to the data presented to quantizer 216 in pixel block encoder 210.

[0023] The adder 226 may invert the operation performed by the subtractor 212. It may receive the same predicted pixel block from the predictor 260 that the subtractor 212 used to generate the residual signal. The adder 226 may add the predicted pixel block to the reconstructed residual value output by the inverse transform unit 224 and may output reconstructed pixel block data.

[0024] As described, frame buffer 230 may assemble a reconstructed frame from the output of pixel block decoder 220. In-loop filter 240 may perform various filtering operations on the recovered pixel block data. For example, in-loop filter 240 may include a deblocking filter, a sample adaptive offset ("SAO") filter, and / or other types of in-loop filters (not shown).

[0025] The reference picture buffer 250 can store filtered frame data for use in predicting other pixel blocks later. Different types of prediction data are available to the predictor 260 for different prediction modes. For example, for an input pixel block, intra-prediction takes prediction reference from the decoded data of the same frame in which the input pixel block is located. Thus, the reference frame store 250 can store the decoded pixel block data of each frame as it was coded. For the same input pixel block, inter-prediction can take prediction reference from previously coded and decoded frame(s) designated as reference frames. Thus, the reference frame store 250 can store these decoded reference frames.

[0026] The predictor 260 can provide the pixel block encoder 210 with a prediction block used to generate the residual. The predictor 260 can perform a prediction search operation according to intra-mode coding, as well as uni-predictive, bi-predictive, and / or multi-hypothesis inter-mode coding. In the case of intra-mode coding, the predictor 260 can search among pixel block data from the same frame as the pixel block being coded that provides the closest match for the input pixel block. In the case of inter-mode coding, the predictor 260 can search among pixel block data of other previously coded frames stored in the reference picture buffer 250 that provide a match for the input pixel block. From among predictions generated according to various modes, the predictor 260 can select the mode that achieves the lowest distortion when the video is decoded given a target bit rate. Exceptions may occur when the coding mode is selected to satisfy a particular channel behavior or to satisfy other policies to which the coding system 200 adheres, such as supporting a random access or data refresh policy.

[0027] Controller 270 may control the overall operation of encoding system 200. Controller 270 may select operating parameters for pixel block encoder 210 and predictor 260 based on an analysis of the input pixel blocks and also external constraints, such as a coding bit rate target and other operating parameters. As relevant to this description, when it selects a quantization parameter qp, the use of a uniform or non-uniform quantizer, and / or a transform mode M, it may provide those parameters to syntax unit 280, which may include data representing those parameters in the data stream of encoded video data output by system 200. Controller 270 may also select between different operating modes, for which the system may generate reference images, and may include metadata identifying the selected mode for each portion of the encoded data.

[0028] In operation, the controller 270 can modify the operating parameters of the quantizer 216 and the transform unit 215 at different granularities of the image data, either per pixel block or at a larger granularity (e.g., per frame, per slice, per largest coding unit (LCU) or coding tree unit (CTU), or another region). In one aspect, the quantization parameters can be modified for each pixel within an encoded frame.

[0029] Additionally, as described, the controller 270 can control the operation of the in-loop filter 250 and the prediction unit 260. Such control can include, for the prediction unit 260, mode selection (lambda, modes to be tested, search window, distortion strategy, etc.), and for the in-loop filter 250, filter parameter selection, parameter reordering, prediction weighting, etc. Figure 3

[0030] 3 is a functional block diagram of a decoding system 300 according to one embodiment of the present disclosure. The decoding system 300 may include a syntax unit 310, a pixel block decoder 320, a frame buffer 330, an in-loop filter 340, a reference picture buffer 350, a predictor 360, and a controller 370.

[0031] The syntax unit 310 can receive the encoded video data stream and parse the encoded data into its component parts. Data representing the coding parameters can be provided to the controller 370, and data representing the coded residual (data output by the pixel block encoder 210 of FIG. 2) can be provided to its respective pixel block decoder 320. The predictor 360 can generate a predictive block from reference data available in the reference picture buffer 350 according to the coding parameter data provided in the coded video data. It can provide the predictive block to the pixel block decoder 320. The pixel block decoder 320 can reverse the coding operation applied by the pixel block encoder 210 (of FIG. 2). The frame buffer 330 can generate a reconstructed frame from the decoded pixel blocks output by the pixel block decoder 320. The in-loop filter 340 can filter the reconstructed pixel frame data. The filtered frame can be output from the decoding system 300. The filtered frame designated to serve as a reference frame can also be stored in the reference picture buffer 350.

[0032] The pixel block decoder 320 may include an entropy decoder 322, an inverse quantizer 324, an inverse transform unit 326, and an adder 328. The entropy decoder 322 performs entropy decoding and may reverse the process performed by the entropy encoder 218 (of FIG. 2). The inverse quantizer 324 may reverse the operation of the quantizer 316 of the pixel block encoder 210 (of FIG. 2). Similarly, the inverse transform unit 326 may reverse the operation of the transform unit 214 (of FIG. 2). They may use a quantization parameter qp and a transform mode M identified from the encoded video data stream. Because quantization may discard data, the pixel block recovered by the inverse quantizer 324 may have coding errors when compared to the input pixel block s presented to the pixel block encoder 210 (of FIG. 2).

[0033] The adder 328 may invert the operation performed by the subtractor 210 (of FIG. 2). It may receive a predicted pixel block from the predictor 360 as determined by a prediction reference in the encoded video data stream. The adder 328 may add the predicted pixel block to the reconstructed residual values ​​output by the inverse transform unit 326 and may output reconstructed pixel block data.

[0034] As described, frame buffer 330 may assemble a reconstructed frame from the output of pixel block decoder 320. In-loop filter 340 may perform various filtering operations on the recovered pixel block data as identified by the encoded video data. For example, in-loop filter 340 may include a deblocking filter, a sample adaptive offset ("SAO") filter, and / or other types of in-loop filters. In this manner, the operation of frame buffer 330 and in-loop filter 340 mimics the operation of the corresponding frame buffer 230 and in-loop filter 240 of encoder 200 (of FIG. 2).

[0035] The reference picture buffer 350 can store filtered frame data for use in later prediction of other pixel blocks. The reference picture buffer 350 can store decoded frames as they are encoded for use in intra prediction. The reference picture buffer 350 can also store decoded reference frames.

[0036] As described, the predictor 360 may provide predicted blocks to the pixel block decoder 320 according to a coding mode identified in the encoded video data. The predictor 360 may provide predicted pixel block data as determined by a prediction reference indicator provided in the encoded video data stream.

[0037] The controller 370 may control the overall operation of the encoding system 300. The controller 370 may set operating parameters for the pixel block decoder 320 and the predictor 360 based on parameters received in the encoded video data stream. As relevant to this description, these operating parameters may include a quantization parameter Qp for the inverse quantizer 324 and a transform mode M for the inverse transform unit 310. As described, the received parameters may be set at various granularities of image data, for example, per pixel block, per frame, per slice, per LCU / CTU, or based on other types of regions defined for the input image. Figure 4

[0038] 4 is a functional block diagram of a quantization selection system 400 according to one embodiment of the present disclosure. In one embodiment, a video encoder and decoder can quantize video according to multiple (e.g., N) quantization step size table templates, with each table 410.0, 410.1, ..., 410.n providing its own relationship between an index value (q_idx) and a quantization parameter adjustment (q_Step). The q_Step values ​​stored in these tables 410.0, 410.1, ..., 410.n can be empirically derived or generated according to a predetermined function or formula. For example, the q_Step values ​​stored in the first table 410.0 can have a linear relationship, such as q_Step=m*q_idx+c (shown in graph 420.0), where m and c are real constants. The q_Step values ​​stored in the second table 410.1 can have an exponential relationship, such as q_Step=2^q_idx / k (shown in graph 420.1). The encoding and decoding system may have as many tables 410.0, 410.1, ..., 410.n as there are mathematical relationships 410.0, 420.1, ..., 420.n between q_idx index values ​​and q_Step quantization step size values, as may be desired for the encoding application. During encoding and decoding operations, the encoder and decoder may adaptively select from the quantization step size values ​​in tables 410.0, 420.1, ..., 420.n to suit the immediate encoding needs.

[0039] In a first embodiment, the quantization index value may be signaled between the encoder and decoder using a table index (tbl_idx) that identifies the table 410.0, 410.1, ..., or 410.n from which the q_Step value is to be read, and an index value (q_idx) that identifies the entry in the selected table 410.0, 410.1, ..., or 410.n from which the q_Step value is to be read. The q_Step value may be input to an adder 430 along with a base quantization parameter value, base_qp, to generate a final quantization parameter, qp. The final quantization parameter may be input to a quantizer 440 and may be applied to the transform coefficients (FIGS. 2 and 3) for either quantization or dequantization, as the case may be.

[0040] In another embodiment, the encoder and decoder can derive their own quantization adjustment tables on a dynamic basis from quantization step size table templates 410.0, 420.1, ..., 420.n. The encoder can derive the desired quantization adjustment table from quantization step size table templates 410.0, 420.1, ..., 420.n and signal the transition points to the decoder, which can generate the corresponding adjustment table.

[0041] In one embodiment, the encoder includes an ordered signal pair: TIdx i , T.P. i Each instance i of a signal pair is used to signal the transition point between tables. i-1 ) to TP i The range of quantization indices to be used up to the quantization index identified by TIdx i Specifically, we can identify a table represented by For i=0, operating_quant_step_size_table[k]=table_TIdxN0[k], where k ranges from 0 to TP0-1. If (i>0), then operating_quant_step_size_table[k]=operating_quant_step_size_table[global_transition_point]+table_TIdxNi[k-global_transition_point]; where global_transition_point=sum of all element counts at each TPi from 0 to i-1, and k ranges from global_transition_point to TPi.

[0042] Consider an example where three signal pairs (TIdxi, TPi), i=0, 1, 2, are provided. In such an example, the quantization adjustment tables can be derived from the q_Step values ​​stored in default quantization adjustment tables 410.0, 410.1, 410.2 (not shown) as follows: [Table 1] In practice, depending on the coding situation at hand, it may be advantageous to combine the endpoints of the segment value ranges extracted from each source table to form a continuous quantization adjustment curve.

[0043] 4 shows a decoder 450 that generates a table select signal tbl_idx to select among quantization adjustment tables 410.0, 410.1, ..., 410.n in response to a quantization index signal q_idx in response to a TIdxi, TPi signal pair. Once the relationship is established by the decoder 450, a selection of the quantization parameter qp can be made when the video data is quantized (encoded) or dequantized (decoded). The quantization index q_idx can be presented to the decoder 450, which generates a table index signal that selects a quantization adjustment table 410.0, 410.1, ..., 410.n from which a q_step value can be read. The quantization index q_idx can also be presented to a selected table (e.g., table 410.0) in order to read a q_step value from the table.

[0044] In one embodiment, system 400 may include adders 460.0, 460.1, ..., 460.n at the outputs of quantization adjustment tables 410.0, 410.1, ..., 410.n that may provide additive offsets to the q_step values ​​read from quantization adjustment tables 410.0, 410.1, ..., 410.n. The offsets may be applied to the q_step values ​​read from quantization adjustment tables 410.0, 410.1, ..., 410.n as may be desired, for example, to provide a continuous curve between segments of q_step values ​​read from quantization adjustment tables 410.0, 410.1, ..., 410.n at transition points. Such offsets may be stored in decoder 450 and output to selected adders 460.0, 460.1, or 460.n when decoder 450 reads the q_step value from the respective quantization adjustment table 410.0, 410.1, or 410.n.

[0045] In another embodiment, a scaling factor can be signaled along with the transition point and applied as a scaling factor to the q_step value read from the quantization adjustment tables 410.0, 410.1, ..., 410.n. The system 400 can include multipliers 470.0, 470.1, ..., 470.n that can apply a scaling weight w to the q_step value read from each quantization adjustment table 410.0, 410.1, ..., 410.n. In such an embodiment, the encoder can transmit signal values ​​as TIdxi, TPi, wi to represent the table (TIdxi) and weight (wi) to be applied to the table identified by the respective transition point TPi. Such weights can be stored by the decoder 450 and applied to the q_step value as determined by the quantization index value q_idx.

[0046] Of course, multipliers 470.0, 470.1, ..., 470.n and adders 460.0, 460.1, ..., 460.n may be used in combination, allowing system designers to define any linear combination of the q_step relationships defined in quantization adjustment tables 410.0, 410.1, ..., 410.n that may be desired to suit the needs of their particular application. Thus, the technique represented by system 400 provides wide flexibility for dynamically defining quantization adjustments during video encoding and decoding operations.

[0047] In another embodiment, the encoder can signal dynamically generated quantization adjustments by defining a selection of quantization adjustment tables 410.0, 410.1, ..., 410.n over the range of q_idx. In this embodiment, the selection may be identified by a transition point, and a start index (regionStartNk) that essentially defines the region of interest with a particular quantization step size template TIdxNk can be signaled. A default value for this start index (regionStartNk) can be assumed to be 0, in which case it will behave as described above. If K=0, operating_quant_step_size_table[m]=weightN0*table_TIdxN0[regionStartN0+m], where m ranges from 0 to TP0. If (K>0), operating_quant_step_size_table[m]=operating_quant_step_size_table[global_transition_point]+weightNk*table_TIdxNk[regionStartNk+m-global_transition_point], where: global_transition_point = sum over all TPk from 0 to k-1, where m ranges from global_transition_point to TPk-1.

[0048] In a further embodiment, the system 400 can independently derive the quantization step regions based on the signaled quantization step size templates as shown below: For all K signaled pairs of (TIdxk, TPk, weightNk, regionStartNk), If K=0, operating_quant_step_size_table[m]=weightN0*table_TIdxN0[regionStartN0+m], where m ranges from 0 to TP0. If (K>0), then operating_quant_step_size_table[m]=weightNk*table_TIdxNk[regionStartNk+m-global_transition_point], where: global_transition_point = sum over all TPk from 0 to k-1, where m ranges from global_transition_point to TPk-1.

[0049] As shown above, the above-described embodiments can adapt the desired q_idx vs. q_Step relationship based on the contents of the default quantization adjustment tables 410.0, 410.1, ..., 410.n, while maintaining a smooth monotonic relationship. Additionally, in the case of multiple tables, multiple transition points can be signaled. In certain embodiments, all transition points may be the same distance apart, and only one value (their distance) may be signaled.

[0050] The aforementioned syntax elements such as table index (tbl_idx), transition point (TPi), weight (wi), and start index (regionStartNk) can be signaled separately for all types such as luma DC, luma AC, chroma U DC, chroma U AC, chroma V DC, and chroma V AC. Figure 5

[0051] 5 illustrates another system 500 according to one embodiment of the present disclosure. System 500 may include multiple default quantization adjustment tables 510.0, 510.1, ..., 510.n, multipliers 520.0, 520.1, ..., 520.n, adder default quantization adjustment tables 530.1, ..., 530.n, memory 540 for quantization adjustment tables, and a bitstream parser 550. Bitstream parser 550 may receive a data stream containing transition point identification data and parse it into constituent data elements (e.g., table indexes, transition points, weights, and / or region start data, depending on the technique being used). The bitstream parser may output these data elements from system 500.

[0052] To build (or modify) quantization adjustment table 540, bitstream parser 550 can output a signal to a selected one of default quantization adjustment tables 510.0, 510.1, ..., 510.n, as identified by transition point data. As described, the table index can identify one of default quantization adjustment tables 510.0, 510.1, ..., 510.n from which stored q_Step data is read. The transition point and / or region start data can identify a location within a selected default quantization adjustment table (e.g., table 510.1) from which stored q_Step data is read. If used, weight information can identify a scaling factor to be applied by multiplier 520.1 when q_Step data is read from default quantization adjustment table 510.1 and stored in quantization adjustment table 540. Adder 530.1 can add an offset to the q_step value as may be desired to maintain continuity between the q_step value stored from the selected default quantization adjustment table 510.1 and the q_step value stored in quantization adjustment table 540 from the previous transition point identification. In many (but not all) coding applications, it is desirable to ensure that the q_step value increases monotonically as the value of q_idx increases. Also, although not shown in FIG. 6, adders 530.1, ..., 530.n can apply an offset value that is signaled in the bitstream and identified by bitstream parser 550. Figure 6

[0053] 6 illustrates a method according to one embodiment of the present disclosure. Method 600 may begin upon receipt of data identifying a transition point (Box 610), which may include data identifying a default q_step curve and a range of q_idx values ​​to which the transition point applies. Optionally, the transition point identification may include an identification offset and / or a scaling weight (Boxes 620, 630). From this transition point identification data, method 600 may derive a quantization curve (Box 640).

[0054] Method 600 can then be applied during runtime operation of a video encoder or decoder. When a q_idx value is received for a given coding element (box 650), method 600 can derive a q_step adjustment value according to the quantization adjustment curve derived in box 640 (box 660). Method 600 can adjust the base quantization parameter by applying the q_step value to the coding element's base_qp value (box 670), and the resulting value can be applied to a quantization operation (for encoding) or an inverse quantization operation (for decoding) (box 680). The operations in boxes 650-680 can be repeated as long as encoding / decoding operations are performed or until new transition point data is received (box 690). When new transition point data is received, the method can modify the quantization adjustment curve according to the operations in boxes 610-640.

[0055] In operation, modifying the quantization adjustment curve does not require wholesale replacement of previously identified transition points. In one embodiment, modified transition point data can be received along with the identification of the quantization indexes to which they apply. A new quantization adjustment curve can be derived by integrating the new transition point identifications with older transition point identifications that apply to other q_idx ranges.

[0056] In one aspect, coding parameters other than quantization parameters can also be derived from the quantization parameter, such as q_step (optional box 675). Coding parameters, such as in-loop filtering parameters, inter-prediction parameters, and intra-prediction parameters, can be determined in the encoder or decoder based on q_step, as described in more detail below in "Improved Quantizer-Based Coding Extensions." Any non-quantized coding parameters determined in box 675 can be used in a corresponding encoder or decoder process (not shown in FIG. 6). For example, the filter strength of an in-loop deblocking filter can be determined based on q_step in box 675, and the determined deblocking filter strength can then be applied to a deblocking filter in encoder filter 240 (FIG. 2) or decoder filter 340 (FIG. 3). Exemplary Syntax for Supporting Generation of Quantization Adjustment Tables

[0057] Below, exemplary syntax is provided to illustrate the integration of the aforementioned techniques into the AV1 encoding protocol. The following examples, as explained, are directed to AV1, but the principles of this disclosure can be applied to various other encoding systems, such as VVC, HEVC, etc.

[0058] Frame quantizer derivation: The function get_dc_quant(plane) returns the quantizer value for the dc coefficient of a particular plane and can be derived as follows: ●If plane is equal to 0, return dc_q(get_qindex(0,segment_id)+DeltaQYDc). ● Else if plane is equal to 1, return dc_q(get_qindex(0,segment_id)+DeltaQUDc). ● Otherwise (plane is equal to 2), return dc_q(get_qindex(0,segment_id)+DeltaQVDc).

[0059] The function get_ac_quant(plane) can return the quantizer value for the ac coefficients of a particular plane, and can be derived as follows: ●If plane is equal to 0, return ac_q(get_qindex(0,segment_id)). ● Else if plane is equal to 1, return ac_q(get_qindex(0,segment_id)+DeltaQUAc). ● Otherwise (plane is equal to 2), return ac_q(get_qindex(0,segment_id)+DeltaQVAc). The function get_qindex(ignoreDeltaQ, segmentId) can return the quantizer index of the current block, for example "q_Idx", which will be used in further description.

[0060] The syntax may include elements as shown below: For simplicity, only the relevant syntax elements are shown and the rest are omitted.

[0061] General frame header OBU syntax elements can include the following elements: [Table 2]

[0062] The uncompressed header syntax can include the following elements: [Table 3] where: separate_Ydc_q_params equal to 1 indicates that the luma DC quantization table parameters are coded separately. separate_Ydc_q_params equal to 0 indicates that the separate luma DC quantization table parameters are not signaled and are not derived from the luma AC quantization table parameters. If not present, its value is assumed to be 0. different_chroma_quant_param equal to 1 indicates that the chroma (U and V) quantization table parameters are coded separately. different_chroma_quant_param equal to 0 indicates that separate chroma (U and V) quantization table parameters are not signaled and are not derived from the luma quantization table parameters. If not present, its value is assumed to be 0. separate_uv_quant_param equal to 1 indicates that the quantization table parameters for the chroma U and chroma V components are coded separately. separate_uv_quant_param equal to 0 indicates that the quantization table parameters for the chroma U and chroma V components are not signaled and are not shared. If not present, its value is assumed to be 0. separate_Udc_q_params equal to 1 indicates that the quantization table parameters for the chroma U DC component are coded separately. separate_Udc_q_params equal to 0 indicates that the quantization table parameters for the chroma U DC component are not coded separately and are not shared with the chroma U AC component. If not present, its value is assumed to be 0. separate_Vdc_q_params equal to 1 indicates that the quantization table parameters for the chroma VDC components are coded separately. separate_Vdc_q_params equal to 0 indicates that the quantization table parameters for the chroma VDC components are not coded separately and are not shared with the chroma VAC components. If not present, its value is assumed to be 0. separate_UVdc_q_params equal to 1 indicates that the quantization table parameters for the chroma U and chroma V DC components are coded separately. separate_UVdc_q_params equal to 0 indicates that the quantization table parameters for the chroma U and chroma V DC components are not coded separately and are not shared with the chroma U AC component. If not present, its value is assumed to be 0.

[0063] The quantTable_params header syntax element can contain the following information: [Table 4] where: type represents the video content according to the following mapping: a type value of 0 is luma AC, 1 is luma DC, 2 is chroma U AC, 3 is chroma V AC, 4 is chroma V AC, and 5 is chroma V DC. num_pivot points[type] specifies the number of pivot points used in total to derive a quantization step size table of a particular type. TableIdx[type][idx] specifies the quantization step size template index used to derive the particular type of quantization step size table and pivot point index. transition_point[type][idx] specifies the transition point index for deriving the quantization step size table and pivot point index of a particular type. weights_applied equal to 1 indicates when weights are applied to derive the quantization step size table and pivot point index for a particular type. weights_applied equal to 0 indicates that weights are inferred to be equal to 1. If not present, the value of weights_applied is equal to 0. weightNrIdx specifies the weight numerator value used to derive the particular type of quantization step size table and pivot point index. weightDrIdx specifies the weight denominator value used to derive the particular type of quantization step size table and pivot point index. non_zero_idx_enabled equal to 1 indicates that flexible starting regions in the quantization step size template are enabled. non_zero_idx_enabled equal to 0 indicates that flexible starting regions in the quantization step size template are not enabled and regionStartIdx is inferred to be 0. If not present, non_zero_idx_enabled is inferred to be 0. regionStartIdx specifies the starting index within a particular quantization step size table template that is used to derive a particular type of quantization step size table and pivot point index.

[0064] Using the above syntax, the quantization step size table can be derived as follows, repeating the process for all types of quantization step size tables desired:

[0065] Local variables can be initialized to 0 (e.g., K=0, global_transition_point=0).

number

[0066] The final quantization step size value for a given q_idx can be derived as derived_quant_step_size_table[type][q_idx].

[0067] The above syntax applies to systems where quantization step size templates are made available, for example, by signaling them in the sequence header using the techniques described above, where the total number of q_idx values ​​is 256, and where three color components (Y, U, V) are present in the header. Of course, the principles of this disclosure apply to systems that use a different number of q_idx values, and to other components (alpha, depth, etc.), as well as subsets of components (monochrome, etc.) as may be desired.

[0068] The following discussion presents an illustrative application of the quantization curve derivation described above. Consider an example in which it is desirable to have two quantization adjustment tables 410.0, 410.1, one that is perfectly linear and the other that varies exponentially.

[0069] The linear quantization step size table 410.0 can be derived, for example, as q_step=clip(DESIRED_MIN, DESIRED_MAX, m*q_idx+c), where m and c are some real constants and DESIRED_MIN, DESIRED_MAX are the desired minimum and maximum values ​​of the quantization step size, respectively.

[0070] The exponentially varying quantization step size table 410.1 can be derived, for example, as q_step=clip(DESIRED_MIN,DESIRED_MAX,ROUND(24*(2^(q_Idx / 16)+k))), where k is some real constant and DESIRED_MIN, DESIRED_MAX are the desired minimum and maximum values ​​of the quantization step size.

[0071] In such applications, the system 400 may signal a "transition point" such as "linear_table_max_idx" that specifies the transition from a linear table to an exponential table.

number

[0072] The linear_table_max_idx or "transition point" can be signaled differently for the luma, chroma U, and chroma V components.

[0073] The linear_table_max_idx or "transition point" can be signaled differently for different bit depths.

[0074] The signaling of "transition points" can be signaled at different levels of granularity: sequence, frame, slice, tile, or block header. Each level of signaling can have different tradeoffs.

[0075] In an example application, data for a plurality of N quantization adjustment curves can be signaled at a sequence or frame granularity (to minimize coding costs), and an index corresponding to a selected one of the quantization adjustment curves can be signaled at a lower granularity, such as at the block level, slice level, segment level, tile level, or specific region level within a frame, to achieve desired adaptability. It can be noted that signaling granularity can provide different tradeoffs in complexity, compression efficiency, and adaptability. Generally, signaling at the lowest granularity is computationally intensive for hardware implementations and may incur greater coding costs. However, it can provide the greatest flexibility or adaptability for making coding decisions, especially for low-bitrate applications. Similarly, signaling related to quantization step size indexes, transition points, weights at transition points, and target regions within the quantization step size indexes can be signaled at various granularities, such as at the block level, slice level, segment level, tile level, or specific region level, frame level, or sequence level. For such data, signaling at a lower granularity offers significant flexibility and content-based adaptability.

[0076] The above discussion presented quantization adjustment in the context of unitary q_step and base_qp values. The foregoing techniques can be applied to multidimensional quantization matrices, where a quantization adjustment curve is defined as a quantization adjustment matrix having a quantization adjustment value that varies according to the transform coefficient being processed (e.g., for a DC coefficient and for multiple AC coefficients generated by transform unit 214 (FIG. 2)) and also by q_idx. Such quantization adjustment matrix data can be explicitly signaled in a sequence or frame header. An index corresponding to a quantization matrix from these "N" quantization adjustment matrices can be provided to the coding element being processed.

[0077] In another embodiment, a pivoting method can be applied to signal the quantization step size table and scaling matrix. The system 400 can specify N indices (using delta to ensure they are monotonically increasing) and then use an interpolation method to interpolate the missing indices. The interpolation can be simply bilinear or can use a more complex model (e.g., bicubic). Such techniques can be combined with piecewise methods, for example, where one segment uses interpolation and the next segment uses a function. Such techniques provide control over the accuracy of quantization in a fine-grained manner and using defined step sizes. DC / AC quantization can use a similar approach to quantization matrices. Signaling quantization matrices using an interpolation (e.g., pivoting-based) method can reduce the high signaling cost associated with quantization matrix signaling. In this approach, system 400 can signal values ​​for specific points, which can be prefixed to predetermined locations, or their locations can be explicitly signaled (in the latter case, it may be preferable to employ uniformly spaced points for all transform types). Different points can be signaled for different transform types, or points at corresponding "scaled" locations are indicated only as pivot points. The quantization scales of intermediate missing points are then interpolated using a specified method, for example, using bilinear interpolation applied horizontally and / or vertically.

[0078] In one embodiment, system 400 signals multiple quantization matrices at each stage (e.g., SPS), thus making it more flexible. In the PPS, the design has the option to indicate which quantization matrix (or multiple quantization matrices) in the SPS it prefers to use by indicating their index. This avoids costly signaling of different quantization matrices multiple times, since only the index needs to be signaled. Furthermore, at the slice / tile level, an index into the PPS quantization matrix list can be further used to allow more control and flexibility of the quantization scaling matrix used within a region. This can also be extended at the block level. A mechanism for predicting between quantization matrices can also be used, i.e., instead of signaling the entire table, deltas from a reference table can be used, or a quantization matrix in one PPS can be predicted from a quantization matrix indicated in the SPS or from another previously transmitted PPS. It may also be possible to indicate a different structure, for example, a new NAL unit type, such as an adaptive NAL unit type that is more significant than the PPS and includes all possible quantization matrices that the PPS can predict. In that case, a decoder or transmission system can more easily determine the relationship between different unit types within a stream for purposes of channel switching, random access, etc. The PPS or APS can also populate a reference buffer of quantization matrices, which can also be utilized for prediction of such elements. Essentially, this ability to predict quantization matrices within a PPS from other previously signaled elements or structures can be considered a form of "inter" prediction of quantization matrices. Spatial / intra prediction of quantization matrices is also possible; that is, if a PPS is allowed to contain multiple quantization matrices, a quantization matrix can be coded in a predictive manner relative to previous quantization matrices signaled in the PPS.

[0079] In further embodiments, the system 400 can control the number of quantization indexes signaled both at the frame / tile level (where a finer number of indices may be possible due to lower cost) and at the block level (which essentially indicates the "quantized quantization step size increment"). For example, at the picture level we can still have 256 or perhaps 128 quantization steps, but at the block level we can indicate the delta offset in increments of N (e.g., N=2 or 4) instead of 1. This can also apply to AC and chroma. More specifically, qpIdx at the block level can be derived as block_qpIdx=base_q_idx+qp_delta*quant_qp_delta.

[0080] In existing implementations of the VVC, HEVC, and AV1 standards, quantization matrices are typically derived based on the transform block size, prediction type, and color components of the block, and are generally fixed for all QPs, but may be different for different transform sizes, prediction modes, and color components. In one embodiment, system 400 can provide the flexibility to control and adjust quantization matrices based on QP as well. That is, instead of using a single quantization matrix for a particular prediction and block type, system 400 can also switch quantization matrices based on QP or QP range. Different matrices may make more sense for low QP vs. medium QP vs. high QP. As an extension, the matrix itself can have terms that are adjusted based on the QP value being used.

[0081] The above description presented the adjustment of quantization values ​​along a single range of quantization indexes, represented as q_idx. In practice, it may be desirable to provide different instantiations of system 400 (FIG. 4) and method 500 (FIG. 5) for different types of data. For example, a different instantiation may be provided for luma data than for chroma data. A different instantiation may be provided for standard dynamic range (SDR) video than for high dynamic range (HDR) video. Different instantiations may be provided based on the bit depth used to represent the video data (e.g., 8-bit color, 10-bit color, or 12-bit color). Different instantiations may be provided for different types of transform coefficients (e.g., for DC coefficients as opposed to AC coefficients). The number and type of instantiations may be selected by a system designer when applying the above-described techniques to their individual needs.

[0082] In a further aspect, the quantization value itself can be quantized before transmission between the encoder and decoder. In many applications, the quantization parameter for a pixel block is expressed as a delta_qp value, which represents a change relative to a base quantization parameter given elsewhere in the coding syntax (e.g., final_qp = base_qp + delta_qp). In further embodiments, it may be beneficial to provide delta_qp as a value that is quantized relative to the quantizer quantization parameter q_qp such that final_qp = base_qp + q_qp * delta_qp. The quantizer quantization parameter q_qp can be signaled in a higher-level syntax element. In many coding applications, signaling the delta_qp value can consume 6-8% of the coding bandwidth, and it is estimated that quantizing the delta_qp value can therefore achieve a significant reduction in coding bit rate.

[0083] In a further aspect, multiple quantization step size tables can be generated (using any combination of the methods described herein) to capture pixel reshaping characteristics. It is possible that adjacent regions can follow different pixel reshaping characteristics or reshaping "modes." To enable flexible use of quantization step size tables, it is further proposed to signal the quantization step size table to be used for that particular region that offers the best trade-off between quality, compression efficiency, and complexity (taking into account the reshaping characteristics / modes). Regions may be of various granularities, such as block, slice, tile, segment and / or frame, or sequence level. Dynamic generation of quantization adjustment curves

[0084] In some aspects, the generation of a set of quantization adjustment curves (such as tables 410.0-410.n in FIG. 4) can be controlled by a quantization step mode selection (qStep_mode). The selected quantization step mode can be signaled in the compressed bitstream, along with any parameters necessary to construct the set of quantization curves in the selected mode.

[0085] For example, a set of quantization curves can be generated according to one of the following candidate quantization step modes: a.qStep_mode=0 may indicate that a set of default quantization adjustment curves is predetermined and fixed. In this case, no further information needs to be signaled in the compressed bitstream to construct a predetermined quantization adjustment curve, and therefore the quantization step mode parameter may not be required. b.qStep_mode=1 may indicate that the quantization adjustment curve is generated with a specified number of intervals whose lengths are related to each other by a predetermined relationship. For example, the intervals are generated with the number of available quantization indexes (q idx ) over a power-of-two relationship (e.g., 8, 8*2 0 , 8*2 1 , 8*23 ) Quantization adjustment values ​​can be generated within each interval according to various means. In one example, a default table can be provided with predetermined quantization adjustment values, and the table values ​​can be extended across each interval by scaling the table values ​​according to the length of each interval and applying an offset according to the highest quantization adjustment value of the preceding interval. The default table can include quantization adjustment values ​​that increase with index value according to a desired profile, such as according to an exponentially increasing profile. Alternatively, the default table can include quantization adjustment values ​​that increase linearly with index value. In this example, the quantization step mode parameters can include the number of curves (NumTransistionInterval_minus1) and the number of quantization step entries in the table that specify each curve (numOstepsinInterval). c.qStep_mode=2 may indicate that a first quantization adjustment curve is signaled and that other quantization adjustment curves are generated by replicating the signaled first curve. In this example, the quantization step mode parameter may include the number of quantization step size entries in a table defining the signaled first curve and the quantization step size value for each entry in the table defining the signaled first curve. d.qStep_mode=3 may indicate that each quantization adjustment curve is explicitly signaled. For example, each curve may be defined by a table, and each table may be signaled. In this example, the quantization step mode parameter may include the number of tables, and then for each table, the number of entries in that table, along with a list of the quantization step sizes in each table.

[0086] A design that includes selectable quantization step modes can provide several advantages. a. Transition Interval Types: An interval window can be thought of as a segment of a curve. It can be beneficial to have interval windows that are either fixed or variable. A variable quantization mode selection scheme provides flexibility to the coding system. b. Symmetry across transition intervals: In one implementation, the Q-step may double after each interval (e.g., the Q-step for the Nth interval is twice the corresponding Q-step from the N-1th interval). A variable quantization mode selection scheme can provide and leverage implicit signaling to exchange Q-step information between the encoder and decoder, thereby increasing coding efficiency in such systems. c. Exponential (especially power-of-two) derivation of interval lengths: In one implementation, for example, the interval length may double after each interval (e.g., the length of the Nth interval is twice that of the N-1th interval). A variable quantization mode selection scheme provides and leverages implicit signaling to exchange interval information between the encoder and decoder, further increasing coding efficiency in such systems.

[0087] In some cases, it may be desirable to have an exponential (especially power-of-two) mapping of qIdx to Qstep. Powers of two can be more efficiently achieved by shift operations, without the need for division operations. When implemented in hardware processing circuits, division operations are often expensive mathematical operations to perform. [Table 5]

[0088] Example syntax for sequence header for the above four quantization step modes: [Table 6]

[0089] In one aspect, qStep_mode can specify the selected mode of qStep derivation. qStep_mode=0 specifies that the tables in Section 1.1 are used for the current sequence. qStep_mode=1 and qStep_mode=2 specify how to derive the qStep value as in Sections 1.2 and 1.3. qStep_mode=3 is unspecified. If not present, the value of qStep_mode is inferred to be 0.

[0090] In other aspects, quantization curve signaling need not be done in the sequence header, but may instead be done at different frequencies or at different levels of granularity within the compressed bitstream. This may be done by specifying quantization adjustment curves in different layers of syntax within the bitstream. For example, quantization curve signaling may be done at the sequence layer, the tile layer, etc. Similarly, in MPEG standards (e.g., VVC), they may be in the sequence, picture, subpicture, slice, or tile headers.

[0091] The following describes, with pseudo-code examples, an exemplary technique for generating quantization curves using the sequence header syntax described above. qStep_mode=0

[0092] In the first example, qStep_mode=0, which applies a default granularity for the qStep size. In this example, the interval window type is set to fixed, the transition interval doubles every interval, and the step size is derived as a power-of-two curve. This implementation allows the quantization table to be defined in the interface specification, which provides a set of default values ​​for encoding applications that do not need to explicitly specify the values ​​of the quantization table. The value of NumTransitionInterval_minus1 can be inferred between devices to be equal to 9, and the numQstepsInInterval for each transition interval can be inferred to be equal to 8, 8, 16, 32, 32, 32, 32, 32, 32, 32. In this example, the values ​​of NumTransitionInterval_minus1 and numQstepsInInterval are derived when the qStep_mode value is set to 0.

number

[0093] In another example, qStep_mode=1, which allows the encoder to define flexible quantization tables with flexible quantization steps. In this example, the interval window type is variable, symmetry across the transition interval = doubling every interval, and the step size is derived as a power-of-two curve. This implementation provides flexibility in defining the Q-step precision over the operating bitrate range. For example, when operating at a higher QP, one can define a finer step size around the operating QP, which helps reduce quantization error and abrupt transitions in encoding quality during an encoding session.

[0094] In this example, the encoder may provide signaling as follows: NumTransitionInterval_minus1 - This value plus 1 specifies the number of intervals within the Qstep range of 256, in which the number of qsteps within the interval is signaled. NumQstepsInInterval[i] - the number of qStep points in the 'i'th transition interval. When NumQstepsInInterval is not signaled for the 'i'th transition interval, NumQstepsInInterval is inferred to be equal to 32. The transition value NumQstepsInInterval[NumTransitionInterval_minus1] is estimated to be 256 - (the sum of NumQstepsInInterval[0] through NumQstepsInInterval[NumTransitionInterval_minus1]). In many cases, in practice, the sum of NumQstepsInInterval[0-NumTransitionInterval_minus1] is less than 256.

[0095] In one aspect where qStep_mode=1, NumQstepsInInterval can be set as a power of 2, and the Q-step table can be derived as follows:

number

[0096] In another embodiment of qStep_mode=1, which may be suitable for methods that can operate with floating-point precision, the Q-step table may be derived as follows:

number

[0097] In a further example, qStep_mode=2, which allows the encoder to signal the offset of one interval and replicate that interval to generate additional intervals.

number

[0098] In yet another example, qStep_mode=3, which allows the coding system to explicitly signal a QstepTableTemplate and derive a QstepTable from it. In this example, a set of tables to serve as templates can be signaled. The final table to be used in the sequence can be derived as a combination of QstepTableTemplates. In the following example illustration, there are the first 64 entries from Table 1 and the next 196 entries from Table 2. If K=0,

number

number

[0099] 8 illustrates a method according to one embodiment of the present disclosure having a quantization step mode. Method 800 may begin upon receiving data identifying a selected quantization step mode and parameters associated with the selected quantization step mode. The parameters associated with the selected quantization step mode may enable derivation of a set of quantization adjustment curves and may include data on the number of transition points, as described above (Box 810). Optionally, the transition point identification may include identification offsets and / or scaling weights (Boxes 820, 830). From the selected quantization step mode and any associated parameters, method 800 may derive a quantization curve (Box 840).

[0100] Method 800 can then be applied during runtime operation of a video encoder or decoder. When a q_idx value is received for a given coding element (box 850), method 800 can derive a q_step adjustment value according to the quantization adjustment curve derived in box 840 (box 860). Method 800 can apply the q_step value to the base_qp value of the coding element (box 870), and the resulting value can be applied to a quantization operation (for encoding) or an inverse quantization operation (for decoding) (box 880). The operations in boxes 850-880 can be repeated as long as an encoding / decoding operation is performed, or until a new quantization step mode or associated parameters are received (box 890). When a new quantization step mode or new parameters associated with a quantization step mode are received, the method can modify the quantization adjustment curve according to the operations in boxes 810-840.

[0101] In operation, modifying the quantization adjustment curve does not require wholesale replacement of previously identified transition points, as described below with respect to predicting the quantization adjustment curve. In one embodiment, the modified transition point data can be received along with the identification of the quantization indexes to which they apply. A new quantization adjustment curve can be derived by integrating the new transition point identifications with older transition point identifications that apply to other q_idx ranges.

[0102] In one aspect, coding parameters other than quantization parameters can also be derived from the quantization parameter, such as q_step (optional box 875). Coding parameters, such as in-loop filtering parameters, inter-prediction parameters, and intra-prediction parameters, can be determined in the encoder or decoder based on q_step, as described in more detail below in "Improved Quantizer-Based Coding Extensions." Any non-quantized coding parameters determined in box 875 can be used in a corresponding encoder or decoder process (not shown in FIG. 8). For example, the filter strength of an in-loop deblocking filter can be determined based on q_step in box 875, and the determined deblocking filter strength can then be applied to a deblocking filter in encoder filter 240 (FIG. 2) or decoder filter 340 (FIG. 3). Alternative sets and predictions of quantization adjustment curves

[0103] Different portions of the encoded video may use alternative sets of quantization adjustment curves. For example, one set of quantization adjustment curves as described above in the sequence header for the four quantization step mode may be combined to generate a new quantization adjustment curve for use during quantization or dequantization of all image data in the video sequence. In other examples, different sets of quantization curves may be defined in the compressed bitstream for DC and AC coefficients, for luma and chroma data, or for I-frames, P-frames, or B-frames.

[0104] In one aspect, the coding system can explicitly signal the initial_Q_Step index, which is the first element of the Q_Step.

[0105] When alternative sets of quantization adjustment curves are included more than once in a compressed bitstream, the bit cost of specifying the adjustment curves can be reduced by predicting one set of adjustment curves from previously signaled quantization adjustment curves.

[0106] In a first aspect, a set of quantization adjustment curves can be specified by a prediction within the parameters that specify the set of quantization adjustment curves. In this type of prediction, the parameters can be delta-coded with respect to previous parameters. For example, while signaling a q-Step table of N elements, the Kth element of the qStep table can be coded as a delta from the (K-1)th element (if available), i.e., deltaqStep[k]. During the derivation of the qStep parameter, qStep[K] is derived as qStep[k-1]+deltaqStep[K]. A flag can be signaled to enable or disable such a delta coding scheme (based on the previous value).

[0107] In a second aspect, a set of quantization adjustment curves can be specified by intra-prediction across temporal parameters. In this type of prediction, parameters are copied from an already singled parameter that precedes in coding order, either in the corresponding header of a temporal picture (picture, tile header, etc.) or in a header within the same picture (e.g., a tile header). A flag can be signaled to enable or disable such a copying scheme (based on temporal values ​​or already coded preceding values). Specific rules can be defined to specify from where information can be copied when multiple tiles are enabled in a sequence. Some exemplary definitions are given below: A flag to specify copying from a global parameter set (e.g., sequence header): If this flag is enabled, parameters are derived by copying from a global parameter set. b. If the flag is disabled in (a), another flag can be signaled to copy from a specific tile header (e.g., from the tile header of a temporal picture with the same tile ID as the current one). In summary, for the current header (eg, tile), it can be signaled to specify where information can be copied from.

[0108] In a third aspect, a set of quantization adjustment curves can be specified by prediction between color component parameters. In this type of prediction, parameters are copied from their corresponding components. For example, parameters are signaled for one of the components (luma), and the other two components, Cb and Cr, use the same parameters as luma. In another example, parameters are signaled for one of the components (AC coefficients), and the DC component uses the same parameters as AC coefficients. A flag can be signaled to enable or disable such a copying scheme (based on the corresponding component in the above example). If disabled, separate parameters can be signaled for each of the above components. Coding extension based on improved quantizers

[0109] A flexible quantizer signaling scheme has been presented above (e.g., Figures 6 and 8), in which the relationship between the quantization index q_idx and the quantization step size q_step is explicitly signaled and adapted to the image / video coding specifications. This quantizer control can be used, for example, for quantizing residual signals. In particular, for one set of tiles, images, or sequences, a user can choose to signal a particular range of q_idx values, e.g., q_idx(m) to q_idx(n), corresponding to a particular q_step value from q_step0(m) to q_step0(n). However, for a different set of tiles, images, or sequences, a different range of q_step values, i.e., q_step1(m) to q_step1(n), can be selected. Such flexibility can be seen as advantageous in different encoders, as it can allow finer and more accurate rate control and rate allocation models to be achieved.

[0110] Nevertheless, several normative and non-normative aspects of a coding system that can support such quantizer flexibility may be affected by such a design. In particular, the choice of quantizer during encoding typically can affect parameters that control coding mode decisions, such as Lagrangian parameters used for Lagrangian-based motion estimation, mode decision, and other rate-distortion-based operations, if supported by the encoder, such as rate-distortion optimized quantization, sample adaptive offset optimization applicable in the context of the HEVC and VVC video coding standards. Such Lagrangian optimization is typically of the form: J = strain + lambda * rate where distortion may be any metric of quality that can indicate how different the coded signal is from the original input and can be calculated, for example, as a sum of absolute values ​​or a squared difference, and rate indicates either the actual or approximate number of bits required to code the corresponding information currently being optimized. The variable lambda is a control parameter that influences the decision process's preference toward either lower distortion or lower rate. In such optimization, multiple alternative coding modes or parameters are evaluated using a Lagrangian cost J, and the coding mode that typically minimizes such cost is selected. The model can be extended to support additional cost parameters, such as decoding complexity / power cost, cost estimates due to packet loss, and quality due to post-processing.

[0111] Control of the parameter lambda is typically achieved by examining the value of q_idx and specifying the formulation of lambda given q_idx, although in some cases other parameters such as resolution, slice type, hierarchical level, and local complexity / variance are also specified. Such relationships have been determined in the past by examining the behavior of sequences coded under different quantizers and coding conditions, given different models, and their effect on the Lagrangian lambda value. However, traditional coding specifications employ a fixed set of q_idx, and therefore it is easy to determine the appropriate relationship between q_idx and the Lagrangian parameter. Nevertheless, doing so in flexible q_idx-to-q_step designs (such as Figures 6 and 8) is somewhat more complicated, given that the mapping from q_idx to q_step is not fixed here. The original calculation of the lambda parameter can still be used, but this may be suboptimal. For example, a common model used in standards such as AVC and HEVC is of the form: Lambda = Alpha * 2^((q_idx-12) / 3)

[0112] The core form of this calculation is based on the unique relationship between q_idx and Qstep in these standards, but alpha is typically adjusted based on the prediction structure and application goals considered during encoding. For intra coding, for example, a typical value used for alpha is equal to 0.68. This calculation is sometimes referred to as the "nominal" calculation of lambda given a "nominal" q_idx.

[0113] While such a fixed formula can also be used with the adaptive q_idx to q_step design described above, it may be suboptimal and not yield the best performance for encoding. Instead, the lambda calculation can be adjusted to be based on a new q_idx vs. q_step relationship. In particular, the lambda model used for AVC and HEVC can be reformulated into the following form: Lambda = Alpha * f(q_step) In AVC and HEVC, q_step is of the form:

number

number

number

[0114] Assuming a similar model is applicable across encoder implementations of other coding standards, the above formula can be used to determine the lambda value for the flexible q_idx mapping method: first determine the operating q_step for the current partition, and then use this q_step in the above formulation to derive lambda.

[0115] For different operations in the encoder, lambda can be calculated slightly differently. For example, for motion estimation, lambda is usually calculated as the square root of lambda for the mode decision process. An extension of the above calculation, i.e., a remapping of the lambda calculation from the corresponding nominal calculation using q_idx to a formulation using q_step, should be very straightforward for those experienced with current technology. It should also be very straightforward to convert other formulas used by encoders that specify the derivation of lambda from the nominal q_idx to utilize the q_step value instead.

[0116] The derivation of lambda may also depend on which distortion function is used. A similar remapping of lambda from the corresponding nominal calculation using q_idx to a formulation using q_step can be extended to first-order distortions (e.g., sum of absolute differences), second-order distortions (e.g., mean squared error), and higher-order trained distortions (e.g., SSIM, VMAF, etc.). For higher-order trained distortions, ScaleFactor (in addition to alpha) can be calculated as a function of the metric score for the region (8x8 block, segment, etc.) and q_idx(q_step).

[0117] While some encoders may be able to calculate the value of lambda for any q_step value, in some cases, for example in a particular hardware implementation, such calculations may be performed using a lookup table of limited size, e.g., size N. In such cases, the lookup table may be designed to provide an accurate mapping between q_step and lambda for N different, uniformly distributed q_step values ​​in the logarithmic domain, and then interpolation techniques may be used to derive a more accurate lambda value for the actual selected q_step.

[0118] For example, assuming N is 64 and one wishes to cover a Qstep range from 32 to 2^14, one can define lambda for the q_step value at location 32+256*n, where n runs from 0 to N-1. If q_idx maps to a q_step value different from the defined q_step value, lambda can be calculated using an interpolation process using the nearest neighbor defined q_step value, or an extrapolation process if necessary. Interpolation can be performed using a variety of interpolation methods, for example, bilinear, bicubic, or piecewise cubic Hermite polynomial interpolation, and can be preferably performed in the logarithmic domain or in the linear domain given the properties of the quantization step size.

[0119] This approach can help reduce the memory requirements of storing q_step in a lambda lookup table, while also keeping the computational complexity low.

[0120] Normative operations such as loop filtering, motion vectors, and weighted prediction, which affect both the encoder and decoder, can also be influenced by consideration of a flexible q_step-to-q_idx mapping. In particular, loop filtering mechanisms such as deblocking, adaptive loop filtering (ALF), and sample adaptive offset (SAO) filtering can adjust the strength of the filtering mechanism performed on samples within a block by considering the corresponding q_idx values ​​of the block and its neighbors. For example, in AVC and HEVC, tables are defined that specify q_idx-dependent thresholds that affect how and when pixels should be filtered. These tables can also be defined as an equation between a threshold T and q_idx, i.e., T = g(q_idx). Such thresholds are defined based on the expected distortion that q_idx may introduce, also taking into account its association with a particular q_step value. However, with the flexible q_idx design described herein, it may be more appropriate to redefine such tables or equations based on the actual q_step value, i.e., T = h(q_step), instead of the index q_idx. The transformation of such an equation, i.e., from q_idx to q_step, can be done very similarly to how the transformation of the Lagrangian parameter derivation formulation was performed above.

[0121] If it is desirable to compute such a thresholding mechanism using a table lookup, as in HEVC, one can either use a fixed table that only considers the "quantized" q_step values ​​indicated by the entries in the fixed table, or by using interpolation techniques that were also utilized in the Lagrangian parameter derivation.

[0122] In one aspect for in-loop deblocking filtering, the deblocking filter strength can be related to the degree of quantization. A higher degree of quantization in a pixel block (or other part of an image) can result in an increase in visual blocking artifacts, and therefore, the use of a stronger deblocking filter can improve compression quality. In-loop deblocking filters can be used in both the encoder and decoder, and the deblocking filter strength can be inferred from other coding parameters instead of being explicitly signaled in the compressed bitstream between the encoder and decoder. In AVC and HEVC, the deblocking filter strength is inferred (at least in part) based on the quantizer index (q_idx) used for the pixel block. With the more flexible quantizer designs described herein, q_idx may not be a reliable predictor of the degree of quantization used, the severity of blockiness artifacts, or the best in-loop filter strength. With these more flexible quantizer designs, the deblocking filter strength can be better predicted from either the q_step or qp (quantization parameter) values.

[0123] Other processes in both the encoder and decoder can also utilize quantization information to perform or adapt their encoding steps. For example, motion vector prediction considers partitions spatially and temporally adjacent to the current block to determine possible motion vector candidates for predicting and encoding the current motion vector. Such derivation can also utilize quantization because it directly relates to the quality of the candidate and, therefore, its motion vector. If a neighborhood is of higher quality than another neighborhood, it is likely that the motion vector of the higher-quality neighborhood will have a higher correlation than the motion vector of a lower-quality neighborhood. Considering the actual q_step used for the neighborhood instead of q_idx can allow for better consideration of the relative quality between different neighboring partitions, thus potentially allowing for a more accurate determination of the final motion vector predictor. Specifically, a higher-quality motion vector candidate can be placed at a lower index position relative to other lower-quality motion vector candidates in the motion vector prediction list. Because some motion vector candidates may be the same or similar, having a more accurate way of placing these candidates in the motion vector list can prove beneficial. Similar techniques can be used to predict other information, such as weighted prediction parameters.

[0124] In one aspect, the prediction search can be based on the degree of quantization of the candidate prediction reference. For various types of prediction from multiple possible prediction references, the accuracy of the prediction accuracy can be related to the degree of quantization. A highly quantized prediction reference may provide poor prediction compared to a less highly quantized prediction reference. Therefore, when selecting a single motion prediction reference in an encoder (e.g., for inter- or intra-prediction of pixel data), the encoder can include a bias toward selecting a prediction reference that was previously quantized to a lower degree, and the degree of quantization can be more accurately determined based on q_step or qp rather than q_idx. In the case of a weighted combination of multiple prediction references in an encoder or decoder, the weighting can be based on q_step or qp (instead of q_idx). For example, linear weighted prediction can be as follows:

number

number

[0125] In another aspect, a quantization step size adjustment curve for a particular pixel region can be constructed based on the pixel intensity distribution of that background region relative to the foreground region. The region here may be a block, a tile, a segment, or an entire frame. For example, a quantization step size value can be derived for an N×N block of pixels in the following steps: In the first step, for each N×N block, a background region is estimated and its average pixel intensity is calculated. The method for estimating the background region may be simple, such as allowing the surrounding M rows / columns of pixels around the N×N block of pixels to constitute the background. Alternatively, background region derivation may involve more advanced methods such as tracking, human pose estimation, background mixture models, fuzzy and deep learning-based methods, etc. In the second step, a contrast coefficient is calculated based on the absolute difference in pixel intensity between background and non-background pixels within the current region (e.g., an N×N block of pixels). The coefficient may also be calculated as a weighted absolute difference based on, for example, the pixel distance of the background pixel relative to the nearest foreground pixel. In the third step, a quantization step size scaling factor is calculated as a function of the background average pixel intensity derived in step 1 and as a function of the contrast coefficient derived in step 2. This scaling factor can be normalized and clipped to fit within the range of neighboring step sizes. This scaling factor can be interpreted as a perceptual mask of the block relative to neighboring blocks, and this factor can be applied as a ratio to one of the quantization curves described in this disclosure. Alternatively, the quantizer steps resulting after applying these scaling factors can form a new quantizer curve after normalization, which can be signaled in the sequence, frame, tile, or segment header.

[0126] In the preceding discussion, the operation of aspects of the present disclosure has been described in the context of video encoders and decoders, such as those shown in FIGS. 2-3. Typically, these components are provided as electronic devices. The video decoder and / or controller may be embodied in integrated circuits, such as application-specific integrated circuits, field-programmable gate arrays, and / or digital signal processors. Alternatively, they may be embodied in computer programs executing on camera devices, personal computers, notebook computers, tablet computers, smartphones, or computer servers. Such computer programs typically include instructions stored on non-transitory physical storage media, such as electronic, magnetic, and / or optical-based storage devices, that are read and executed by a processor. Decoders are typically packaged within consumer electronics devices, such as smartphones, tablet computers, gaming systems, DVD players, portable media players, and the like, and may also be packaged within consumer software applications, such as video games, media players, media editors, and the like. Of course, these components may also be provided as hybrid systems, distributing functionality between dedicated hardware components and programmed general-purpose processors, if desired.

[0127] The video encoder and decoder can exchange video over channels in a variety of ways. They can communicate with each other via a telecommunications and / or computer network, as shown in Figure 1. In yet other applications, the video encoder can output video data to a storage device, such as an electrical, magnetic, and / or optical storage medium, which can later be provided to the decoder. In such applications, the decoder can retrieve the encoded video data from the storage device and decode it.

Claims

1. 1. A quantization method comprising: deriving a new quantization adjustment curve from transition point data signaled in a data stream representing encoded video data, the transition point data identifying elements of a default adjustment curve to be used in deriving the new quantization adjustment curve; deriving, for an element of a video to be processed, a quantization adjustment value from the new quantization adjustment curve according to an index value associated with the element of the video to be processed; deriving a quantization parameter from the quantization adjustment value; applying said quantization parameters to said elements of video in a quantization operation; A method comprising:

2. 2. The method of claim 1, wherein the transition point data includes a table index that identifies a default adjustment curve and a range of indices into the table that are used to derive the new quantization adjustment curve.

3. The method of claim 1 , wherein the transition point data includes scaling values ​​that identify scaling to be applied to the elements of a default adjustment curve for derivation of the new quantization adjustment curve.

4. The method of claim 1 , wherein the quantization operation is a quantization of transform coefficients performed by a video encoder.

5. The method of claim 1 , wherein the quantization operation is an inverse quantization of transform coefficients performed by a video decoder.

6. The method of claim 1 , wherein the new quantization adjustment curve at each index position along the curve represents a quantization matrix of quantization adjustments applied to a separate matrix of transform coefficients.

7. each of the default adjustment curves is represented by a table of quantization step sizes; the transition point data defines transitions between the tables indicating which of the tables contribute to the new quantization adjustment curve over what range of indices into the tables; the quantization adjustment value is derived from the quantization step size entry of the table selected by the transition point data of the index value; The method of claim 1.

8. 2. The method of claim 1, wherein different sets of transition point data exist for different portions of the data stream, and a new quantization adjustment curve is derived for each of the different portions of the data stream.

9. The method of claim 8 , wherein the different portions are AC coefficients and DC coefficients.

10. The method of claim 8 , wherein the different parts are luma and chroma coefficients.

11. The method of claim 8 , wherein the different portions are I-frames, P-frames, and B-frames.

12. The method of claim 8 , wherein a later set of transition point data is predicted from a previous set of transition point data.

13. 2. The method of claim 1, wherein the transition point data includes a selected quantization step mode and selected parameters of the selected quantization step mode.

14. deriving a filter strength based on the quantization adjustment value; applying an in-loop filter to the element of video with the derived filter strength; The method of claim 1 further comprising:

15. 1. A quantization method comprising: deriving a new quantization adjustment curve from a quantization step mode and any associated mode parameters of the selected quantization step mode signaled in a data stream representing encoded video data; deriving, for an element of a video to be processed, a quantization adjustment value from the new quantization adjustment curve according to an index value associated with the element of the video to be processed; deriving a quantization parameter from the quantization adjustment value; applying said quantization parameters to said elements of video in a quantization operation; A method comprising:

16. The method of claim 15 , wherein the selected quantization step mode is a default mode, and no mode parameter is associated with the selected quantization step mode.

17. 16. The method of claim 15, wherein the selected quantization step mode comprises a linear quantization adjustment curve having a slope twice that of a previous quantization adjustment curve.

18. 16. The method of claim 15, wherein the selected quantization step mode includes associated parameters that specify a single quantization adjustment curve, and wherein subsequence quantization adjustment curves are determined by replicating the single quantization adjustment curve.

19. The method of claim 15 , wherein the selected quantization step mode includes associated parameters that define a table of quantization step sizes for each quantization adjustment curve.

20. deriving a filter strength based on the quantization adjustment value; applying an in-loop filter to the element of video with the derived filter strength; 16. The method of claim 15, further comprising:

21. 1. A coding system comprising: a quantizer that applies a quantization parameter to elements of the video; a controller for controlling the quantizer; a memory for storing instructions; wherein the instructions, when executed by the controller, deriving the new quantization adjustment curve from transition point data signaled in a data stream representing encoded video data, the transition point data identifying elements of a default adjustment curve to be used in deriving the new quantization adjustment curve; deriving a quantization adjustment value for the element of video to be processed from the new quantization adjustment curve according to an index value associated with the element of video to be processed; deriving the quantization parameter from the quantization adjustment value; Coding system.

22. A non-transitory storage medium storing instructions that, when executed on a processor, deriving a new quantization adjustment curve from a quantization step mode and any associated mode parameters of the selected quantization step mode signaled in a data stream representing encoded video data; deriving, for an element of a video to be processed, a quantization adjustment value from the new quantization adjustment curve according to an index value associated with the element of a video to be processed; deriving a quantization parameter from the quantization adjustment value; applying said quantization parameters to said elements of video in a quantization operation; Non-transitory storage medium.

Images