Video decoding method, decoder, and storage medium

Abstract

The present disclosure relates to a video decoding method and a decoder, a storage medium, and discloses a video compression and decompression technology that provides improved bandwidth control for a video compression and decompression system. Specifically, the video encoding and decoding technology quantizes the input video in multiple dimensions. According to these techniques, pixel residuals can be generated from a comparison of an array of input data with an array of prediction data. The pixel residuals can be quantized in a first dimension. After quantization, the quantized pixel residuals can be transformed into an array of transform coefficients. The transform coefficients can be quantized in a second dimension and entropy encoded. Decoding technology reverses these processes. In other embodiments, multiple quantizers can be provided upstream of the transform stage in parallel or in cascade, which provides greater flexibility for the video encoder to strive for quantized data in different dimensions, thereby balancing the competing interests of compression efficiency and the quality of the reconstructed video.

Description

[0001] This application is a divisional application of the invention patent application with application number 201780053103.6, application date July 27, 2017, and invention name “Video encoding method and encoder, video decoding method and decoder, storage medium”.

[0002] Priority claim

[0003] The present disclosure benefits from priority to U.S. Patent Application No. 15 / 250,608, filed on August 29, 2016, entitled “Multidimensional Quantization Techniques For Video Coding / Decoding Systems,” which is incorporated herein by reference in its entirety. Background Art

[0004] The present disclosure relates to video compression and decompression techniques, and more particularly, to providing improved bandwidth control and encoding performance for video compression and decompression systems.

[0005] Modern video compression systems, such as MPEG-4AVC / H.264, MPEG-H Part II / HEVC / H.265, VP9, ​​VP10 and others, employ a variety of tools, including spatial and temporal prediction, transform, quantization, loop filtering and entropy coding, as part of their encoding process to achieve compression. After the prediction process is performed, transform coding is usually applied to the residual data of a specific block size to better decorrelate this information and achieve energy compression. The coefficients produced after the transform process are quantized and then entropy encoded. This is equivalent to quantization in a single dimension. Uniform quantization is usually used.

[0006] In earlier systems, the discrete cosine transform ("DCT") was used, however, more recently integer approximations of this transform as well as the discrete sine transform ("DST") are used. Other transforms are also used, such as Walsh Hadamard, Haar, etc. Different block sizes are often used for both prediction and transform, such as 4x4, 8x8, 16x16, 32x32, and 64x64 block prediction. Non-square transforms and shape adaptive transforms are also used by some systems, while an adaptive color space transform on the residual data may also be applied before applying the transform.

[0007] For some scenarios, especially for high bitrate / low compression ratio, it is found that skipping the transform process of the block is actually advantageous and provides better coding efficiency than transform coding after quantization. Such a scheme is adopted in HEVC and is named "transform skip" coding mode. On the other hand, in addition to transform coding and quantization, some codecs such as H.263+ also introduce the idea of ​​reducing the resolution of residual data before applying transform and quantization. The decoder will then have to dequantize the data, invert the transform coefficients, and then amplify the resulting block using a predefined interpolation filter to finally reconstruct the residual signal. The residual signal is then added to the prediction signal to generate the final reconstructed block. Additional loop filtering can then be applied. This coding mode is named "reduced resolution update" ("RRU") coding mode. When proposed in H.263 and in MPEG-4AVC / H.264 and HEVC / H.265, this coding mode is applied to the entire image.

[0008] Existing coding techniques offer only limited flexibility to control loop quantization of video. Existing encoders are only able to quantize a single feature of input data at the sub-frame / sub-picture level - the quantization of transform coefficients, which is performed at the block or macroblock level. Some types of quantization are not performed in the loop at all and cannot be controlled at the sub-frame level. Therefore, the inventors believe that there is a need for coding techniques that allow video encoders to quantize input image data along multiple dimensions and select quantization parameters for those dimensions at the sub-frame level. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a simplified block diagram of a video delivery system according to an embodiment of the present disclosure.

[0010] Figure 2 is a functional block diagram of an encoding system according to an embodiment of the present disclosure.

[0011] Figure 3 is a functional block diagram of a decoding system according to an embodiment of the present disclosure.

[0012] Figure 4 Shown according to Figure 2 The coding system and Figure 3 The decoding system is processed by an exemplary pixel block.

[0013] Figure 5 A decoding method according to an embodiment of the present disclosure is shown.

[0014] Figure 6 is a functional block diagram of an encoding system according to an embodiment of the present disclosure.

[0015] Figure 7 For Figure 6A functional block diagram of a decoding system that is operable by an encoding system.

[0016] Figure 8 is a functional block diagram of an encoding system according to another embodiment of the present disclosure.

[0017] Fig. 9 is a functional block diagram of a decoding system according to an embodiment of the present disclosure.

[0018] Fig.10 is a functional block diagram of an encoding system according to another embodiment of the present disclosure.

[0019] Fig.11 is a functional block diagram of a decoding system according to an embodiment of the present disclosure.

[0020] Fig.12 and Fig.13 Exemplary selections of quantization of frames according to embodiments of the present disclosure are shown. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure provide video encoding and decoding techniques for quantizing input video along multiple dimensions. According to these techniques, pixel residuals can be generated by comparing an array of input data with an array of prediction data. Pixel residuals can be quantized in a first dimension. After quantization, the quantized pixel residuals can be transformed into an array of transform coefficients. The transform coefficients can be quantized in a second dimension and entropy encoded. Decoding techniques reverse these processes. In other embodiments, multiple quantizers can be provided upstream of the transform stage in parallel or in cascade, which provides greater flexibility for video encoders to strive for quantized data in different dimensions, thereby balancing the competing interests of compression efficiency and the quality of restored video. Parameter selection of various quantizers can be performed at subframe granularity.

[0022] Figure 1(a) shows a simplified block diagram of a video delivery system 100 according to one embodiment of the present disclosure. The system 100 may include a plurality of terminals 110, 120 interconnected via a network. The terminals 110, 120 may encode video data for transmission to their corresponding terminals via the network. Thus, the first terminal 110 may capture video data locally, encode the video data and transmit the encoded video data to the corresponding terminal 120 via a channel. The receiving terminal 120 may receive the encoded video data, decode it, and present it locally, such as on a display at the terminal 120. If the terminals are engaged in a two-way exchange of video data, the terminal 120 may capture video data locally, encode the video data and transmit the encoded video data to the corresponding terminal 110 via another channel. The receiving terminal 110 may receive the encoded video data transmitted from the terminal 120, decode it, and present it locally, such as on its own display. The process may operate on both frame and field picture encoding, but for simplicity, this discussion will describe the technology in the context of an entire frame.

[0023] The video encoding system 100 can be used in a variety of applications. In a first application, the terminals 110, 120 can support real-time two-way exchange of encoded video to establish a video conferencing session between them. In another application, the terminal 110 can encode pre-generated video (e.g., a television or movie program) and store the encoded video for delivery to one or often to multiple clients (e.g., terminal 120) for downloading. Thus, the video being encoded can be real-time or pre-generated, and the terminal 110 can act as a media server to deliver the encoded video according to a one-to-one or one-to-many distribution model. For the purposes of this discussion, unless otherwise specified, the type of video and the video publishing scheme are irrelevant.

[0024] exist Figure 1 In (a), the terminals 110 and 120 are respectively shown as a smart phone and a tablet computer, but the principles of the present disclosure are not limited thereto. The embodiments of the present disclosure can also be used in computers (both desktop computers and laptop computers), computer servers, media players, dedicated video conferencing equipment and / or dedicated video encoding equipment.

[0025] The network represents any number of networks that transmit the encoded video data between the terminals 110, 120, including, for example, a wired communication network and / or a wireless communication network. The communication network can exchange data in a circuit switching channel and / or a packet switching channel. Representative networks include telecommunication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, unless otherwise specified, the architecture and topology of the network are immaterial to the operation of the present invention.

[0026] Figure 1(b) is a functional block diagram showing components of an encoding terminal. The encoding terminal may include a video source 130, a preprocessor 135, an encoding system 140, and a transmitter 150. The video source 130 may supply the video to be encoded. The video source 130 may be provided as a camera that captures image data of a local environment or a storage device that stores videos from other sources. The preprocessor 135 may perform signal conditioning operations on the video to be encoded for the encoded video data. For example, the preprocessor 135 may change the frame rate, frame resolution, and other characteristics of the source video. The preprocessor 135 may also perform filtering operations on the source video.

[0027] The encoding system 140 may perform encoding operations on the video to reduce its bandwidth. Typically, the encoding system 140 exploits temporal and / or spatial redundancy within the source video. For example, the encoding system 140 may perform motion compensated predictive encoding, in which a video frame or field is parsed into sub-units (referred to as "pixel blocks" for convenience), and each pixel block is differentially encoded relative to a predicted pixel block, which is derived from previously encoded video data. A given pixel block may be encoded according to any of a variety of predictive coding modes, such as:

[0028] Intra-frame coding, where an input pixel block is differentially encoded relative to previously encoded / decoded data of a common frame;

[0029] · uni-predictive inter-frame coding, where the input pixel block is differentially encoded with respect to data from previously encoded / decoded frames; and

[0030] • Bi-predictive inter-frame coding, where an input pixel block is differentially encoded with respect to data from a pair of previously encoded / decoded frames.

[0031] • Combined intra-inter coding, where the input pixel block is differentially encoded with respect to data both from previously encoded / decoded frames and from the current / common frame.

[0032] • Multi-hypothesis intra-inter coding, where an input pixel block is differentially encoded with respect to data from several previously encoded / decoded frames and potentially data from the current / common frame.

[0033] The pixel blocks may also be encoded according to other coding modes such as the transform skip and RRU coding modes discussed previously.

[0034] The encoding system 140 may include an encoder 142, a decoder 143, a loop filter 144, a picture buffer 145, and a predictor 146. The encoder 142 may apply a differential encoding technique to an input pixel block using the predicted pixel block data provided by the predictor 146. The decoder 143 may invert the differential encoding technique applied by the encoder 142 to a subset of the encoded frames designated as reference frames. The loop filter 144 may apply a filtering technique to a reconstructed reference frame generated by the decoder 143. The picture buffer 145 may store the reconstructed reference frame for prediction operations. The predictor 146 may predict the data of the input pixel block from within the reference frame stored in the picture buffer.

[0035] The transmitter 150 may transmit the encoded video data to the decoding terminal through the channel CH.

[0036] Figure 1 (c) is a functional block diagram showing components of a decoding terminal according to an embodiment of the present disclosure. The decoding terminal may include a receiver 160 to receive encoded video data from a channel, a video decoding system 170 to decode the encoded data; a post-processor 180, and a video sink 190 to consume the video data.

[0037] Receiver 160 may receive the data stream from the network and may route the components of the data stream to the appropriate units within video decoding system 170. Figure 1 (b) and Figure 1 (c) shows functional units for video encoding and decoding, but the terminals 110, 120 will typically include an encoding / decoding system for audio data associated with the video, and possibly other processing units (not shown). Thus, the receiver 160 can parse the encoded video data from other elements of the data stream and route it to the video decoding system 170.

[0038] The video decoding system 170 may perform a decoding operation that reverses the encoding operation performed by the encoding system 140. The video decoder may include a decoder 172, a loop filter 173, a picture buffer 174, and a predictor 175. The decoder 172 may reverse the differential encoding technique applied by the encoder 142 into an encoded frame. The loop filter 144 may apply a filtering technique to the reconstructed frame data generated by the decoder 172. For example, the loop filter 144 may perform various filtering operations (e.g., deblocking, de-ringing filtering, sample adaptive offset processing, etc.). The filtered frame data may be output from the decoding system. The picture buffer 174 may store a reconstructed reference frame for a prediction operation. The predictor 175 may predict the data of the input pixel block from the reference frame stored by the picture buffer based on the prediction reference data provided in the encoded video data.

[0039] The post-processor 180 may perform operations to adjust the reconstructed video data for display. For example, the post-processor 180 may perform various filtering operations (e.g., deblocking, de-ringing filtering, etc.), which may blur visual artifacts in the output video generated by the encoding / decoding process. The post-processor 180 may also change the resolution, frame rate, color space, etc. of the reconstructed video to conform to the requirements of the video sink 190.

[0040] The video sink 190 represents various hardware and / or software components in the decoding terminal that can consume the reconstructed video. The video sink 190 may generally include one or more display devices that can present the reconstructed video. Alternatively, the video sink 190 may be represented by a memory system that stores the reconstructed video for later use. The video sink 190 may also include one or more applications that process the reconstructed video data according to controls provided in the application. In some embodiments, the video sink may represent a transmission system that transmits the reconstructed video to a display on another device separate from the decoding terminal; for example, the reconstructed video generated by a laptop computer may be transmitted to a large flat panel display for viewing.

[0041] For encoding terminal and decoding terminal ( Figure 1 (b) to Figure 1 The above discussion of (c)) shows operations performed to encode and decode video data in a single direction between terminals, such as from terminal 110 to terminal 120 ( Figure 1 (a)). In an application where a bidirectional exchange of video is to be performed between the terminals 110, 120, each terminal 110, 120 will have a connection to the encoding terminal ( Figure 1 (b)) and each terminal 110, 120 will also have a functional unit associated with the decoding terminal ( Figure 1 (c)) associated functional units. In fact, in some applications, terminals 110, 120 may exchange multiple streams of encoded video in a single direction, in which case a single terminal (such as terminal 110) will have an encoding terminal ( Figure 1 (b)). Such specific implementations are fully consistent with this discussion.

[0042] Figure 22 is a functional block diagram of an encoding system 200 according to an embodiment of the present disclosure. The system 200 may include a pixel block encoder 210, a pixel block decoder 220, a loop filter 230, a prediction buffer 240, a predictor 250, a controller 260, and a syntax unit 270. The pixel block encoder and decoder 210, 220, and the predictor 250 may iteratively operate on each pixel block of a frame. The predictor 250 may predict the data used during the encoding of the newly presented input pixel block. The pixel block encoder 210 may encode the new pixel block by a predictive coding technique and provide the encoded pixel block data to the syntax unit 270. The pixel block decoder 220 may decode the encoded pixel block data, thereby generating decoded pixel block data. The loop filter 230 may perform various filtering operations on the decoded frame data assembled from the decoded pixel blocks obtained from the pixel block decoder 220. The filtered frame data may be stored in the prediction buffer 240, which may be used as a source for predicting pixel blocks received later. The syntax unit 270 may assemble a data stream from the encoded pixel block data that conforms to the management coding protocol.

[0043] The pixel block encoder 210 may include a subtractor 212, a reducer 213, a residual quantizer 214, a transform unit 215, a coefficient quantizer 216, and an entropy encoder 217. The pixel block encoder 210 may accept a pixel block of input data at the subtractor 212. The subtractor 212 may receive a predicted pixel block from the predictor 250 and thereby generate a pixel residual array representing the difference between the input pixel block and the predicted pixel block. The reducer 213 may perform spatial resolution reduction on the residual data output from the subtractor 212. The residual quantizer 214 may perform quantization on the sample data output from the reducer 213. The quantizer 214 may be a uniform or non-uniform quantizer. The transform unit 215 may apply a transform to the sample data output from the residual quantizer 214 to convert the data from the pixel domain to the transform coefficient domain. The coefficient quantizer 216 may perform quantization on the transform coefficients output by the transform unit 215. The quantizer 216 may be a uniform or non-uniform quantizer. The entropy encoder 217 may reduce the output bandwidth of the coefficient quantizer by encoding the output, for example, with a variable length codeword.

[0044] During operation, the downsizer 213, the residual quantizer 214, and the coefficient quantizer 216 may operate according to encoding parameters that control the operation of each unit. For example, the downsizer 213 may operate according to a resolution quantization parameter (Q R ) which determines the level of downscaling to be applied to its input pixel block. Similarly, the residual quantizer 214 may be based on the coefficient quantization parameter (Q SP ) which determines the quantization level applied to the residual samples input to the quantizer 214. And the coefficient quantizer 216 can be based on the coefficient quantization parameter (Q P) which determines the quantization level to be applied to the transform coefficients input to the coefficient quantizer 216. These quantizers can be based on the signal mode M QP and M QR The signal mode specifies whether the quantizer is a uniform or non-uniform quantizer. Thus, each of the reducer 213, the residual quantizer 214, and the coefficient quantizer 216 operates on a different dimension because they quantize different features of the image data. The quantization dimensions are not necessarily orthogonal to each other. The quantization parameter Q R , Q SP and Q P It may be determined by controller 260 and may be signaled explicitly or implicitly in the encoded video data output by encoding system 200 .

[0045] In one embodiment, the quantization parameter Q R , Q SP and Q P can be applied to its corresponding input data as a multi-valued quantization parameter that can vary, for example, between different pixel positions within a pixel domain pixel block or between different coefficient positions within a transform domain pixel block. R , Q SP and Q P Can be provided as an array of quantization parameters.

[0046] When events warrant, the transform unit 215 can operate in a variety of transform modes. For example, the transform unit 215 can be selected to apply DCT, DST, Walsh Hadamard transform, Haar transform, Daubechies wavelet transform, etc. In an embodiment, the controller 260 can select the coding mode M applied by the transform unit 215, and can configure the transform unit 215 accordingly. The coding mode M can also be explicitly or implicitly signaled in the encoded video data.

[0047] The pixel block decoder 220 may reverse the encoding operations of the pixel block encoder 210. For example, the pixel block decoder 220 may include a coefficient dequantizer 222, an inverse transform unit 223, a residual dequantizer 224, an amplifier 225, and an adder 226. The pixel block decoder 220 may obtain its input data from the output of the coefficient quantizer 216. Although permitted, the pixel block decoder 220 does not have to perform entropy decoding of the entropy encoded data because entropy encoding is a lossless event. The coefficient dequantizer 222 may reverse the operation of the coefficient quantizer 216 of the pixel block encoder 210. The dequantizer 222 may perform the decoding signal M. QPSimilarly, the inverse transform unit 223, the residual dequantizer 224 and the amplifier 225 may invert the operations of the transform unit 215, the residual quantizer 214 and the downsizer 213, respectively. They may use the same quantization parameter Q R , Q SP and Q P and transform mode M as the corresponding part in the pixel block encoder 210. The residual dequantizer 224 may perform the decoding signal M QR The operation of the reducer 213, the residual quantizer 214, and the coefficient quantizer 216 may truncate the data at various points, and therefore, the data recovered by the coefficient dequantizer 222, the residual dequantizer 224, and the amplifier 225 may have coding errors when compared to the data presented to the corresponding parts in the pixel block encoder 210.

[0048] The adder 226 may reverse the operation performed by the subtractor 212. It may receive the same predicted pixel block that the subtractor 212 used to generate the residual signal from the predictor 250. The adder 226 may add the predicted pixel block to the reconstructed residual value output by the amplifier 225, and may output the reconstructed pixel block data.

[0049] The loop filter 230 may perform various filtering operations on the restored pixel block data. For example, the loop filter 230 may include a deblocking filter 232 and a sample adaptive offset ("SAO") filter 233. The deblocking filter 232 may filter data at the seams between the reconstructed pixel blocks to reduce the discontinuity between the pixel blocks due to encoding. The SAO filter may, for example, add an offset to the pixel value according to the SAO "type" based on the edge direction / shape and / or pixel / color component level. The loop filter 230 may operate according to parameters selected by the controller 260.

[0050] Prediction buffer 240 can store filtered pixel data for subsequent prediction of other pixel blocks. For different prediction modes, predictor 250 can use different types of prediction data. For example, for an input pixel block, intra-frame prediction obtains a prediction reference from decoded data of the same frame in which the input pixel block is located. Therefore, prediction buffer 240 can store decoded frame block data for each frame during encoding. For the same input pixel block, inter-frame prediction can use a prediction reference from a previously encoded and decoded frame, which is designated as a "reference frame". Therefore, prediction buffer 240 can store these decoded reference frames.

[0051] As discussed, the predictor 250 may provide prediction data to the pixel block encoder 210 for generating a residual. The predictor 250 may include an inter-frame predictor 252, an intra-frame predictor 253, and a mode decision unit 254. The inter-frame predictor 252 may receive pixel block data representing a new pixel block to be encoded, and may search the prediction buffer 240 for pixel block data from a reference frame for encoding the input pixel block. The inter-frame predictor 252 may support multiple prediction modes, such as P-mode encoding and B-mode encoding. The inter-frame predictor 252 may select an inter-frame prediction mode and provide prediction data that provides the closest match to the input pixel block being encoded. The inter-frame predictor 252 may generate a prediction reference indicator, such as a motion vector, to identify which portion of the reference frame is selected as a prediction source for the input pixel block.

[0052] The intra predictor 253 may support intra (I) mode encoding. The intra predictor 253 may search for encoded pixel block data from the same frame as the encoded pixel block that provides the closest match to the input pixel block. The intra predictor 253 may also generate a prediction reference indicator to identify which portion of the frame is selected as the prediction source for the input pixel block.

[0053] The mode decision unit 254 may select a final encoding mode to be applied to the input pixel block. Typically, the mode decision unit 254 selects a prediction mode that will achieve the lowest distortion when decoding the video given a target bit rate. Exceptions may occur when the encoding mode is selected to meet other strategies followed by the encoding system 200, such as meeting specific channel behaviors, or supporting random access or data refresh strategies. The mode decision unit 254 may output prediction data to the pixel block encoder and decoder 210, 220, and may provide the controller 260 with an identification of the selected prediction mode and a prediction reference indicator corresponding to the selected mode.

[0054] The controller 260 may control the overall operation of the encoding system 200. The controller 260 may select operating parameters of the pixel block encoder 210 and the predictor 250 based on an analysis of the input pixel block and external constraints such as encoding bit rate targets and other operating parameters. Relevant to the present discussion, when it selects quantization parameters such as the reduction parameter Q R , residual quantization parameter Q SP , coefficient quantizer Q P, use uniform or non-uniform quantizers, and / or transform mode M, it may provide those parameters to syntax unit 270, which may include data representing those parameters in a data stream of encoded video data output by system 200. Controller 260 may also determine that one or more quantizer units (e.g., coefficient quantizer 216) should not apply quantization, in which case controller 260 may disable the quantizer. As described above, controller 260 may also determine the operating mode of transform unit 215.

[0055] As discussed, the quantization parameter Q R , Q SP and / or Q P Rather than being provided as a single value that is applied uniformly to all pixels of a pixel block, it may be provided as an array of corresponding quantization parameters whose values ​​vary at each pixel position. In one embodiment, a plurality of quantizer arrays may be defined according to a predetermined protocol, each protocol having corresponding quantization values ​​at different pixel positions; during operation, the controller 260 may identify the array to be used by providing an index or other identifier.

[0056] In another embodiment, the quantization parameter Q R , Q SP and / or Q P The quantization parameter Q (whether a single value or an array of values) may be derived from features of the image data. SP can be selected based on the strength of the predicted pixel value corresponding to the pixel residual. In this case, Q SP The value of can vary for each pixel depending on the strength value of the predictor, e.g., for samples with larger predictor values, Q SP The value of may be higher than samples with smaller predictor values. These quantization parameters may be defined using linear, ie, scalar, formulas, or using more complex non-linear / non-uniform formulas.

[0057] In one embodiment, the quantizer may be represented by a predetermined function of the following form: q(i, j) = f k (p(i, j), i, j), where q represents the quantizer in question, i and j represent the position of the value within the pixel block being quantized (pixel domain value or transform domain value), p(i, j) represents the pixel intensity value being quantized, and f k represents the function that relates p, i, and j to q. As part of the signaling, the controller 260 may provide data identifying the function being used, as well as any other data that the corresponding function operates on (eg, any scalars and offsets that may operate at the corresponding function).

[0058] The quantization process may also be specified using a lookup table (not shown). The lookup table may involve a quantization / dequantization process that uses interpolation techniques to determine certain values ​​not included in the lookup table. Alternatively, the quantization parameter Q R and / or Q SP , Q P The first set of quantization parameters Q R and / or Q SP , Q P A set of quantization parameters Q may be signaled in a higher level structure of the coding sequence (e.g., a frame or slice header) and signaled R and / or Q SP , Q P The replacement of may be signaled on a pixel block by pixel block basis when the constructed pixel blocks are encoded.

[0059] During operation, controller 260 may modify operating parameters of downsizer 213, residual quantizer 214, coefficient quantizer 216, and transform unit 215 at different granularities of image data, either on a per-pixel-block basis or on a larger granularity, such as per-frame, per-slice, per-largest coding unit ("LCU"), or another region. In an embodiment, quantization parameters may be modified within a coded frame on a per-pixel basis.

[0060] Additionally, as discussed, controller 260 may control the operation of loop filter 230 and predictor 250. For predictor 250, such control may include mode selection (λ, mode to be tested, search window, distortion strategy, etc.), and for loop filter 230, selection of filter parameters, reordering parameters, weighted prediction, etc.

[0061] Figure 3 3 is a functional block diagram of a decoding system 300 according to an embodiment of the present disclosure. The decoding system 300 may include a syntax unit 310, a pixel block decoder 320, a loop filter 330, a prediction buffer 340, and a predictor 350. The syntax unit 310 may receive an encoded video data stream and may parse the encoded data into its components. Data representing encoding parameters may be provided to a controller 360, and data representing an encoding residual ( Figure 2 The pixel block encoder 210 output data) may be provided to the pixel block decoder 320. The pixel block decoder 320 may invert the encoding operation ( Figure 2). Loop filter 330 may filter the reconstructed pixel block data. The reconstructed pixel block data may be assembled into frames for display and output from decoding system 300 as output video. Frames may also be stored in prediction buffer 340 for prediction operations. Predictor 350 may provide prediction data to pixel block decoder 320, the prediction data being determined by the encoded data received in the encoded video data stream.

[0062] The pixel block decoder 320 may include an entropy decoder 321, a coefficient dequantizer 322, an inverse transform unit 323, a residual dequantizer 324, an amplifier 325, and an adder 326. The entropy decoder 321 may perform entropy decoding to reverse the process performed by the entropy encoder 217 ( Figure 2 The coefficient dequantizer 322 may invert the operation of the coefficient quantizer 216 of the pixel block encoder 210 ( Figure 2 Similarly, the inverse transform unit 323, the residual dequantizer 324, and the amplifier 325 may inverse the operations of the transform unit 215, the residual quantizer 214, and the downsizer 213, respectively ( Figure 2 ). They can use the quantization parameter Q R ,Q SP , and Q P , and quantizer mode type M QP and M QR , the quantizer mode type is provided in the encoded video data stream. Because the operation of the reducer 213, the residual quantizer 214 and the coefficient quantizer 216 truncates the data at various points, the data recovered by the coefficient dequantizer 322, the residual dequantizer 324 and the amplifier 325 may have encoding errors ( Figure 2 ).

[0063] Adder 326 may invert the operation performed by subtractor 212 ( Figure 2 ). It may receive a predicted pixel block from the predictor 350, which is determined by a prediction reference in the encoded video data stream. The adder 326 may add the predicted pixel block to the reconstructed residual value output by the amplifier 325, and may output the reconstructed pixel block data.

[0064] The loop filter 330 may perform various filtering operations on the reconstructed pixel block data. As shown, the loop filter 330 may include a deblocking filter 332 and an SAO filter 333. The deblocking filter 332 may filter data at the seams between the reconstructed pixel blocks to reduce the interruption between the pixel blocks due to encoding. The SAO filter may, for example, add an offset to the pixel value according to the SAO type based on the edge direction / shape and / or pixel level. Other types of loop filters may also be used in a similar manner. The operation of the deblocking filter 332 and the SAO filter 333 will ideally simulate the operation of their counterparts in the encoding system 200 ( Figure 2 ). Therefore, in the absence of transmission errors or other anomalies, the decoded frame data obtained from the loop filter 330 of the decoding system 300 will be different from the decoded frame data obtained from the loop filter 230 of the encoding system 200 ( Figure 2 ) are the same; thus, the encoding system 200 and the decoding system 300 should store a common set of reference pictures in their respective prediction buffers 240, 340.

[0065] The prediction buffer 340 may store filtered pixel data for subsequent prediction of other pixel blocks. The prediction buffer 340 may store decoded pixel block data for each frame when encoded for intra-frame prediction. The prediction buffer 340 may also store decoded reference frames.

[0066] As discussed, predictor 350 may provide prediction data to pixel block decoder 320. Predictor 350 may provide prediction pixel block data determined by prediction reference indicators provided in the encoded video data stream.

[0067] Controller 360 may control the overall operation of decoding system 300. Controller 360 may set operating parameters of pixel block decoder 320 and predictor 350 based on parameters received in the encoded video data stream. As relevant to this discussion, these operating parameters may include quantization parameters, such as reduction parameters Q R , residual quantization parameter Q SP / or coefficient quantizer Q P and operation mode M of the inverse transform unit 323. As discussed, the received parameters may be set at various granularities of the image data, for example, on a per-pixel-block basis, per-frame basis, per-slice basis, per-LCU basis, or based on other types of regions of data defined for the input image.

[0068] Figure 4 Shown according to Figure 2 The coding system 200 and Figure 3 The decoding system 300 is processed by an exemplary pixel block. Initially, as Figure 4As shown in (a), the pixel block 410 may be an M×N pixel array, each pixel array being represented by a bit depth B for each color component. After resolution quantization (reduction), the pixel block 420 may be an M / QF X ×N / QF Y Pixel array, which still has a bit depth B for each color component. After coefficient quantization, pixel block 430 can be M / QF X ×N / QF Y Pixel array with varying bit depth. Here, the sample value is divided by the sample precision quantization value (Q SP ). After transformation and quantization, the pixel block 440 may be a transform coefficient array M / QF X ×N / QF Y , whose value will be quantized by the quantization parameter QP.

[0069] Figure 4 (b) shows the processing performed by the coded pixel block during decoding. Initially, the coded pixel block 450 may be a transform coefficient array M / QF X ×N / QF Y , whose value will be determined by Figure 4 The quantization parameter Q shown in (a) P Quantization. After inverse transformation and quantization, pixel block 460 may be M / QF X ×N / QF Y An array of pixels whose values ​​correspond to a quantization parameter Q P The pixel blocks 430 that have the induced loss. After sample precision quantization, the pixel blocks 470 can be M / QF X ×N / QF Y pixel array, which has a bit depth B for each color component, although with a quantization parameter Q SP Further losses induced. After reverse resolution quantization (upscaling), pixel block 480 may be an M×N array of pixels, each represented with a bit depth B per color component, again with coding losses.

[0070] like Figure 4 As shown, Figure 2 and Figure 3 Embodiments of the present invention provide for quantization of image data along several dimensions. In this embodiment, in addition to the quantization of coefficient transforms, quantization can be applied to the resolution and bit depth of the image data. Therefore, these techniques provide greater flexibility for encoding / decoding systems to achieve bandwidth compression and maintain encoding quality.

[0071] Figure 55 shows a decoding method 500 according to one embodiment of the present disclosure. The method 500 may operate on the encoded video data on a pixel block by pixel block basis. The method 500 may process metadata provided with the encoded video data to determine which encoding modes to apply to encode the decoded pixel blocks (block 510). The method may determine whether the encoded pixel block residual is part of a transform block based on the encoding mode data (block 515). If yes, the method 500 may determine whether the encoded pixel block residual is part of a transform block based on the quantization parameter (Q P ) to dequantize the transform coefficients (block 520) and an inverse transform may be applied according to the transform mode M identified by the encoded video data (block 525). P The transform mode M may be identified explicitly or implicitly by the encoded video data.

[0072] At block 530, method 500 may determine whether the residual data present at block 515 or reconstructed at block 525 is in the domain of the prediction data. If not, method 500 may map the residual data to the domain of the prediction data, which is the pixel domain (block 535). Then, or if it is determined at block 530 that the residual data is already in the domain of the prediction data, method 500 may merge the prediction data with the data reconstructed by processing blocks 510-535 (block 540).

[0073] The mapping of the residual data to the prediction data domain may occur by reversing the quantization operation applied during the encoding of the pixel block. Figure 5 shows that the decoded Figure 2 Thus, the method 500 may determine whether to apply residual quantization (block 545), and if so, the method 500 may use the parameter Q provided with the encoded video data. SP 550). Similarly, the method 500 may determine whether downsampling is applied (block 555), and if so, whether the downsampling is applied only to chroma or to all color components (block 560). If downsampling is applied to all components, the method 500 may upscale all components of the reconstructed data (block 565). If downsampling is applied only to chroma, the method 500 may upscale the chroma components of the reconstructed data (block 570). Likewise, the quantization parameter Q for downsampling may be 560. Q May be provided with the encoded video either explicitly or implicitly.

[0074] Figure 66 is a functional block diagram of an encoding system 600 according to an embodiment of the present disclosure. The system 600 may include a pixel block encoder 610, a pixel block decoder 620, a pair of loop filter systems 630.1, 630.2, a pair of prediction buffers 640.1, 640.2, a prediction system 650, a controller 660 and a syntax unit 670. As with the previous embodiments, the pixel block encoder and decoder 610, 620 and the prediction system 650 may operate iteratively on individual pixel blocks of image data, but the loop filters 630.1, 630.2 and the prediction buffers 640.1, 640.2 may operate on larger data units. The prediction system 650 may predict data used during the encoding of a newly presented input pixel block.

[0075] The pixel block encoder 610 may encode the new pixel block by predictive coding techniques and provide the encoded pixel block data to the syntax unit 670. The syntax unit 670 may construct a transport bitstream from the encoded video data and other sources (not shown), such as sources of encoded audio data and / or application data. The syntax unit 670 may assemble a data stream from the encoded pixel block data that conforms to the governing encoding protocol.

[0076] The pixel block decoder 620 may be a decoder local to the pixel block encoder 610, which decodes the coded pixel block data to generate decoded pixel block data. In this embodiment, the pixel block decoder 620 may generate two versions of decoded pixel blocks. The first version may be generated by limited decoding (for convenience, referred to as "best effort" decoding) of the coded pixel block data, for example, by reversing coefficient sampling, reversing the coefficient transformation process performed by the pixel block encoder 610 and performing limited prediction operations. The second version may be generated by reversing all encoding processes performed by the pixel block encoder 610 ("best effort" decoding), including, for example, downsampling, residual quantization, etc. The pixel block decoder 620 may output the two versions of the decoded pixel blocks to the corresponding loop filters 630.1, 630.2.

[0077] The loop filters 630.1, 630.2 and the prediction buffers 640.1, 640.2 may be provided in a paired relationship. The first loop filter 630.1 and the prediction buffer 640.1 may operate on the decoded pixel block data output from the full decoding operation performed by the pixel block decoder 620. The second loop filter 630.2 and the prediction buffer 640.2 may operate on the decoded pixel block data output from the restricted decoding operation performed by the pixel block decoder 620. In various cases, the loop filters 630.1, 630.2 may perform various filtering operations on their respective decoded frame data. The filtered frame data generated by the loop filters 630.1, 630.2 may be stored in their respective prediction buffers 640.1, 640.2, where they may be used as a prediction source for pixel blocks received later.

[0078] The prediction system 650 can generate prediction data for an input pixel block by performing a motion search within a prediction buffer 640.1 associated with best effort decoding. The prediction system 650 can output the prediction data from the prediction buffer 640.1 to the pixel block encoder 610 and the pixel block decoder 620 for the best effort decoding process. The prediction system 650 can also provide the corresponding pixel block from the prediction buffer 640.2 to the pixel block decoder 620 for best effort decoding.

[0079] The pixel block encoder 610 may include a subtractor 612, a reducer 613, a residual quantizer 614, a transform unit 615, a coefficient quantizer 616, and an entropy encoder 617. The pixel block encoder 610 may accept a pixel block of input data at the subtractor 612. The subtractor 612 may receive a predicted pixel block from the prediction system 650 and thereby generate a pixel residual array representing a pixel-by-pixel difference between the input pixel block and the predicted pixel block. The reducer 613 may perform spatial resolution reduction on the residual data output from the subtractor 612. The residual quantizer 614 may perform quantization on the sample data output from the reducer 613. The transform unit 615 may apply a transform to the sample data output from the residual quantizer 614 to convert the data from the pixel domain to the transform coefficient domain. The coefficient quantizer 616 may perform quantization on the transform coefficients output by the transform unit 615. The entropy encoder 617 may reduce the output bandwidth of the coefficient quantizer by, for example, encoding the output by a variable length codeword.

[0080] During operation, the downsizer 613, the residual quantizer 614, and the coefficient quantizer 616 may operate according to encoding parameters that control the operation of each unit. For example, the downsizer 613 may operate according to the resolution quantization parameter Q R The resolution quantization parameter determines the level of downscaling to be applied to its input pixel block. Similarly, the residual quantizer 614 may be based on the residual quantization parameter Q SPThe coefficient quantization parameter determines the quantization level applied to the residual samples input to the residual quantizer 614. And the coefficient quantizer 616 can be based on the coefficient quantization parameter Q P The coefficient quantization parameter determines the level of quantization to be applied to the transform coefficients input to the coefficient quantizer 616. As with the previous embodiment, each of the downsizer 613, the residual quantizer 614, and the coefficient quantizer 616 operates in a different dimension because they quantize different features of the image data; the quantization dimensions are not necessarily orthogonal to each other. The quantization parameter Q R , Q SP and Q P It may be determined by controller 660 and may be signaled explicitly or implicitly in the encoded video data output by encoding system 600. These quantizers may also operate in a uniform or non-uniform manner.

[0081] Transform unit 615 may operate in a variety of transform modes when events warrant. For example, transform unit 615 may be selected to apply DCT, DST, Walsh Hadamard transform, Haar transform, Daubechies wavelet transform, etc. In an embodiment, controller 660 may select the coding mode M applied by transform unit 615, and may configure transform unit 615 accordingly. Coding mode M may also be explicitly or implicitly signaled in the encoded video data.

[0082] The pixel block decoder 620 may reverse the encoding operations of the pixel block encoder 610. For example, the pixel block decoder 620 may include a coefficient dequantizer 622, an inverse transform unit 623, a residual dequantizer 624, an amplifier 625, and a pair of adders 626.1, 626.2. The pixel block decoder 620 may obtain its input data from the output of the coefficient quantizer 616. Although allowed, the pixel block decoder 620 does not have to perform entropy decoding of the entropy encoded data because entropy encoding is a lossless event. The coefficient dequantizer 622 may reverse the operation of the coefficient quantizer 616 of the pixel block encoder 610. Similarly, the inverse transform unit 623, the residual dequantizer 624 and the amplifier 625 may reverse the operations of the transform unit 615, the residual quantizer 614 and the reducer 613, respectively. They may use the same quantization parameter Q R , Q SP and Q P and transform mode M as the counterpart in the pixel block encoder 610. The operations of the reducer 613, the residual quantizer 614, and the coefficient quantizer 616 may truncate the data at various points, and thus, the data recovered by the coefficient dequantizer 622, the residual dequantizer 624, and the amplifier 625 may have coding errors when compared to the data presented to the counterpart in the pixel block encoder 610.

[0083] The adder 626.1 may invert the operation performed by the subtractor 612. It may receive the same predicted pixel block from the prediction system 650 that the subtractor 612 used to generate the residual signal. The adder 626.1 may add the predicted pixel block to the reconstructed residual value output by the amplifier 625 and may output the reconstructed pixel block data.

[0084] The coefficient dequantizer 622, the inverse transform unit 623, the residual dequantizer 624, the amplifier 625 and the first adder 626.1 may define a full-power decoding path of the pixel block decoder 620. It comprises a complete array of decoding units that operate as counterparts of the subunits of the corresponding pixel block encoder 610, namely the subtractor 612, the reducer 613, the residual quantizer 614, the transform unit 615 and the coefficient quantizer 616.

[0085] The second adder 626.2 may take its input from the inverse transform unit 623, which represents pixel residual data generated from a set of decoding units that is not a complete set of counterparts of the coding units of the pixel block encoder 610. Thus, the second adder 626.2 defines the best-effort decoding path of the pixel block decoder 620. The second adder 626.2 accepts prediction data from the prediction system 650 obtained from the best-effort prediction buffer 640.2 using the prediction reference derived from the best-effort prediction buffer 640.1. That is, if the prediction system 650 selects a pixel block from a given frame from the best-effort prediction buffer 640.1 as a prediction reference for the input pixel block, it provides the selected pixel block to the subtractor 612 and the best-effort adder 626.1. The prediction system 650 also supplies the pixel block to the second adder 626.2 from the same position of the same frame as stored in the best-effort prediction buffer 640.2.

[0086] The loop filters 630.1, 630.2 may perform various filtering operations on their corresponding pixel block data. For example, the loop filters 630.1, 630.2 may include a deblocking filter and an SAO filter (not shown). The deblocking filter may filter data at the seams between reconstructed pixel blocks to reduce interruptions between pixel blocks due to encoding. The SAO filter may add an offset to the pixel value according to the SAO type, for example based on edge direction / shape and / or pixel level. The loop filters 630.1, 630.2 may operate according to parameters selected by the controller 660.

[0087] The prediction buffers 640.1, 640.2 may store filtered pixel data from the corresponding loop filters 630.1, 630.2 for subsequent prediction of other pixel blocks. For different prediction modes, the prediction system 650 may use different types of prediction data. As discussed, for an input pixel block, intra-frame prediction obtains a prediction reference from decoded data of the same frame, locating the input pixel block in the same frame. Therefore, the prediction buffers 640.1, 640.2 may store decoded frame block data for each frame at the time of encoding. For the same input pixel block, inter-frame prediction may employ a prediction reference from a previously encoded and decoded frame, which is designated as a "reference frame". Therefore, the prediction buffers 640.1, 640.2 may store these decoded reference frames.

[0088] As discussed, the prediction system 650 may provide prediction data to the pixel block encoder 610 for generating residuals. The prediction system 650 may include a mode decision unit 652 and a pair of predictors 654.1, 654.2. The mode decision unit 652 may include the inter-frame predictor and the intra-frame predictor of the aforementioned embodiment. Therefore, the mode decision unit 652 may receive pixel block data representing a new pixel block to be encoded, and may search for pixel block data from a reference frame in the full-power prediction buffer 640.1 for encoding the input pixel block. The decision unit 652 may support multiple prediction modes, such as a single list prediction mode and a dual prediction mode. When inter-frame coding is selected, the mode decision unit may generate a prediction reference indicator, such as a motion vector, to identify which parts of the reference frame are selected as prediction sources for the input pixel block.

[0089] The mode decision unit 652 may also support intra (I) mode coding. Thus, the mode decision unit 652 may search the full power prediction buffer 640.1 for coded pixel block data from the same frame as the coded pixel block that provides the closest match to the input pixel block. When I coding is selected, the mode decision unit 652 may also generate a prediction reference indicator to identify which portion of the frame is selected as the prediction source for the input pixel block.

[0090] As discussed, in general, mode decision unit 652 selects the prediction mode that will achieve the lowest distortion when decoding video. Exceptions may occur when encoding modes are selected to satisfy other policies followed by encoding system 600, such as supporting random access or data refresh policies. In an embodiment, mode decision unit 652 may monitor the accumulated encoding errors in the decoded video data obtained by the best effort decoding path and the best effort decoding path. For example, if it determines that the accumulated encoding error in the best effort decoding path exceeds a predetermined value, it may change its default prediction decision.

[0091] The best effort predictor 654.1 may supply prediction data from the best effort prediction buffer 640.1 identified by the prediction reference generated by the mode decision unit 652 to the subtractor 612 and the best effort adder 626.1. The best effort predictor 654.2 may supply prediction data from the best effort prediction buffer 640.2 identified by the prediction reference generated by the mode decision unit 652 to the best effort adder 626.2.

[0092] The controller 660 may control the overall operation of the encoding system 600. The controller 660 may select operating parameters for the pixel block encoder 610 and the prediction system 650 based on an analysis of the input pixel block and external constraints such as encoding bit rate targets and other operating parameters. Relevant to the present discussion, when it selects quantization parameters such as the reduction parameter Q R , residual quantization parameter Q SP , coefficient quantizer Q P , and / or transform mode M, which may provide those parameters to syntax unit 670, which may include data representing those parameters in a data stream of encoded video data output by system 600. Controller 660 may also determine that one or more quantizer units (e.g., coefficient quantizer 616) should not apply quantization, in which case controller 660 may disable the quantizer. As described above, controller 660 may also determine the operating mode of transform unit 615.

[0093] As discussed, the quantization parameter Q R , Q SP and / or Q P Rather than being provided as a single value that is applied uniformly to all pixels of a pixel block, it may be provided as an array of corresponding quantization parameters whose values ​​vary at each pixel position. In one embodiment, a plurality of quantizer arrays may be defined according to a predetermined protocol, each protocol having corresponding quantization values ​​at different pixel positions; during operation, the controller 660 may identify the array to be used by providing an index or other identifier.

[0094] In another embodiment, the quantization parameter Q R , Q SP and / or Q P The quantization parameter Q (whether a single value or an array of values) may be derived from features of the image data. SP The quantization parameters may be selected for each sample position based on the strength of its corresponding prediction sample, e.g., a larger quantizer value may be selected for a larger prediction sample value than for a smaller prediction sample value. These quantization parameters may be defined using a linear, i.e., scalar, formula, or using a more complex non-linear / non-uniform formula.

[0095] In one embodiment, the quantizer may be represented by a predetermined function of the following form: q(i, j) = fk (p(i, j), i, j), where q represents the quantizer in question, i and j represent the position of the value within the pixel block being quantized (pixel domain value or transform domain value), p(i, j) represents the pixel intensity value being quantized, and f k represents the function that relates p, i, and j to q. As part of the signaling, the controller 260 may provide data identifying the function being used, as well as any other data that the corresponding function operates on (eg, any scalars and offsets that may operate at the corresponding function).

[0096] The quantization process may also be specified using a lookup table (not shown). The lookup table may involve a quantization / dequantization process that uses interpolation techniques to determine certain values ​​not included in the lookup table. Alternatively, the quantization parameter Q R and / or Q SP , Q P The first set of quantization parameters Q R and / or Q SP , Q P A set of quantization parameters Q may be signaled in a higher level structure of the coding sequence (e.g., a frame or slice header) and signaled R and / or Q SP , Q P The replacement of may be signaled on a pixel block by pixel block basis when the constructed pixel blocks are encoded.

[0097] During operation, controller 660 may modify operating parameters of downsizer 613, residual quantizer 614, and coefficient quantizer 616, and transform unit 615 at different granularities of image data, either on a per-pixel-block basis or on a larger granularity (e.g., per-frame, per-slice, per-LCU, or another region). In an embodiment, the quantization parameters may be modified within a coded frame on a per-pixel basis.

[0098] Additionally, as discussed, the controller 660 may control the operation of the loop filters 630.1, 630.2 and the prediction system 650. For the prediction system 650, such control may include mode selection (λ, mode to be tested, search window, distortion strategy, etc.), and for the loop filters 630.1, 630.2, selection of filter parameters, reordering parameters, weighted prediction, etc.

[0099] Figure 6 The encoding system 600 provides Figure 2 The coding flexibility provided in the implementation scheme of the invention and, in addition, the application of various types of decoders. It can be used in the absence of transmission errors with Figure 3The encoding system 600 may also work with the decoding system 300, which may alternate between full decoding mode and best effort mode, represented by the bypass path 327 ( Figure 3 ).

[0100] Figure 6 The encoding system 600 can also generate encoded video data that is operated using a less capable video decoder, such as the video decoding system 700. Figure 7 The decoder 720 is shown with Figure 3 Compared with the decoding system 300, the residual dequantizer 324 or the amplifier 325 is not provided. Also, if the encoding system 600 ( Figure 6 ) makes encoding decisions that account for and mitigate the accumulation of encoding errors in the best-effort decoding path, then the decoding system 700, such as Figure 7 As shown in , the encoded video can be decoded with deliverable encoding performance.

[0101] Figure 8 8 is a functional block diagram of an encoding system 800 according to another embodiment of the present disclosure. The system 800 may include a pixel block encoder 810, a pixel block decoder 820, a loop filter system 830, a prediction buffer 840, a predictor 850, a controller 860 and a syntax unit 870. The pixel block encoder and decoder 810, 820 and the predictor 850 may operate iteratively on each pixel block. The predictor 850 may predict the data used during the encoding of the newly presented pixel block. The pixel block encoder 810 may encode the new pixel block by a predictive coding technique and provide the encoded pixel block data to the syntax unit 870. The pixel block decoder 820 may decode the encoded pixel block data, thereby generating decoded pixel block data. The loop filter 830 may perform various filtering operations on the decoded frame data assembled from the decoded pixel blocks obtained from the pixel block decoder 820. The filtered frame data may be stored in the prediction buffer 840, which may be used as a source for predicting pixel blocks received later. The syntax unit 870 may assemble a data stream from the encoded pixel block data that conforms to the management coding protocol.

[0102] The pixel block encoder 810 may include a subtractor 812, a plurality of quantizers 813.1-813.N, a transform unit 814, a coefficient quantizer 815, and an entropy encoder 816. The pixel block encoder 810 may accept a pixel block of input data at the subtractor 812. The subtractor 812 may receive a predicted pixel block from the predictor 850 and thereby generate a pixel residual array representing the difference between the input pixel block and the predicted pixel block. The quantizers 813.1, 813.N may both perform quantization on the data they receive in different domains of the image data. The transform unit 814 may apply a transform to the data input thereto from the last one of the quantizers 813.N to convert the input data from the pixel domain to the domain of the transform coefficients. The coefficient quantizer 815 may perform quantization on the transform coefficients output by the transform unit 814. The entropy encoder 816 may reduce the output bandwidth of the coefficient quantizer by, for example, encoding the output by a variable length codeword.

[0103] The quantizers 813.1-813.N can each perform quantization on the data they receive in the corresponding domain of the image data. Here, the quantization dimensions do not have to be orthogonal to each other. The quantizers 813.1-813.N can include the reducers and residual quantizers of the above-mentioned embodiments. They can also include other types of quantizers, such as color component specific quantizers, frequency domain quantizers, orientation specific quantizers (e.g., horizontal only, vertical only, mixed), brightness-specific residual scalers, etc. The quantizers 813.1-813.N can provide quantization based on corresponding quantization parameters Q1-Q2 that define the quantization levels they provide. N Quantization parameters Q1-Q N This may be signaled to the decoding system explicitly or implicitly in the encoded bitstream.In some cases, individual quantizers 813.1, 813.2, ..., 813.N may be disabled, in which case they may simply pass their input data to the next quantizer without change.

[0104] The transform unit 814 may operate in a variety of transform modes when events warrant. For example, the transform unit 814 may be selected to apply a DCT transform, a DST transform, a Walsh Hadamard transform, a Haar transform, a Daubechies wavelet transform, etc. In an embodiment, the controller 860 may select the encoding mode applied by the transform unit 814 and may configure the transform unit 814 accordingly. The transform mode M may also be explicitly or implicitly signaled in the encoded video data.

[0105] The pixel block decoder 820 may include a decoding unit that reverses the encoding operation of the pixel block encoder 810. For example, the pixel block decoder 820 may include a coefficient dequantizer 822, an inverse transform unit 823, a plurality of dequantizers 824.1-824.N, and an adder 825. The pixel block decoder 820 may obtain its input data from the output of the coefficient quantizer 815. Again, although permitted, the pixel block decoder 820 does not have to perform entropy decoding of the entropy encoded data because entropy encoding is a lossless event. The coefficient dequantizer 822 may reverse the operation of the coefficient quantizer 815 of the pixel block encoder 810. Similarly, the inverse transform unit 823 and the dequantizers 824.1-824.N may reverse the operation of the transform unit 814 and the quantizers 813.1-813.N, respectively. They may use the same quantization parameters Q1-Q N and transform mode M as the counterpart in the pixel block encoder 810. The operation of the quantizers 813.1-813.N and the coefficient quantizer 815 may truncate the data at various points, and therefore, the data recovered by the dequantizers 824.1-824.N may have encoding errors when compared to the data presented to the counterpart in the pixel block encoder 810.

[0106] The adder 825 may reverse the operation performed by the subtractor 812. It may receive the same predicted pixel block from the predictor 850 that the subtractor 812 used to generate the residual signal. The adder 825 may add the predicted pixel block to the reconstructed residual value output by the final dequantizer 824.1 and may output the reconstructed pixel block data.

[0107] The loop filter 830 may perform various filtering operations on the reconstructed pixel block data. For example, the loop filter 830 may include a deblocking filter 832 and an SAO filter 833. The deblocking filter 832 may filter data at the seams between the reconstructed pixel blocks to reduce the interruption between the pixel blocks due to encoding. The SAO filter 833 may, for example, add an offset to the pixel value according to the SAO type based on the edge direction / shape and / or pixel level. The loop filter 830 may operate according to the parameters selected by the controller 860.

[0108] Prediction buffer 840 can store filtered pixel data for subsequent prediction of other pixel blocks. For different prediction modes, predictor 850 can use different types of prediction data. For example, for an input pixel block, intra-frame prediction obtains a prediction reference from decoded data of the same frame in which the input pixel block is located. Therefore, prediction buffer 840 can store decoded frame block data for each frame during encoding. For the same input pixel block, inter-frame prediction can use a prediction reference from a previously encoded and decoded frame, which is designated as a reference frame. Therefore, prediction buffer 840 can store these decoded reference frames.

[0109] As discussed, the predictor 850 may provide prediction data to the pixel block encoder 810 for generating a residual. The predictor 850 may include a motion estimator 852, an intra-frame predictor 853, and a mode decision unit 854. The motion estimator 852 may receive pixel block data representing a new pixel block to be encoded, and may search for pixel block data from a reference frame in a prediction buffer 840 for encoding the input pixel block. The motion estimator 852 may support multiple prediction modes, such as P and B mode encoding, supplying prediction data from one or a pair of reference frames. The motion estimator 852 may select an inter-frame prediction mode and provide prediction data that provides the closest match to the input pixel block being encoded. The motion estimator 852 may generate a prediction reference indicator, such as a motion vector, to identify which portions of the reference frame are selected as prediction sources for the input pixel block.

[0110] The intra-frame predictor 853 may support intra-frame coding. The intra-frame predictor 853 may search for coded pixel block data from the same frame as the coded pixel block that provides the closest match to the input pixel block. The intra-frame predictor 853 may also generate a prediction reference indicator to identify which portion of the frame is selected as the prediction source for the input pixel block.

[0111] In another embodiment, the predictor 850 may determine that prediction should not be performed when encoding an input pixel block. In this case, the predictor 850 may disable the subtractor 812 (not effectively providing prediction data to the subtractor 812), and the subtractor 812 may output the pixel value to the quantizer chain 813.1-813.N. In this mode of operation, the quantizers 813.1-813.N will operate on pixel values ​​rather than pixel residuals. Similarly, the pixel block decoder 820 may disable the adder 825 when decoding a pixel block encoded in this manner. In addition, the controller 860 may provide a mode decision signal in the decoded bit stream, which when processed by the decoder ( Fig. 9 ), when decoding a pixel block encoded in this way, the decoded bit stream will also have the adder in the decoder disabled.

[0112] The mode decision unit 854 may select a final encoding mode to be applied to the input pixel block. Typically, when decoding video, the mode decision unit 854 selects a prediction mode that will achieve the lowest distortion. Exceptions may occur when the encoding mode is selected to meet other strategies followed by the encoding system 800, such as supporting random access or data refresh strategies. The mode decision unit 854 may output prediction data to the pixel block encoder and decoder 810, 820, and may provide an identification of the selected prediction mode and a prediction reference indicator corresponding to the selected mode to the controller 860.

[0113] The controller 860 may control the overall operation of the encoding system 800. The controller 860 may select operating parameters of the pixel block encoder 810 and the predictor 850 based on analysis of the input pixel block and external constraints such as encoding bit rate targets and other operating parameters. Relevant to the present discussion, when it selects quantization parameters Q1-Q N , and transform mode M, it may provide those parameters to syntax unit 870, which may include data representing those parameters in a data stream of encoded video data output by system 800. Controller 860 may also determine that one or more quantizer units (e.g., quantizer 813.2) should not apply quantization, in which case controller 860 may disable quantizer 813.2. As described above, controller 860 may also determine the operating mode of transform unit 814.

[0114] In another embodiment, the quantization parameters Q1-Q N (whether a single value or an array of values) may be derived from features of the image data. For example, a quantization parameter Q1 may be selected based on the strength of the predicted pixel value corresponding to the pixel residual. In this case, the value of Q1 may vary for each pixel depending on the strength value of the predictor, e.g., the value of Q1 may be higher for samples with larger predictor values ​​than for samples with smaller predictor values. These quantization parameters may be defined using linear, i.e., scalar, formulas, or using more complex non-linear / non-uniform formulas.

[0115] In one embodiment, the quantizer may be represented by a predetermined function of the following form: q(i, j) = f k (p(i, j), i, j), where q represents the quantizer in question, i and j represent the position of the value within the pixel block being quantized (pixel domain value or transform domain value), p(i, j) represents the pixel intensity value being quantized, and f k represents the function that relates p, i, and j to q. As part of the signaling, the controller 260 may provide data identifying the function being used, as well as any other data that the corresponding function operates on (eg, any scalars and offsets that may operate at the corresponding function).

[0116] The quantization process may also be specified using a lookup table (not shown). The lookup table may involve a quantization / dequantization process that uses interpolation techniques to determine certain values ​​not included in the lookup table. Alternatively, the quantization parameters Q1-Q N The quantization parameters Q1-Q N A set of quantization parameters Q1-Q2 may be signaled in a higher level structure of the coding sequence (e.g., a frame or slice header) and signaled NThe replacement of may be signaled on a pixel block by pixel block basis when the constructed pixel blocks are encoded.

[0117] During operation, controller 860 may modify operating parameters of quantizers 813, 814, and 815 and transform unit 814 at different granularities of image data, either on a per-pixel block basis or on a larger granularity basis, e.g., per-frame, per-slice, per-LCU, or another region. In an embodiment, the quantization parameters may be modified on a per-pixel basis.

[0118] Additionally, as discussed, controller 860 may control the operation of loop filter 830 and predictor 850. For predictor 850, such control may include mode selection (λ, mode to be tested, search window, distortion strategy, etc.), and for loop filter 830, selection of filter parameters, reordering parameters, weighted prediction, etc.

[0119] Fig. 9 9 is a functional block diagram of a decoding system 900 according to an embodiment of the present disclosure. The decoding system 900 may include a syntax unit 910, a pixel block decoder 920, a loop filter 930, a prediction buffer 940, a predictor 950, and a controller 960. The syntax unit 910 may receive an encoded video data stream and may parse the encoded data into its components. Data representing encoding parameters may be provided to the controller 960, and data representing an encoded residual ( Figure 8 The pixel block encoder 810 output data) may be provided to the pixel block decoder 920. The pixel block decoder 920 may invert the encoding operation ( Figure 8 ). The loop filter 930 may filter the reconstructed pixel block data. The reconstructed pixel block data may be assembled into frames for display and output from the decoding system 900 as output video. The frames may also be stored in the prediction buffer 940 for prediction operations. The predictor 950 may provide prediction data to the pixel block decoder 920, the prediction data being determined by the coded data received in the coded video data stream.

[0120] The pixel block decoder 920 may include an entropy decoder 921, a coefficient dequantizer 922, an inverse transform unit 923, a plurality of dequantizers 924.1-924.N, and an adder 926. The entropy decoder 921 may perform entropy decoding to reverse the process performed by the entropy encoder 816 ( Figure 8 The coefficient dequantizer 922 may invert the operation of the coefficient quantizer 815 of the pixel block encoder 810 ( Figure 8 Similarly, the inverse transform unit 923 and the dequantizers 924.1-924.N may inverse the transform unit 814 and the quantizers 813.1-813.N, respectively ( Figure 8 ) operations. They can use the quantization parameters Q1-Q provided in the encoded video data streamN and transform mode M. Since the operation of quantizers 813.1-813.N and coefficient quantizer 815 truncates the data at various points, the data reconstructed by coefficient dequantizer 922 and dequantizers 924.1-924.N will likely have coding errors ( Figure 8 ).

[0121] Adder 926 may invert the operation performed by subtractor 812 ( Figure 8 ). It may receive a predicted pixel block from the predictor 950, which is determined by a prediction reference in the coded video data stream. The adder 926 may add the predicted pixel block to the reconstructed residual value input to it from the previous dequantizer 924.1, and may output the reconstructed pixel block data.

[0122] The loop filter 930 may perform various filtering operations on the reconstructed pixel block data. As shown, the loop filter 930 may include a deblocking filter 932 and an SAO filter 933. The deblocking filter 932 may filter data at the seams between the reconstructed pixel blocks to reduce the interruption between the pixel blocks due to encoding. The SAO filter may, for example, add an offset to the pixel value according to the SAO type based on the edge direction / shape and / or pixel level. The operation of the deblocking filter 932 and the SAO filter 933 will ideally simulate the operation of their counterparts in the encoding system 800 ( Figure 8 ). Therefore, in the absence of transmission errors or other anomalies, the decoded frame data obtained from the loop filter 930 of the decoding system 900 will be the same as the decoded frame data obtained from the loop filter 830 of the encoding system 800 ( Figure 8 ) are the same; thus, the encoding system 800 and the decoding system 900 should store a common set of reference pictures in their respective prediction buffers 840, 940.

[0123] The prediction buffer 940 may store filtered pixel data for subsequent prediction of other pixel blocks. The prediction buffer 940 may store decoded pixel block data for each frame when encoded for intra-frame prediction. The prediction buffer 940 may also store decoded reference frames.

[0124] As discussed, the predictor 950 may provide prediction data to the pixel block decoder 920. The predictor 950 may provide prediction pixel block data determined by prediction reference indicators provided in the encoded video data stream.

[0125] The controller 960 may control the overall operation of the decoding system 900. The controller 960 may set the operating parameters of the pixel block decoder 920 and the predictor 950 based on the parameters received in the encoded video data stream. Relevant to the present discussion, these operating parameters may include the quantization parameters Q1-Q2 for the dequantizers 924.1-924.N and the inverse transform unit 923. N and transform mode M. As discussed, the received parameters may be set at various granularities of the image data, for example, on a per-pixel basis, per-pixel block basis, per-frame basis, per-slice basis, per-LCU basis, or based on other types of regions of data defined for the input image.

[0126] Figure 8 and Fig. 9 The embodiment provides quantization of image data along several dimensions, such as Figure 2 and Figure 3 Implementation schemes with a greater number of dimensions. In a specific implementation, circuit designers may find it convenient to use as many types of quantizers 813.1-813.N and dequantizers 924.1-924.N as possible to design encoding and decoding systems suitable for their applications. In the absence of coordination between the distributors of the encoding system and the decoding system, the respective encoding system and the decoding system may have unmatched sets of quantizers 813.1-813.N and dequantizers 924.1-924.N. In this case, the encoding system and the decoding system may exchange data identifying the capabilities of their quantizers 813.1-813.N and dequantizers 924.1-924.N during the initialization phase of the video encoding session. Through negotiation, the encoding system and the decoding system can determine whether they share any quantizers 813.1-813.N and dequantizers 924.1-924.N in common. If so, the encoding system and the decoding system may disable those quantizers 813.1-813.N and dequantizers 924.1-924.N that are not shared together, set them to a pass state, and use those quantizers 813.1-813.N and dequantizers 924.1-924.N that are shared together for video encoding. Therefore, the encoding system and the decoding system can achieve the advantages inherent in the present disclosure even if their capabilities are different.

[0127] Fig.101 is a functional block diagram of an encoding system 1000 according to another embodiment of the present disclosure. System 1000 may include a pixel block encoder 1010, a pixel block decoder 1020, a loop filter system 1030, a prediction buffer 1040, a predictor 1050, a controller 1060 and a syntax unit 1070. The pixel block encoder and decoder 1010, 1020 and the predictor 1050 may iteratively operate on each pixel block. The predictor 1050 may predict the data used during the encoding of the newly presented pixel block. The pixel block encoder 1010 may encode new pixel blocks in several parallel encoding chains and may present the encoded pixel block data to the syntax unit 1070. The pixel block decoder 1020 may also decode the encoded pixel block data in the parallel decoding chain, thereby generating decoded pixel block data. The loop filter 1030 may perform various filtering operations on the decoded frame data assembled from the decoded pixel blocks obtained from the pixel block decoder 1020. The filtered frame data may be stored in a prediction buffer 1040, which may be used as a source for prediction of later received pixel blocks. Syntax unit 1070 may assemble a data stream from coded pixel block data that conforms to a governing coding protocol.

[0128] The pixel block encoder 1010 may include a subtractor 1012 and parallel encoding chains, each encoding chain including a corresponding quantizer 1013.1, 1013.2, ..., or 1013.N, a corresponding transform unit 1014.1, 1014.2, ..., or 1014.N, a corresponding coefficient quantizer 1015.1, 1015.2, ..., or 1015.N, and a corresponding entropy encoder 1016.1, 1016.2, ..., or 1016.N. The multiplexer 1017 may merge the outputs of the entropy encoders 1016.1, 1016.2, ..., or 1016.N into an integrated encoded video stream, which may be output from the pixel block encoder 1010 to the syntax unit 1070.

[0129] The pixel block encoder 1010 may accept a pixel block of input data at a subtractor 1012. The subtractor 1012 may receive a predicted pixel block from a predictor 1050 and generate therefrom a pixel residual array representing the pixel-by-pixel difference between the input pixel block and the predicted pixel block. The quantizers 1013.1-1013.N may each perform quantization on the data they receive in different domains of the image data. Here, the quantization dimensions need not be orthogonal to each other. The transform units 1014.1-1014.N may apply a transform to the data input to them from the corresponding quantizers 1013.1-1013.N to convert the input data from the pixel domain to the transform coefficient domain. The coefficient quantizers 1015.1-1015.N may quantize the transform coefficients input to them from their corresponding transform units 1014.1-1014.N. The entropy encoders 1016.1-1016.N may reduce the bandwidth of input data provided thereto from their respective coefficient quantizers 1015.1-1015.N, for example by encoding the output via variable length codewords.

[0130] The quantizers 1013.1-1013.N may each perform quantization on the data they receive in the corresponding domain of the image data. The quantizers 1013.1-1013.N may be provided as, for example, reducers, residual quantizers, color component specific quantizers, frequency domain quantizers, orientation specific quantizers (e.g., horizontal only, vertical only, mixed), brightness specific residual reducers, etc. The quantizers 1013.1-1013.N may operate according to corresponding quantization parameters supplied by the controller 1060 that define the quantization levels they provide. The quantization parameters may be explicitly or implicitly signaled to the decoding system in the encoded bitstream. In some cases, individual quantizers 1013.1, 1013.2, ..., 1013.N may be disabled, where they may simply pass their input data to the next quantizer without change. If two or more quantizers 1013.1, 1013.2, ..., 1013.N are disabled, some related encoding chains may be disabled altogether.

[0131] Transform units 1014.1-1014.N may operate in a variety of transform modes when events warrant. For example, transform unit 1014 may be selected to apply DCT, DST, Walsh Hadamard transform, Haar transform, Daubechies wavelet transform, etc. In addition, transform units 1014.1-1014.N may be disabled when transform SKIP mode encoding is applied in the corresponding encoding chain. In an embodiment, controller 1060 may select the encoding mode applied by transform units 1014.1-1014.N, which encoding mode may vary from chain to chain, and controller 1060 may configure transform units 1014.1-1014.N accordingly. The transform mode may be explicitly or implicitly signaled in the encoded video data.

[0132] The pixel block decoder 1020 may include a decoding unit that reverses the encoding operation of the pixel block encoder 1010. For example, the pixel block decoder 1020 may include coefficient dequantizers 1022.1-1022.N, inverse transform units 1023.1-1023.N, and dequantizers 1024.1-1024.N, arranged in a chain as in the pixel block encoder 1010. The pixel block decoder 1020 may also include an averager 1025 and an adder 1026.

[0133] The decoding chain of the pixel block decoder 1020 may obtain its input data from the output of the corresponding encoding chain of the pixel block encoder 1010. Again, although permitted, the pixel block decoder 1020 does not have to perform entropy decoding of the entropy encoded data because entropy encoding is a lossless event. The coefficient dequantizers 1022.1-1022.N may reverse the operation of the corresponding quantizers 1015.1-1015.N of the pixel block encoder 1010. Similarly, the inverse transform units 1023.1-1023.N and the dequantizers 1024.1-1024.N may reverse the operation of the transform units 1014.1-1014.N and the quantizers 1013.1-1013.N, respectively. They may use the same quantization parameters and transform modes as their counterparts in the pixel block encoder 1010. The operation of quantizers 1013.1-1013.N and coefficient quantizer 1015 may truncate data in various aspects, and therefore, the residual data reconstructed by coefficient dequantizers 1022.1-1022.N and dequantizers 1024.1-1024.N may have encoding errors when compared to the data presented to the corresponding parts in the pixel block encoder 1010.

[0134] Averager 1025 may average the outputs of the various decoding chains. The contributions of the different encoding chains may be given equal weight, or alternatively, may be weighted based on weights specified by controller 1060. In an embodiment, controller 1060 may measure the distortion of the reconstructed data from the different chains, and may select an averaging weight that reduces such distortion when averaged. The output of averager 1025 may be input to adder 1026 as reconstructed residual data.

[0135] The adder 1026 may reverse the operation performed by the subtractor 1012. It may receive the same predicted pixel block from the predictor 1050 that the subtractor 1012 used to generate the residual signal. The adder 1026 may add the predicted pixel block to the reconstructed residual value output by the averager 1025 and may output the reconstructed pixel block data.

[0136] The loop filter 1030 may perform various filtering operations on the restored pixel block data. For example, the loop filter 1030 may include a deblocking filter 1032 and an SAO filter 1033. The deblocking filter 1032 may filter data at the seams between the reconstructed pixel blocks to reduce the interruption between the pixel blocks due to encoding. The SAO filter 1033 may, for example, add an offset to the pixel value according to the SAO type based on the edge direction / shape and / or pixel level. The loop filter 1030 may operate according to the parameters selected by the controller 1060.

[0137] Prediction buffer 1040 can store filtered pixel data for subsequent prediction of other pixel blocks. For different prediction modes, predictor 1050 can use different types of prediction data. For example, for an input pixel block, intra-frame prediction obtains a prediction reference from decoded data of the same frame in which the input pixel block is located. Therefore, prediction buffer 1040 can store decoded frame block data for each frame during encoding. For the same input pixel block, inter-frame prediction can use a prediction reference from a previously encoded and decoded frame, which is designated as a reference frame. Therefore, prediction buffer 1040 can store these decoded reference frames.

[0138] As discussed, the predictor 1050 may provide prediction data to the pixel block encoder 1010 for generating a residual. The predictor 1050 may include a motion estimator 1052, an intra predictor 1053, and a mode decision unit 1054. The motion estimator 1052 may receive pixel block data representing a new pixel block to be encoded, and may search the prediction buffer 1040 for pixel block data from a reference frame for encoding the input pixel block. The motion estimator 1052 may support multiple prediction modes, such as P and B modes, supplying prediction data from one or a pair of reference frames. The motion estimator 1052 may select an inter-prediction mode and provide prediction data that provides the closest match to the input pixel block being encoded. The motion estimator 1052 may generate a prediction reference indicator, such as a motion vector, to identify which portion of the reference frame is selected as a prediction source for the input pixel block.

[0139] The intra-frame predictor 1053 may support intra-frame coding. The intra-frame predictor 1053 may search for coded pixel block data from the same frame as the coded pixel block that provides the closest match to the input pixel block. The intra-frame predictor 1053 may also generate a prediction reference indicator to identify which portion of the frame is selected as the prediction source for the input pixel block.

[0140] The mode decision unit 1054 may select a final encoding mode to be applied to the input pixel block. Typically, when decoding video, the mode decision unit 1054 selects a prediction mode that will achieve the lowest distortion. Exceptions may occur when the encoding mode is selected to meet other strategies followed by the encoding system 1000, such as supporting random access or data refresh strategies. The mode decision unit 1054 may output prediction data to the pixel block encoder and decoder 1010, 1020, and may provide an identification of the selected prediction mode and a prediction reference indicator corresponding to the selected mode to the controller 1060.

[0141] Controller 1060 may control the overall operation of encoding system 1000. Controller 1060 may select operating parameters for pixel block encoder 1010 and predictor 1050 based on analysis of input pixel blocks and external constraints such as encoding bit rate targets and other operating parameters. Relevant to the present discussion, when it selects quantization parameters used by quantizers 1013.1-1013.N and transform modes, it may provide those parameters to syntax unit 1070, which may include data representing those parameters in a data stream of encoded video data output by system 1000. Controller 1060 may also determine that one or more quantizer units (e.g., quantizer 1013.2) should not apply quantization, in which case controller 1060 may disable quantizer 1013.2. As described above, controller 1060 may also determine the operating mode of transform unit 1014.

[0142] As discussed, the quantization parameter need not be provided as a single value that is applied uniformly to all pixels of a pixel block, but may be provided as an array of corresponding quantization parameters whose values ​​vary at each pixel position. In one embodiment, a plurality of quantizer arrays may be defined according to a predetermined protocol, each protocol having corresponding quantization values ​​at different pixel positions; during operation, the controller 1060 may identify the array to be used by providing an index or other identifier.

[0143] In another embodiment, the quantization parameter (whether a single value or an array of values) may be derived from characteristics of the image data. For example, the quantization parameter may be selected based on the strength of the predicted pixel value corresponding to the pixel residual. In this case, the value of the quantization parameter may vary for each pixel according to the strength value of the predictor, e.g., the value of the quantization parameter may be higher for samples with larger predictor values ​​than for samples with smaller predictor values. These quantization parameters may be defined using linear, i.e., scalar, formulas, or using more complex non-linear / non-uniform formulas.

[0144] In one embodiment, the quantizer may be represented by a predetermined function of the following form: q(i, j) = f k(p(i, j), i, j), where q represents the quantizer in question, i and j represent the position of the value within the pixel block being quantized (pixel domain value or transform domain value), p(i, j) represents the pixel intensity value being quantized, and f k represents the function that relates p, i, and j to q. As part of the signaling, the controller 260 may provide data identifying the function being used, as well as any other data that the corresponding function operates on (eg, any scalars and offsets that may operate at the corresponding function).

[0145] The quantization process may also be specified using a lookup table form (not shown). The lookup table form may involve a quantization / dequantization process that uses interpolation techniques to determine certain values ​​not contained in the lookup table. Alternatively, the quantization parameters may be predicted based on the quantization parameters of previously encoded neighboring pixel blocks. In another embodiment, the first set of quantization parameters may be signaled in a higher level structure of the coding sequence (e.g., a frame or slice header) and an alternative to the signaled set of quantization parameters may be signaled on a pixel block by pixel block basis as pixel blocks of that structure are encoded.

[0146] During operation, controller 1060 may modify operating parameters of quantizers 1013, 1014, and 1015 and transform unit 1014 at different granularities of image data, either on a per-pixel-block basis or on a larger granularity, e.g., per-frame, per-slice, per-LCU, or another region. In an embodiment, the quantization parameters may be modified on a per-pixel basis.

[0147] Additionally, as discussed, controller 1060 may control the operation of loop filter 1030 and predictor 1050. For predictor 1050, such control may include mode selection (λ, mode to be tested, search window, distortion strategy, etc.), and for loop filter 1030, selection of filter parameters, reordering parameters, weighted prediction, etc.

[0148] Fig.11 1 is a functional block diagram of a decoding system 1100 according to an embodiment of the present disclosure. The decoding system 1100 may include a syntax unit 1110, a pixel block decoder 1120, a loop filter 1130, a prediction buffer 1140, a predictor 1150, and a controller 1160. The syntax unit 1110 may receive an encoded video data stream and may parse the encoded data into its components. Data representing encoding parameters may be provided to the controller 1160, and data representing an encoded residual ( Fig.10 The pixel block encoder 1010 output data) may be provided to the pixel block decoder 1120. The pixel block decoder 1120 may invert the encoding operation ( Fig.10). The loop filter 1130 may filter the reconstructed pixel block data. The reconstructed pixel block data may be assembled into frames for display and output from the decoding system 1100 as output video. The frames may also be stored in the prediction buffer 1140 for prediction operations. The predictor 1150 may provide prediction data to the pixel block decoder 1120, the prediction data being determined by the encoded data received in the encoded video data stream.

[0149] The pixel block decoder 1120 may include an entropy decoder 1121, a demultiplexer 1122, a plurality of coefficient dequantizers 1123.1-1123.N, a plurality of inverse transform units 1124.1-1124.N, a plurality of dequantizers 1125.1-1125.N, an averager 1126, and an adder 1127. The entropy decoder 1121 may perform entropy decoding to reverse the process performed by the entropy encoder 1016 ( Fig.10 The demultiplexer 1122 can transmit the encoded video data from the pixel block encoder 1010 ( Fig.10 ) is routed to a corresponding decoding chain in the pixel block decoder 1120. Each coefficient dequantizer 1123.1, 1123.2, ..., 1123.N may invert the operation of a corresponding coefficient quantizer 1015.1, 1015.2, ..., 1015.N of the pixel block encoder 1010 ( Fig.10 ). Similarly, the inverse transform units 1124.1, 1124.2, ..., 1124.N and the dequantizers 1125.1, 1125.2, ..., 1125.N may respectively invert the operations of their corresponding units from the pixel block encoder 1010 ( Fig.10 ). They may use quantization parameters provided in the encoded video data stream. Because the operation of the quantizers 1013.1-1013.N and the coefficient quantizers 1015.1-1015.N truncates the data at various points, the residual data reconstructed by the coefficient dequantizers 1123.1, 1123.2, ..., 1123.N and the dequantizers 1125.1-1125.N may have encoding errors ( Fig.10 ).

[0150] The outputs of the various decoding chains may be averaged by an averager 1126. The contributions of the different encoding chains may be given equal weight, or alternatively may be weighted based on weights specified by a controller 1160, the weights being derived from the encoded video data. The output of the averager 1126 may be input to an adder 1127 as reconstructed residual data.

[0151] Adder 1127 can reverse the operation performed by subtractor 1012 ( Fig.10). It may receive a predicted pixel block from predictor 1150, the predicted pixel block being determined by a prediction reference in the coded video data stream. Adder 1127 may add the predicted pixel block to the reconstructed residual value input to it from averager 1126.

[0152] The loop filter 1130 may perform various filtering operations on the reconstructed pixel block data. As shown, the loop filter 1130 may include a deblocking filter 1132 and an SAO filter 1133. The deblocking filter 1132 may filter data at the seams between the reconstructed pixel blocks to reduce the interruption between the pixel blocks due to encoding. The SAO filter may, for example, add an offset to the pixel value according to the SAO type based on the edge direction / shape and / or pixel level. The operation of the deblocking filter 1132 and the SAO filter 1133 will ideally simulate the operation of their counterparts in the encoding system 1000 ( Fig.10 ). Therefore, in the absence of transmission errors or other anomalies, the decoded frame data obtained from the loop filter 1130 of the decoding system 1100 will be identical to the decoded frame data obtained from the loop filter 1030 of the encoding system 1000 ( Fig.10 ) are the same; thus, the encoding system 1000 and the decoding system 1100 should store a common set of reference pictures in their respective prediction buffers 1040, 1140.

[0153] The prediction buffer 1140 may store filtered pixel data for subsequent prediction of other pixel blocks. The prediction buffer 1140 may store decoded pixel block data for each frame when encoded for intra prediction. The prediction buffer 1140 may also store decoded reference frames.

[0154] As discussed, the predictor 1150 may provide prediction data to the pixel block decoder 1120. The predictor 1150 may provide prediction pixel block data determined by prediction reference indicators provided in the encoded video data stream.

[0155] The controller 1160 may control the overall operation of the decoding system 1100. The controller 1160 may set operating parameters for the pixel block decoder 1120 and the predictor 1150 based on parameters received in the encoded video data stream. Relevant to the present discussion, these operating parameters may include quantization parameters and operating modes. As discussed, the received parameters may be set at various granularities of the image data, for example, on a per-pixel block basis, per-frame, per-slice basis, per-LCU basis, or based on other regions of data defined for the input image.

[0156] Fig.10 and Fig.11 The embodiment provides quantization of image data along several dimensions, such as Figure 2 and Figure 3Implementation of a larger number of dimensions. Figure 8 and Fig. 9 Even if the encoding system and the decoding system do not have the same quantizer capabilities, Fig.10 and Fig.11 The encoding system and decoding system can also achieve the advantages inherent in the present discussion.

[0157] in addition, Fig.10 The implementation of provides the coding system with the opportunity to test certain quantization techniques during encoding to evaluate the encoding quality and bandwidth performance. Fig.10 The encoding system 1000 may determine that the result of a given encoding chain is poor. In response to such a determination, the encoding system may set the weight W of the corresponding encoding chain to 0 and omit the encoded data of the chain from the encoded video data stream.

[0158] As discussed, the operation of the quantizer and dequantizer of the foregoing embodiments may be reconfigured for each pixel block that is encoded and then decoded.Thus, a single image may undergo a wide array of quantization operations. Fig.12 and Fig.13 An exemplary selection of quantization of frames is shown.

[0159] Also as discussed, the selection of the quantization mode for a given frame may be performed at different granularities. For example, it may be convenient to reselect the quantization mode for each pixel block being encoded. However, in other embodiments, it may be convenient to select the quantization mode on a per-slice, per-LCU, or per-frame basis.

[0160] The foregoing discussion describes various embodiments of the present disclosure in the context of encoding systems, decoding systems, and functional units that can implement them. In practice, these systems can be applied to a variety of devices, such as mobile devices with integrated video cameras (e.g., phones with camera capabilities, entertainment systems, and computers) and / or wired communication systems such as video conferencing equipment and camera-enabled desktop computers. In some applications, the functional blocks described above can be provided as elements of an integrated software system, wherein the blocks can be provided as elements of a computer program, which are stored in a memory as program instructions and executed by a general-purpose processing system. In other applications, the functional blocks can be provided as discrete circuit components of a processing system, such as functional units within a digital signal processor or an application-specific integrated circuit. Other applications of the present disclosure can be embodied as a hybrid system of dedicated hardware and software components. In addition, the functional blocks described herein do not need to be provided as separate components. For example, although Figure 1-Figure 11The components of the video encoder and decoder are shown as separate units, in one or more embodiments, some or all of them may be integrated, and they need not be separate units. Unless otherwise noted, these implementation details are not important to the operation of the present disclosure.

[0161] In addition, the drawings shown herein only provide necessary details to present the subject matter of the present disclosure. In implementation, the video encoder and decoder will generally include functional units other than those described herein, including buffers for storing data in the illustrated encoding pipeline and communication transceivers for managing communications with a communication network and corresponding encoder / decoder devices. For clarity, the above discussion omits such elements.

[0162] Several embodiments of the present disclosure are specifically shown and / or described herein. However, it should be understood that modifications and variations of the present disclosure are covered by the above teachings and are within the scope of the appended claims without departing from the spirit and intended scope of the present disclosure.

Claims

1. A video decoding method, comprising: decoding reference picture data of an encoded video sequence of frames received from a channel; predicting a prediction data block from decoded reference picture data; entropy decoding the data of the encoded video sequence to produce a block of residual transform coefficients and corresponding quantization parameters including an indication of residual downsampling; transforming residual transform coefficients in the transform domain to blocks of reconstructed residual data in the pixel domain; In a case where the quantization parameter indicating residual downsampling is applied only to the chroma component of the first block, the chroma component of the corresponding reconstructed residual data is amplified, and a block of reconstructed pixel values ​​is generated based on the corresponding prediction data block, the corresponding block of the chroma component of the amplified reconstructed residual data, and the corresponding block of the reconstructed residual data of the other color components.

2. The video decoding method according to claim 1, further comprising: generating a block of reconstructed pixel values ​​from a corresponding block of prediction data and a corresponding block of reconstructed residual data in the pixel domain, where a quantization parameter indicating residual downsampling is not applied to the first block; as well as In a case where a quantization parameter indicating residual downsampling is applied to all color components of the first block, all color components of the reconstructed residual data in the pixel domain are upscaled, and a block of reconstructed pixel values ​​is generated based on the corresponding block of prediction data and the corresponding block of the upscaled reconstructed residual data.

3. The video decoding method according to claim 1, further comprising: Dequantizing the reconstructed residual data using a precision dequantizer based on the quantization parameter to generate precision dequantized residual data; The generation of the blocks of enlarged and reconstructed pixel values ​​is based on precision dequantized residual data.

4. The video decoding method according to claim 3, wherein: The precision dequantizer uniformly applies a linear quantizer to each pixel-domain residual value in the color components of the block.

5. The video decoding method according to claim 3, wherein: The precision dequantizer applies a non-linear quantizer to each pixel-domain residual value in the block.

6. The video decoding method according to claim 3, wherein: The quantization parameter corresponding to the block is predicted based on the intensity values ​​in the corresponding block of prediction data.

7. A video decoder comprising: a predictor that predicts a prediction data block from decoded reference picture data; an entropy decoder having an input for coded video data, and an output of a block of residual transform coefficients and corresponding quantization parameters comprising an indication of residual downsampling; a transform unit having an input connected to the output of the entropy decoder, for transforming a block of residual transform coefficients into a block of reconstructed residual data; an amplifier for spatially upsampling color components of the reconstructed residual data; as well as an adder for adding the color components of the prediction data block to the corresponding color components of the reconstructed residual data output from the transform unit or the corresponding color components of the amplified residual data output from the amplifier based on the corresponding indication of the residual downsampling, In which, when the quantization parameter indicating residual downsampling is applied only to the chrominance component of the first block, the adder generates a block of reconstructed pixel values ​​based on the corresponding prediction data block, the corresponding block of the chrominance component of the amplified reconstructed residual data, and the corresponding block of the reconstructed residual data of other color components.

8. The video decoder according to claim 7, further comprising: The adder generates a block of reconstructed pixel values ​​from a corresponding block of prediction data and a corresponding block of reconstructed residual data in the pixel domain in a case where a quantization parameter indicating residual downsampling is not applied to the first block; as well as The adder generates a block of reconstructed pixel values ​​from a corresponding block of prediction data and a corresponding block of upscaled reconstructed residual data in a case where a quantization parameter indicating residual downsampling is applied to all color components of the first block.

9. The video decoder of claim 7, further comprising: a precision dequantizer, the precision dequantizer dequantizing the reconstructed residual data based on a quantization parameter to generate precision dequantized residual data; The generation of the blocks of enlarged and reconstructed pixel values ​​is based on precision dequantized residual data.

10. The video decoder according to claim 9, wherein: The precision dequantizer uniformly applies a linear quantizer to each pixel-domain residual value in the color components of the block.

11. The video decoder according to claim 9, wherein: The precision dequantizer applies a non-linear quantizer to each pixel-domain residual value in the block.

12. The video decoder according to claim 9, wherein: The quantization parameter corresponding to the block is predicted based on the intensity values ​​in the corresponding block of prediction data.

13. A computer-readable memory storing instructions, which, when executed on a processor, cause the video decoding method according to any one of claims 1 to 6 to be performed.

14. A video decoding method, comprising: predicting a prediction data block from decoded reference picture data; entropy decoding the encoded video data of the encoded video sequence to produce blocks of transform coefficients and a corresponding quantization parameter for each block; transforming transform coefficients in the transform domain to blocks of reconstructed residual data in the pixel domain; first dequantizing the blocks of reconstructed residual data into blocks of first dequantized residual data by inverting a first quantization type based on a corresponding first quantization parameter for each block, wherein the first quantization parameter indicates which blocks of the frame the first quantization applies to and which other blocks of the same frame the first quantization does not apply to; second dequantizing the blocks of first dequantized residual data into blocks of second dequantized residual data by inverting a second quantization type based on a corresponding second quantization parameter for each block, wherein the second quantization parameter indicates which blocks of the frame the second quantization applies to and which other blocks of the same frame the second quantization does not apply to; as well as Reconstructed pixel values ​​are generated based on the second block of dequantized residual data and based on the prediction data.

15. The video decoding method according to claim 14, wherein: The first quantization type is a spatial scaler, and the spatial scaler is applied to the chroma components of the residual but not to the luma components of the residual.

16. The video decoding method of claim 14, wherein the first quantization type is a spatial scaler, and the spatial scaler is applied only to a subset of blocks of residual data within a frame and not to other blocks of residual data within the frame.

17. The video decoding method according to claim 14, wherein: The first quantization type is a spatial scaler, The spatial sealer is applied only to chroma components of the residual data within a first subset of blocks within a frame and not to luma components of the residual data, and The spatial scaler is applied to both chroma components and luma components of the residual data within a second subset of blocks within the frame.

18. The video decoding method of claim 14, wherein the first quantization type is a precision quantizer, and the precision quantizer uniformly applies a linear quantizer to each pixel domain residual value within a color component of a block.

19. The video decoding method of claim 14, wherein the first quantization type is a precision quantizer, and the precision quantizer applies a non-linear quantizer to each pixel domain residual value within a block.

20. The video decoding method of claim 14, wherein the first quantization type is a precision quantizer, and a quantization parameter corresponding to the block is predicted based on an intensity value in a corresponding block of prediction data.

21. A computer-readable memory storing instructions which, when executed on a processor, cause the video decoding method according to any one of claims 14-20 to be performed.

22. A decoding device comprising a processor and a memory storing computer program instructions, wherein the computer program instructions, when executed by the processor, cause the video decoding method according to any one of claims 14 to 20 to be performed.

23. A video decoding method, comprising: decoding reference picture data of an encoded video sequence of frames received from a channel; predicting a prediction data block from decoded reference picture data; entropy decoding the data of the encoded video sequence to produce a block of residual transform coefficients and a corresponding residual resolution quantization parameter including an indication of residual downsampling; transforming residual transform coefficients in the transform domain to blocks of reconstructed residual data in the pixel domain; performing a first dequantization on the blocks of reconstructed residual data using a resolution scaler according to a corresponding residual resolution quantization parameter, wherein the residual resolution quantization parameter varies within a frame of a video sequence of frames; as well as Where the corresponding residual resolution quantization parameter indicating residual downsampling is applied to all components of the first block, all components of the corresponding reconstructed residual data are upscaled and a block of reconstructed pixel values ​​is generated from the corresponding block of prediction data.

24. The video decoding method according to claim 23, wherein: The resolution scaler is applied to the chroma components of the residual but not to the luma component of the residual.

25. The video decoding method according to claim 23, wherein: The resolution scaler is applied only to a subset of blocks of residual data within a frame and not to other blocks of residual data within the frame.

26. The video decoding method according to claim 23, wherein: The resolution scaler is applied only to chroma components of the residual data within a first subset of blocks within a frame and not to luma components of the residual data, and The resolution scaler is applied to both chroma components and luma components of the residual data within a second subset of blocks within the frame.

27. The video decoding method of claim 23, further comprising applying a precision dequantizer to the block of residual transform coefficients prior to the first dequantization, wherein the precision dequantizer is linear with respect to coefficient values.

28. The video decoding method of claim 23, further comprising applying a precision dequantizer to the block of residual transform coefficients prior to the first dequantization, wherein the precision dequantizer is non-linear with respect to coefficient values.

29. A computer-readable memory storing instructions which, when executed on a processor, cause the video decoding method according to any one of claims 23-28 to be performed.

30. A decoding device comprising a processor and a memory storing computer program instructions, which, when executed by the processor, cause the video decoding method according to any one of claims 23-28 to be performed.

Images