Method for coding videos using dictionaries

Abstract

A method for decoding a video encoded as a bitstream in a video decoder, comprising the steps of: Maintaining (570) a set of dictionaries generated from decoded prediction residual signals, and wherein elements of the dictionaries of the set of dictionaries have associated indices; Entropy decoding (510) and inverse quantizing (520) a current macroblock of the bitstream to generate decoded coefficients; selecting (560), for the current macroblock, a particular dictionary of the set of dictionaries according to a prediction mode (504) signaled in the bitstream, and selecting (550) particular elements of the particular dictionary according to a copy mode signal (503) in the bitstream and the associated index; scaling and combining (530), using the decoded coefficients, the determined elements (551) to reconstruct a current decoded macroblock prediction residual signal; and Combining (Σ) the current decoded macroblock prediction residual signal with previously decoded macroblocks to generate an output macroblock of reconstructed video, where the steps are performed in a decoder.

Description

[Technical field]

[0001] This invention relates generally to video coding and decoding, and more particularly to a method for coding a video using dictionaries. [State of the art]

[0002] A video coding standards project, called "High Efficiency Video Coding" (HEVC), has been initiated by a "Joint Collaborative Team on Video Coding" (JCT-VC). One goal of this standard is to improve coding capabilities over the older H.264 / AVC standard for expanded applications and larger bit ranges. The initial basic structure of the HEVC video coding standard is not significantly different from the previous video coding standard, including a block-based prediction technique, a 2D discrete cosine transform (DCT), and context-based entropy coding. The new coding tools are more advanced and flexible, but with increased computational complexity. As with conventional coding techniques, the encoder and decoder operate on a sequence of video frames. The video frames are divided into macroblocks of pixels.The macroblocks can be spatially adjacent within a frame (for intra-mode coding) and temporally adjacent within consecutive frames (for inter-mode coding).

[0003] Orthogonal and biorthogonal complete dictionaries such as DCT or wavelet have been used for dominant transform domain representation in image and video coding. A sparse and redundant representation of signals using an overcomplete dictionary has been successfully used for various applications such as image denoising.

[0004] An overcomplete video coding technique can achieve competitive coding gains at very low bit rates compared to conventional video coding standards. Essentially, the block-based 2D DCT transform is replaced by an extension of larger and more appropriate basis functions in overcomplete video coding. In low-bit-rate video coding, residual signals are represented with fewer non-zero DCT coefficients due to a larger quantization parameter (QP), and thus only low-frequency components appear in a macroblock. In this scenario, the set of overcomplete dictionaries can provide a more diverse and accurate representation of residual signals than the full set of dictionaries. Thus, the residual signal can be better approximated with fewer coefficients.

[0005] Conventional overcomplete video coding forms a set of dictionaries with modulated Gabor functions. Adaptation pursuits (MP) are used to select the most appropriate dictionary elements in the representation. MP determine a suboptimal solution for the sparse signal representation. The set of dictionaries can be varied by concatenating dictionaries generated by multiple analytic functions such as wavelets, curvelets, and discrete Fourier transforms (DFT). Curvelets are an extension of the wavelet concept. Curvelets use a nonadaptive technique for multiscale object representation. Wavelets generalize the Fourier transform by using a basis that represents both location and spatial frequency. For 2D or 3D signals, directed wavelet transforms use basis functions that are also localized in orientation.A curvelet transform differs from other directed wavelet transforms in that the degree of localization in orientation varies with scale.

[0006] WO 2005 / 015501 A1 describes a video coding method applied to an input sequence of frames in which each frame is divided into blocks of arbitrary size. This method comprises, for at least a portion of the blocks of the current frame, the steps of generating motion-compensated frames on a block basis, each obtained from each current original frame and a previous reconstructed frame; computation of frame residual signals compensated from the motion sequences using a so-called matching pursuit (MP) algorithm to decompose each of the generated residual signals into coded dictionary functions called atoms; processing the other blocks of the current frame using other coding techniques; encoding these atoms and the motion vectors determined during the motion compensation step to generate an output coded bitstream.According to the invention, the method is such that when using the MP algorithm on the coding side, a specific dictionary is available for each block shape.

[0007] WO 2010 / 087807 A1 discloses apparatus for transform selection in video coding, comprising a video encoder for coding at least one block in an image by selecting a transform to be applied to a remainder of the block from a set of two or more available transforms. The transform is selected based on an interprediction mode selected for predicting at least one reference for the block, one or more values ​​corresponding to a motion vector, a value of a remainder of one or more previously encoded blocks, a value of prediction data for the block, one or more transform selections from one or more neighboring reconstructed blocks, and a quantization step used to transform coefficients for the remainder of the block.

[0008] However, despite their simplicity, such models have drawbacks. Natural images or videos often contain features that are not well represented by these models. In these cases, poor reconstruction or image errors such as overshoot can be introduced into the decoded image or video.

[0009] Dictionary training can be used because residual signals tend to have a directional orientation after prediction. Therefore, a set of dictionaries can be well-designed by reflecting the characteristics of the residual signals. Mode-dependent direction transformation can be used for intra-coding. A complete dictionary can be formed using intra-prediction residuals according to the directional prediction. Dictionary training can also be adapted to intra-prediction in image coding applications. [Summary of the invention]

[0010] Embodiments of the invention provide a method for encoding and decoding a video using a set of dictionaries. The set of dictionaries is trained using samples, where the samples are inter- or intra-predicted residual signals. The set of dictionary elements can also be used for adjacent macroblocks. An optimal number of dictionary elements is determined using rate-distortion optimization. Effect of the invention

[0011] The invention provides a dictionary-based video coding technique based on context-adaptive, overcomplete dictionaries. We train a set of dictionaries using inter- and / or intra-predicted residual samples and apply the dictionaries to a sparse signal representation. Dictionary elements are available for reuse to encode subsequent samples. [Brief description of the drawings] Fig. 1 is a block diagram of a prior art encoder; Fig. 2 is a block diagram of an encoder according to embodiments of the invention; and Fig. 3 is a schematic of a process for forming overcomplete dictionaries according to embodiments of the invention; and Fig. 4 is a block diagram of a prior art decoder; Fig. 5 is a block diagram of a decoder according to embodiments of the invention; and Fig. 6 is a block diagram of a copy mode evaluation device according to the embodiments of the invention. [Description of the embodiments]

[0012] A signal y∈ℜ n can be defined as a sparse linear combination of m elements in overcomplete dictionaries D∈ℜ nXmwhere m is much larger than n and D is a full-rank matrix.

[0013] We define dictionaries as overcomplete if the number of non-zero elements is greater than the number of non-zero elements in a basis. Overcompleteness achieves a more stable and compact representation.

[0014] We approximate the signal y with a smallest number of dictionary elements. The sparse representation is determined according to minX‖x‖0s.t.‖y−Dx‖2≤δ, where x∈ℜ m is a vector of coefficients representing the signal y, |.| is an I p -norm of the signal, and δ is a small, predetermined threshold.

[0015] The dictionaries are trained to better fit the sample data. Thus, the minimization problem can be transformed into equation (1) to find the best dictionaries for a given sparsity constraint C for the representation of the signal y as minX,D‖y-Dx‖2s.t.‖x‖0≤C, to obtain.

[0016] The dictionaries are trained to provide a better representation of the actual signal when the number of non-zero coefficients is less than or equal to C.

[0017] As in Fig. As shown in Figure 3, unlike conventional image coding, we perform either intra- or inter-prediction and then encode residual data samples 451 of the video signal. A classifier 210 generates context I211 and context K212 for the corresponding training 221-222 of contexts I and K. As shown in Fig. 3, the dictionary 231 of context K contains more diagonal elements than the more random elements of the dictionary 232 for context I. The diagonal elements are useful for edge representations in inter-coding.

[0018] Thus, in one embodiment of the invention, the dictionaries are trained using the prediction residual signals 451 before encoding a current video sequence. Such training is typically performed offline using a separate set of training videos. Both the encoder and decoder have the same set of dictionaries, which are typically dedicated to encoding the video sequence.

[0019] In another embodiment of the invention, the dictionaries are trained using the prediction residual signals 451 during the encoding and decoding of a current video sequence. In this embodiment, the data used for training includes the prediction residual signals associated with previously decoded samples of the current video available at both the encoder and decoder. Both the encoder and decoder maintain the same set of dictionaries, but the dictionaries are dynamic and are updated on a periodic basis during the encoding and decoding process of the video sequence.

[0020] Dictionary training can neglect local changes in residual signal characteristics from a relatively small set of dictionary items generated from a very large set of training data. Thus, some salient features of residual samples can be deleted from the dictionary set. To solve this problem, we apply the contexts 211-212 of the residual samples when training the dictionaries 221-222.

[0021] The classifier 210 can vary depending on the type of residual sample. For example, with an intra-coded residual, we rely on the assumption that the residual samples have a certain characteristic with the directed intra-prediction in an encoder / decoder (codec) according to the H.264 / AVC standard. Therefore, the multiple sets of dictionaries are each formed from the samples 451 according to the prediction directions.

[0022] After the intra-prediction mode is determined for a given macroblock, the classifier 210 is used to indicate which context 211-212 to use, and then the corresponding dictionary is applied during encoding. Thus, no additional side information is required in the decoder. In the inter-coded residual signal, the classifier checks the energies of the samples and passes the energies to the corresponding training modules to generate our context-adaptive dictionaries. Thus, the contexts are determined by data already available in the decoder. Elements of dictionary-based video coding

[0023] After the sets of dictionaries are trained on universal data samples, the dictionaries are used for a sparse representation of the input residual signal during encoding.

[0024] Fig. Figure 1 shows a conventional encoder. A macroblock 301 is input, and a bitstream 309 is output. The macroblock is transformed 310 and quantized 320 into a quantized signal. The quantized signal is entropy coded 330 to generate the bitstream 309. The output signal of the entropy encoder is also inversely quantized 340, inversely transformed 350, and used for intra-prediction 360 when combined with the input macroblock 301. Conventional decoder

[0025] Fig. Figure 4 shows a conventional decoder. The input signal is a bitstream 1451, and the output signal is a macroblock 1457. The bitstream is entropy decoded 1452 and inversely quantized 1453. The decoded transform coefficients are inversely transformed 1454. An intra- or inter-prediction residual 1455 is combined 1456 to generate a decoded macroblock. The macroblock is output and stored in a buffer 2456 for use in reconstructing future decoded macroblocks. Dictionary-based encoder

[0026] Fig. Figure 2 shows the encoder according to embodiments of the invention. The encoder steps can be performed in a processor connected to a memory and input / output interfaces, as is known in the art.

[0027] A current macroblock 301 is input to the encoder. Dictionaries for a particular segment of the video are trained 401 using the prediction residual signals 451 before encoding the particular segment of the video. The best matching basis among the dictionary elements (atoms) is selected 402 prior to quantization 320. A process such as matching tracking (MP) may be used for element selection 402.

[0028] A copy mode evaluator 450 receives as inputs dictionary indices used for previous blocks 452, prediction residual signal 451, a dictionary pointer, dictionary element indices, and a distortion of the current block when encoded using its own dictionary elements, i.e., elements selected without using the copy mode evaluator. The evaluator generates a copy mode signal 453 that depends on the output of an optimized rate distortion decision process that evaluates whether using a set of previously computed dictionary indices is better than using newly computed dictionary indices. The copy mode evaluator (CME) is Fig. 6 shown.

[0029] According to Fig. 6, inputs to the CME are the dictionary indices used for previously encoded blocks 452 and the prediction residual for the current block 301. A particular dictionary (AD) is used for the current block to determine scaling coefficients 402, and then reconstructed decoded prediction residuals 451 are input to the RD cost optimization process 600 to generate the copy mode signal 453.

[0030] Continuing with Fig. 2, the selected dictionary elements are inversely quantized 340 to generate an inversely quantized signal, which is entropy encoded 330 to generate the bitstream 309. The video is reconstructed 403 from an inversely quantized 340 signal. The reconstructed video is then used for intra-prediction 360.

[0031] MP is known to be a suboptimal and compliant solution in sparse signal representation. In a preferred embodiment of this invention, we use MP to select appropriate elements in the dictionaries 402. During each iteration, MP forms a linear combination of dictionary elements by minimizing the residual signal of the reconstructed video. Thus, it is required to contain the coefficient values ​​and a set of one or more indices indicating which dictionary elements are used to reconstruct the signal in the decoder. The iterations do not exceed the sparsity constraint C in Equation 2 and terminate when the rate distortion (RD) cost is minimized such that the signals can be optimally represented with a number of non-zero coefficients less than or equal to the sparsity constraint.

[0032] After the dictionary elements are selected 402, the coefficients are passed to quantization 320 and entropy coding 330. A Laplace distribution can be adapted to represent the coefficient distributions of the set of dictionaries used by our method. Thus, a uniform quantizer is efficiently adapted to the method.

[0033] In the entropy encoder 330, the scaling coefficient values ​​are binarized using a Huffman table based on statistics of the scaling coefficients. The occurrence of dictionary indices is largely uniform across the indices, so any adaptive or fixed sampling order cannot efficiently collect the non-zero coefficients. Thus, the indices are encoded with fixed-length codes whose size log2⌈m⌉ where m is the number of dictionary elements.

[0034] During reconstruction 403, which is also performed by the decoder, the quantized scaling coefficients are used for the weights in the linear combination of dictionary elements. The set of dictionary elements used is represented by a set of indices.

[0035] In addition to the processing described above, the encoder also includes the copy mode evaluator (CME) 450. The inputs to the CME are the current prediction residual signals to be encoded and the set of dictionary pointers and element indices used to encode previously encoded blocks. The dictionary pointer indicates which dictionary was used to encode a previously encoded block, and the element indices indicate which elements within that dictionary were used to encode a previously encoded block.

[0036] The CME then re-encodes the current prediction residue using the same elements used to encode a previously encoded block. This process is repeated for several other previously encoded blocks. Blocks adjacent to the current block are typically examined, e.g., the block to the left, the block above it, the block above it and to the left, and the block above it and to the right.

[0037] The distortion values ​​generated for all cases, along with the distortion generated using the previously described dictionary element selection, are compared. If the dictionary element results in the least distortion, the dictionary element indices and corresponding scaling coefficients are included in the output bitstream.

[0038] If one of the copy modes resulted in the least distortion, the copy mode signal and an indicator indicating which copy mode was the best are included in the bitstream. When the copy mode is used, there is no need to signal dictionary element indices at this time, since the indices were already transmitted when a previous block was encoded. Dictionary-based decoder

[0039] Fig. Figure 5 shows the decoder according to embodiments of the invention. The input signal is an encoded video bitstream 501, and the output signal is a decoded macroblock 509. Each current macroblock in the bitstream is entropy decoded 510 and inversely quantized 520 to generate decoded coefficients 521. The coefficients are input to the scaler and combiner 530. The coefficients are used to scale one or more dictionary elements. The scaled elements are typically linearly combined to reconstruct the decoded macroblock prediction residual signal. This residual is combined with previously decoded macroblocks stored in a buffer 540 to generate the output macroblock 509.

[0040] The indices 502 decoded from the bitstream 501 are stored in an index buffer 505. The indices are used to select 550 the dictionary elements 551 that will be used to reconstruct the current prediction residue. In cases where indices are not received via the bitstream, the copy mode signal 503 is used to indicate which set of previously used dictionary elements is used to decode the current block. For example, the copy mode value may indicate that the elements used to generate the block to the left of the current block are used to reconstruct the current block. In another example, the elements used to generate the block above the current block may be used. The main purpose of this copy mode signal is that it requires far fewer bits compared to the indices.

[0041] Since individual stored dictionaries 570 may be available depending on the prediction mode used in the encoder, the prediction mode signal 504 is used to select 560 which of the stored dictionaries to use in the decoder. The prediction mode 504 is also used by the intra / inter prediction module 580 in the decoder, similar to the way the prediction mode is used in a conventional decoder. Dictionary element selection for optimal RD

[0042] The dictionary elements are selected by the MP 550, and the coefficients are determined during the iterations. An important feature of most video coding standards is the RD optimization, which is used for several decision processes, such as when to terminate the iterative MP process performed in 550 and subsequently described in equation (3), and in the module 600 of the copy mode evaluator after Fig. 6. Instead of a fixed number of coefficients, the encoder can provide the best sparse approximation by minimizing the RD cost defined by D(N)+λR(N), where R(N) is the estimated number of bits, D(N) is the total statistical error (MSE) between the original and the reconstructed signal, and λ is a Lagrangian multiplier.

[0043] The RD bias also indicates when the encoder should stop iterating in the MP. If N is dropped, the optimal number of non-zero coefficients can be determined by N*=arg minN∈{0,1,...,C}{D(N)+λR(N)}=arg minN∈{0,1,...,C}{D+λ(RH+RC+RI)}, where C is the given sparsity boundary condition and R H , R C and R I display the required bits for the header, the MP scaling coefficients, and the index representation, respectively.

[0044] The prefix information contains the number of non-zero coefficients, i.e., the number of iterations, so that the decoder determines the correct number of elements during reconstruction. The encoder provides the best sparsity approximation in the constraints of a given bit budget by the optimal value. This process is also available for use in the decoder, e.g., in the dictionary element selector 550 in Fig. 5. Advanced coding technique with dictionary index copying

[0045] In addition to the dictionary index selection process using MP, we also provide an index copying method. In general, neighboring blocks can be assumed to have similar contexts in the spatial domain. And the dictionary elements behave as basis functions during reconstruction. Thus, the same set of dictionary elements can be reused between neighboring blocks if similar patterns appear in the neighboring residual signals. The coefficients of the current signal are determined by applying the previous dictionary elements. Therefore, the encoder can reduce the number of bits required for index coding. A flag is signaled to indicate the source of the index sets selected by examining the RD cost between the candidates.

[0046] The index copy method can be efficient when the signal characteristics in adjacent blocks are essentially stable. Therefore, the method can be adapted to adjacent blocks in homogeneous areas of a video. [Industrial applicability]

[0047] The method according to this invention is applicable to many video coding and decoding processes in many fields.

Claims

[1] A method for decoding a video encoded as a bitstream in a video decoder, comprising the steps of: Maintaining (570) a set of dictionaries generated from decoded prediction residual signals, and wherein elements of the dictionaries of the set of dictionaries have associated indices; Entropy decoding (510) and inverse quantizing (520) a current macroblock of the bitstream to generate decoded coefficients; selecting (560), for the current macroblock, a particular dictionary of the set of dictionaries according to a prediction mode (504) signaled in the bitstream, and selecting (550) particular elements of the particular dictionary according to a copy mode signal (503) in the bitstream and the associated index; scaling and combining (530), using the decoded coefficients, the determined elements (551) to reconstruct a current decoded macroblock prediction residual signal; and Combining (Σ) the current decoded macroblock prediction residual signal with previously decoded macroblocks to generate an output macroblock of reconstructed video, where the steps are performed in a decoder. [2] A method according to claim 1, wherein the set of dictionaries is overcomplete. [3] The method of claim 1, wherein the items in the dictionaries are selected using an adaptation tracking (MP) process. [4] The method of claim 3, wherein the MP process forms a linear combination of dictionary elements by minimizing a residue of the reconstructed video. [5] The method of claim 3, wherein the MP process performs iterations until the rate distortion cost is minimized. [6] A method according to claim 1, wherein the coefficients are binarized via a Huffman table based on statistics of the coefficients. [7] A method according to claim 1, wherein the decoded prediction residual signals are obtained from previously decoded macroblocks of the video. [8] The method of claim 1, wherein the decoded prediction residual signals are obtained from decoded macroblocks of a set of training videos. [9] The method of claim 1, further comprising: Maintaining the set of dictionaries in an encoder to produce the encoded video. [10] A method according to claim 1, wherein the same dictionary elements used to reconstruct decoded macroblocks are used to reconstruct the current macroblock. [11] A method according to claim 9, wherein a cost function is used to select the set of dictionary elements that have already been used to decode a previous macroblock. [12] A method according to claim 9, wherein a copy mode signal is used to indicate when previously used dictionary elements are applied while reconstructing the current macroblock. [13] The method of claim 1, wherein a rate distortion cost metric includes a rate required by the indices, the scaling coefficients, and any subordinate data required to reconstruct the macroblock in the decoder. [14] A method according to claim 1, wherein the indices are stored in a buffer for later use. [15] A method according to claim 9, wherein the copy mode signal is applied to a particular dictionary which has been trained prior to decoding. [16] A method according to claim 9, wherein the copy mode signal is applied to a dictionary calculated during the encoding and decoding processes. [17] The method of claim 1, wherein the video is encoded by forming a set of coding dictionaries from samples of the video, the samples being prediction residuals, further comprising: applying (402) the set of coding dictionaries to the video to generate a sparse representation of the video using elements selected from the set of dictionaries; quantizing (320) the sparse representation to generate a quantized signal; and Entropy coding (330) the quantized signal to generate a bit stream.

Images