Video encoding and decoding

Abstract

The present invention relates to a method for decoding a video bitstream, the method comprising the following steps: receiving a bitstream representing the following: a residual sample generated by subtracting an encoder-filtered motion compensation prediction sample from an image sample; and a motion vector used to form the motion compensation prediction sample; the encoder filtering process is performed on the motion compensation prediction sample in an encoder having at least one parameter; providing the motion compensation prediction sample from a pre-reconstructed image using the motion vector; decoder-filtering the motion compensation prediction sample according to the at least one parameter; and reconstructing the image by adding the filtered motion compensation prediction sample to the residual sample. A system and apparatus corresponding to this method are also disclosed.

Description

Technology Field

[0001] The present invention relates to video encoding and decoding, and in particular to video encoding and decoding when motion compensation prediction is used. Background Technology

[0002] Forming a residual by subtracting a prediction based on a preceding (or subsequent) frame from the current frame is a well-known principle in video encoding. In these so-called "inter" coding techniques, coding efficiency requires that the prediction be motion-compensated. Motion vectors between corresponding blocks in temporally spaced frames are measured by the encoder and passed to the decoder as a bitstream. There, the decoded motion vectors are used to derive motion-compensated predictions from the previously decoded frame, which are then added to the subsequently decoded residual.

[0003] Difficulties arose when the video to be encoded contained motion blur—that is, when the motion of the underlying scene was fast enough to cause significant displacement within the camera shutter aperture. It is well known that this phenomenon can be modeled by a low-pass filter applied to the original image over the entire aperture of the camera shutter. Assuming translational motion, the impulse response of the motion blur filter is proportional to the motion components along the horizontal and vertical directions. Therefore, if motion blur occurs in a given frame, the reference frame used for motion compensation (by the decoder) for each coding block will contain a sharper version of the current block. The overall result is that the sharper edges of the reference frame will still remain in the residue, leading to a wider spectrum and less efficient encoding.

[0004] One attempt to address this problem was discussed in U.S. Patent Application 2006 / 0171569, which proposed applying a blurring filter to a "sharper" frame before measuring the motion vector. Subsequently, because the motion vector will be identified between blocks of the motion-blurred frame and corresponding blocks of synthetically blurred frames (rather than sharp frames), a better motion compensation prediction with less residue is expected. Now, it is common to have the encoder select from a number of different frames to use in forming the motion compensation prediction, and it was proposed in U.S. 2006 / 0171569 that the list of frames selected by the encoder includes synthetically blurred frames. The use of the blurred frame is signaled to the decoder, and since there may be options for the type of blurring filter used by the encoder, it is also necessary to provide information about the blurring filter used to the decoder. The problem to be solved

[0005] While the use of such conventional blurring filters has several advantages, there is room for improvement. means of solving the problem

[0006] According to one aspect, a method for decoding a video bitstream comprises the following steps: receiving a bitstream representing the following: a residual sample generated by subtracting an encoder-filtered motion compensation prediction sample from an image sample; and a motion vector used to form the motion compensation prediction sample; an encoder filtering process is performed on the motion compensation prediction sample in an encoder having at least one parameter; providing the motion compensation prediction sample from a pre-reconstructed image using the motion vector; decoder-filtering the motion compensation prediction sample according to the at least one parameter; and reconstructing the image by adding the filtered motion compensation prediction sample to the residual sample.

[0007] Advantageously, the method comprises the step of analyzing a motion compensation prediction sample (or a pre-reconstructed image) to infer at least one parameter of an encoder filtering process, and the step of decoder filtering the motion compensation prediction sample is performed according to the inferred at least one parameter.

[0008] In another perspective, a method for encoding video comprises the following steps: forming a motion compensation prediction sample using a motion vector and a reconstructed image; performing an encoder filtering process on the motion compensation prediction sample; forming a residual sample by subtracting the motion compensation prediction sample from an image sample; and forming a bitstream representing the residual sample, the motion vector, and optionally at least one parameter of the encoder filtering process.

[0009] Advantageously, the method further includes the step of analyzing a motion compensation prediction sample (or a pre-reconstructed image) to determine at least one parameter of an encoder filtering process.

[0010] Preferably, the analyzing step includes determining the existence of any dominant direction in a motion compensation prediction sample or a pre-reconstructed image to infer parameters or parameters related to the existence or details of any anisotropy in the encoder filtering process.

[0011] Encoder and decoder filtering may use a filter aperture that defines weighted contributions from the current motion compensation prediction sample and neighboring motion compensation prediction samples in a block or another set of motion compensation prediction samples. The parameter or one of the parameters, preferably the inferred parameter or one of the inferred parameters, may be related to the anisotropy of the filter aperture and may be a binary flag indicating the isotropy or anisotropy of the filter aperture, and / or may indicate the dominant direction of the anisotropic filter aperture, preferably one of a set of quantized directions. The parameter or one of the parameters, preferably the inferred parameter or one of the inferred parameters, may represent the weights of contributions from a plurality of samples and / or the current motion compensation prediction sample in the filter aperture.

[0012] In another aspect, a method for decoding a video bitstream comprises: receiving a bitstream representing the following: a residual sample generated by subtracting a motion compensation prediction sample from an image sample; and a motion vector used to form the motion compensation prediction sample; an encoder filtering process performed in an encoder having at least one parameter; providing a motion compensation prediction sample from a pre-reconstructed image sample using the motion vector; and reconstructing an image by adding the motion compensation prediction sample to the residual sample; and is characterized by the following: performing an analysis in a decoder to infer at least one parameter of the encoder filtering process; and decoder filtering the motion compensation prediction sample or the pre-reconstructed image sample according to the inferred at least one parameter and optionally according to at least one parameter appearing in the bitstream.

[0013] Advantageously, the analysis step includes the step of determining the existence and / or direction of any dominant direction in a sample array to infer parameters or parameters associated with any anisotropy in an encoder filtering process.

[0014] In another aspect, a method for encoding a video bitstream comprises: forming a motion compensation prediction sample using a motion vector and a reconstructed image sample; subtracting the motion compensation prediction sample from the image sample to form a residual sample; and forming a bitstream representing the residual sample and the motion vector; and is characterized by: performing an analysis to determine at least one parameter of an encoder filtering process that does not appear in the bitstream; and encoder filtering the motion compensation prediction sample or the pre-reconstructed image sample according to the at least one parameter.

[0015] Advantageously, the analyzing step includes determining the existence and / or direction of any dominant direction in a sample array to determine parameters or parameters associated with any anisotropy.

[0016] The above sample array may include motion compensation prediction samples or pre-reconstructed images.

[0017] Encoder or decoder filtering may use a filter aperture that defines weighted contributions from the current sample and neighboring samples in a block or another set of samples, and said parameter or one of said parameters, preferably said inferred parameter or one of said inferred parameters, is associated with the anisotropy of the filter aperture and may be a binary flag indicating the isotropy or anisotropy of the filter aperture. said parameter or one of said parameters, preferably said inferred parameter or one of said inferred parameters, may indicate the dominant direction of the anisotropic filter aperture, preferably one of a set of quantized directions. said parameter or one of said parameters, preferably said inferred parameter or one of said inferred parameters, may indicate the weight of the contributions of a plurality of samples and / or current motion compensation prediction samples in said filter aperture.

[0018] A computer program product adapted for a programmable device to implement any one of these methods may be provided.

[0019] A device configured to implement a method following any one of these methods may be provided. Brief explanation of the drawing

[0020] The present invention will be described by way of example with reference to the attached drawings, wherein: Figure 1 is a block diagram of an encoder; Figure 2 shows a table of filter kernels that can be used; Figure 3 illustrates a method for detecting the gradient dominant direction (GDD) of an image; and Figure 4 is a block diagram of the decoder. Specific details for implementing the invention

[0021] FIG. 1 illustrates an encoder in which motion compensation predictions are subtracted from the current frame to form an encoded bitstream by providing a remainder that is transformed (14), quantized (16), and entropy coded (18). After transformation (14) and quantization (16), the current input frame undergoes inverse quantization (42) and inverse transformation (44) (as is known in the art). The resulting frame is a frame that can be obtained from a decoder; this can then be used as a base frame for a subsequent frame. This frame is combined with the current MC prediction frame to obtain a reference frame (decoded picture buffer (46)) from which a suitable motion compensation prediction for a subsequent frame can be determined.

[0022] When a subsequent input frame is received, this subsequent input frame is combined with a reference frame (from the decoded picture buffer (46)) to obtain a motion estimate (48). This determines a motion vector suitable for mapping a block in the reference frame to a corresponding block in the (subsequent) input frame. The vectors from the reference frame (from the decoded picture buffer (46)) and the motion estimate (48) are then used to obtain a motion compensation prediction (12).

[0023] The optimal MC prediction is then subtracted from the subsequent input frame to form the residue that is transformed (14), quantized (16), entropy encoded (18), and transmitted.

[0024] Up to this point, the encoder in Fig. 1 is a conventional method.

[0025] According to one aspect of the present disclosure, the motion compensation prediction is passed to a smoothing filter that provides an alternative, smoothed motion compensation prediction.

[0026] Thus, the general block diagram of a hybrid motion compensation prediction video encoder is modified through the addition of a smoothing filter. The smoothing filter is an alternative predictor P MCPSIt operates on the predictor (P) (provided by motion compensation) to obtain. Given that applied smoothing may not be beneficial for all coding blocks—for example, since P already provides good coding efficiency—a flag needs to be present in the bitstream for correct decoding. Therefore, on the encoder side for each coding block, the residue is calculated with and without smoothing, and the mode that minimizes the ratio distortion Lagrangian cost is ultimately selected. Steps in this aspect of the encoder workflow—for efficiency—are described below using pseudocode representing the smoothing process of MCPS.

[0027]

[0028] The smoothing process can be represented as a spatial filter with an aperture or kernel that defines which samples contribute to the output and which weights are used.

[0029] In one example, the smoothing process MCPS uses two types of MA (Mobile Average)-like and symmetric kernels. The first type is a 2D and cross-shaped filter, while the second type corresponds to a 1D filter applied along four directions: horizontal, vertical, 45 degrees, and 135 degrees. Other sizes, as well as other filters and directions, may be used. Since the filter of the first type is a 2D filter operating along the horizontal and vertical dimensions, it will be referred to as isotropic below, whereas the kernel of the second type will be referred to as directional below. As mentioned above, the amount of blur depends on the motion activity of different image regions. Accordingly, the sizes of the different kernels used can vary to further improve coding efficiency. In the case of the MCPS method, two example kernel sizes are used: 3 and 5, where the chroma component uses only size 3. Overall, the (exemplary) kernels specified in MCPS are listed in Fig. 2.

[0030] Smoothing uses integer arithmetic with 6-bit internal precision so that the gain of each kernel becomes 64. It should be noted that in this example, the 6-bit value for internal precision was chosen to match the same precision used to represent the kernel coefficients specified in HEVC for motion compensation. Other values ​​may be suitable for future standards.

[0031] There are a total of eight different kernels, and if the encoder intends to perform a thorough search for the best mode during rate distortion optimization, all eight full encodings of the resulting residues must be evaluated along with the case without smoothing. However, the resulting complexity can be enormous in some applications. To alleviate the computational burden and / or provide the benefits described below, the encoder according to this example is selected only from the filter type (i.e., isotropic or directional) and the associated size (i.e., 3 or 5). The remaining parameter, namely the directionality, is determined through a preliminary analysis of motion compensation prediction—in this case—to identify the dominant direction. This can be done by measuring and analyzing gradients.

[0032] For example, as illustrated in FIG. 3, when directional smoothing is selected, the direction in which the kernel should be applied is determined based on the Gradient Dominant Direction (GDD) for the predictor block P. The GDD is defined as the direction that occurs most frequently across all samples belonging to a given motion-compensated predictor P among four specified directions (i.e., horizontal, vertical, diagonal 45, and diagonal 135). Assuming a predictor of size N × N, a 3 × 3 Sobel operator is applied across all samples P(x, y) (32), where x, y = 1, … , N-1. The operator is preferably not calculated for the four outer boundaries of P (i.e., top, bottom, left, and right), so that padding is avoided and 4 × N - 4 operations can be omitted. The operator is applied both horizontally and vertically to obtain Gx and Gy. The direction pointed to by the gradient vector forms a horizontal angle α as follows:

[0033]

[0034] In this case, the symbol '-' describes a different reference system used for the image with respect to a Cartesian 2D plane (wrt). The arctan function is approximated with integer arithmetic and 10-bit precision. The angle α is then uniformly quantized in the direction described above using a quantization step of 45 degrees and a dead-zone range set to 22.5 degrees (34). After quantization (34), a four-bin histogram is calculated (36), and the GDD is determined by the peaks of the histogram (38). In the case of the example shown in FIG. 3, the horizontal direction is selected as dominant.

[0035] With the introduction of directional and variable-size kernels, the general workflow presented above is extended as follows:

[0036]

[0037]

[0038] The metadata signaled at the decoder side specifies whether smoothing is applied, and if so, which type and size are selected. In the HEVC example, the metadata can be transmitted in Coding Units (CUs) and can refer to three color components: therefore, if MCPS is used in the metadata signal, it will be applied to Y, Cb, and Cr. Information associated with MCPS is conveyed via three binary flags as shown in Table 1:

[0039]

[0040] Table 1: Flags for signaling metadata required by MCPS

[0041] In the HEVC example, each flag is coded in CABAC using one context per flag. Finally, MCPS usage can be signaled in both the Sequence Parameter Set (SPS) and the slide header. Slice-level flags are transmitted only when MCPS is enabled at the SPS level. The value of the slice-level flag is determined after the compression of all CTUs belonging to that slice is complete. If no CU has a flag use_mcps_flag that is 1, the slice header flag is set to 0 or 1. It is worth noting that while the value of the slice-level flag should be determined from an RDO perspective, this requires compressing the slice twice, which could be prohibited in terms of encoder complexity. A person of ordinary technical expertise will understand that different signaling techniques may be used, especially in the context of future standards.

[0042] When using integer arithmetic and short-sized kernels, the additional complexity brought about by MCPS—the processing required for the core MCPS algorithm—is quite limited.

[0043] In the decoder, for example as illustrated in FIG. 4, the corresponding smoothing filter (64) operates on the motion compensation prediction (62) to provide an alternative, smoothed prediction for inverse quantization and inverse transformed residue addition.

[0044] The decoder determines the following from the described binary flags: whether smoothing should be used; whether the filter is isotropic or directional; and the filter size. However, if a directional smoothing filter is used, the decoder can infer the direction of the filter to be used. This is achieved by applying the essentially same GDD process to motion compensation prediction.

[0045] A significant advantage is achieved where, in the event of a slight increase in complexity in the decoder, the overhead imposed on the bitstream by this technique is limited to the three binary flags in this example.

[0046] You will understand that the direction of filtering can be quantified more finely by using a larger number of filter kernels. This is because the direction of filtering is inferred in the decoder rather than signaled in the bitstream.

[0047] During transformation, other filter parameters can be inferred or determined by measurements in the encoder and then inferred in the decoder, thereby further reducing the amount of information required to be signaled in the bitstream.

[0048] For example, a process similar to the GDD process can use the degree of anisotropy of the samples to determine whether an isotropic or directional filter should be used. More specifically, the described histogram can be used to infer the type of filter to use. A similar process is performed in the decoder to infer the type of filter to use.

[0049] In another example, the size of the filter to be used can be based on: the spatial resolution of the sequence; the direction of the filter; or the histogram of the gradient direction—specifically, the relative amount of the direction.

[0050] The number and quality of filters used can be extended, for example, to include additional directions and more complex kernel shapes. The weights of the coefficients, particularly the center coefficients of the filter kernels, can be selected, for example, based on the visual characteristics of the content.

[0051] Where appropriate, the decoder can infer all relevant parameters (including their existence) of the encoded filtering process. Then, the parameters do not need to be indicated in the bitstream.

[0052] There are other variations that can have useful applications, particularly when computational resources are limited. Of course, such limitations will apply more generally to decoders.

[0053] For example: Given that MCPS is an additional filtering cascaded to motion compensation, it can be assumed that there is a possibility of combining both motion compensation and smoothing by convolving the associated kernels. In particular, considering that there are many distinct MCPS modes, some pre-calculated kernels can be stored in memory and used later. However, as explained in the preceding section and with reference to Fig. 1, it should be noted that in the case of directional filtering, motion compensation and MCPS cannot be combined because GDD is calculated for samples obtained after motion compensation. To resolve this interleaving dependency, MCPS can use frames obtained by assuming integer pixel motion vector accuracy—and frames obtained using sub-pixel interpolation are not—as predictors (P). Then, smoothing and sub-pixel motion compensation are combined by convolving the associated kernels. In this case, since GDD can be available after integer precision motion compensation but before sub-pixel motion compensation is applied, P MCPS It may be available after a single filtering step.

[0054] In the encoder, the GDD process will operate on the output of the decoded picture buffer, which has then been separated from the smoothing filter. Similarly, in the decoder, the GDD process will operate on the pre-decoded picture.

[0055] It will be recognized that current and proposed standards allow for convexity based on block selection of reference frames as well as motion vectors. Thus, MCPS can be performed individually for each motion vector or for a combination of each motion vector and a reference frame.

[0056] Various other modifications are apparent to a person skilled in the art; for example, while the detailed description of the invention has considered a method primarily used for blurring, this method may also be used for de-blurring frames. This may involve the use of known blur detection techniques rather than GDD, and de-blurring filters rather than smoothing filters. More generally, the invention may be useful for reducing residues within any set of images, and a certain effect may exist in only one of the images.

[0057] Extensions of the method, for example, use for de-blurring, may involve changes to the transmitted flags, and different flags may be transmitted depending on the filter to be applied. Corresponding to the de-blurring example, there may be a selection of flags for the following: no filter; blurring filter; de-blurring filter (of course, this can be extended to include additional flags). Characteristics of filter application, for example, the calculation of the gradient-dominant direction, may be calculated invariantly by the decoder to minimize transmission costs.

[0058] Alternatively, the type of filter to be applied can be inferred from the image being filtered; for example, a blurred image may result in the application of a de-blurring filter. There may be a flag that is set only when an unusual filter is applied—for example, using a blurring filter on an already blurred image.

[0059] Filters can also rely on historical data; for example, if a filter is applied to a prior block or frame, the direction or size of the applied filter can be selected accordingly.

[0060] It should be noted that the smoothing and related decisions described in this document may be applied to any level of granularity during the encoding loop, and may be coding units or prediction units (in the case of HEVC standards) or any kind of partition in other possible future standards.

[0061] Although the present disclosure has focused so far on motion compensation prediction as smoothing or other filtering, the ability to select among a wide range of filters—without a corresponding increase in the signaling burden applied to the bitstream—may be advantageous when smoothing or other filtering is performed on a reference frame prior to motion compensation, for example, as shown in US2006 / 0171569.

Claims

Claim 1 A method for decoding a video bitstream comprises the steps of: receiving a bitstream representing a residual sample generated by subtracting an encoder-filtered motion compensation prediction sample from an image sample; and a motion vector used to form the motion compensation prediction sample; wherein the encoder filtering process is performed on the motion compensation prediction sample in an encoder having one or more parameters; providing the motion compensation prediction sample from a pre-reconstructed image using the motion vector; analyzing the motion compensation prediction sample and / or the pre-reconstructed image to infer at least one parameter among one or more parameters of the encoder filtering process; and decoder-filtering the motion compensation prediction sample according to the at least one parameter. A video bitstream decoding method comprising: a step of reconstructing an image by adding the filtered motion compensation prediction sample to the residual sample; wherein the analyzing step comprises determining the existence and / or direction of any dominant direction in the motion compensation prediction sample and / or the previously reconstructed image, wherein the decoder filtering uses a filter aperture that defines weighted contributions from the current motion compensation prediction sample and neighboring motion compensation prediction samples in a block or another set of motion compensation prediction samples, and wherein at least one parameter is related to the anisotropy of the filter aperture. Claim 2 A video bitstream decoding method according to claim 1, wherein at least one parameter is a binary flag indicating the isotropy or anisotropy of the filter aperture. Claim 3 A video bitstream decoding method according to claim 1, wherein at least one parameter indicates the dominant direction of an anisotropic filter aperture. Claim 4 A video bitstream decoding method according to claim 1, wherein at least one parameter represents a plurality of samples in the filter aperture. Claim 5 A video encoding method comprises: forming a motion compensation prediction sample using a motion vector and a reconstructed image; performing an encoder filtering process on the motion compensation prediction sample; subtracting the motion compensation prediction sample from an image sample to form a residual sample; and forming a bitstream representing the residual sample, the motion vector, and optionally one or more parameters of the encoder filtering process, wherein the video encoding method further comprises: analyzing the motion compensation prediction sample and / or the pre-reconstructed image to determine at least one parameter among one or more parameters of the encoder filtering process; and encoder filtering the motion compensation prediction sample according to the at least one parameter, wherein the analyzing step includes determining the existence and / or direction of any dominant direction in the motion compensation prediction sample and / or the pre-reconstructed image, wherein the encoder filtering uses a filter aperture that defines weighted contributions from the current motion compensation prediction sample and neighboring motion compensation prediction samples in a block or another set of motion compensation prediction samples, and the at least one parameter is related to the anisotropy of the filter aperture. Claim 6 A computer-readable recording medium having a program stored therein, configured such that a programmable device implements the method according to any one of claims 1 to 5. Claim 7 An apparatus configured to implement a method according to any one of claims 1 to 5. Claim 8 A system comprising a device configured to implement the method according to claim 5, communicating with a plurality of devices configured to implement the method according to any one of claims 1 to 4. Claim 9 delete Claim 10 delete Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete Claim 21 delete Claim 22 delete Claim 23 delete Claim 24 delete Claim 25 delete Claim 26 delete Claim 27 delete Claim 28 delete Claim 29 delete Claim 30 delete Claim 31 delete Claim 32 delete Claim 33 delete Claim 34 delete Claim 35 delete Claim 36 delete Claim 37 delete Claim 38 delete Claim 39 delete Claim 40 delete Claim 41 delete

Images