Reconstruction by hybrid prediction and residual

Abstract

In one implementation, a simple addition of a predicted sample block (Pred (p)) and a decoded residual sample block (Res p) in a reconstruction process is replaced by a mixing process. The mixing process may be considered as a function f (Pred (p), Res p) involving operations that are more complex than simple addition. For example, in a mixing process, a decoded prediction residual (Res p) may be scaled by a scaling factor and / or shifted by an offset. Prediction samples (Pred (p)) may also be scaled during the mixing process. At the encoder side, the (Res p) mixing parameter (s) for the block may be selected by minimizing the loss function. At the decoder side, a set of mixing parameters may be predefined or decoded, and a particular mixing parameter may be selected by the decoder based on a predicted sample value, a quantization parameter (QP) for a block, or an index signaled for the current block.

Description

Technical Field

[0001] The present embodiments generally relate to methods and apparatus for reconstruction in video encoding and decoding. Background Art

[0002] To achieve high compression efficiency, image and video coding schemes usually use prediction and transform to balance the spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame picture correlation, and then the difference between the original block and the predicted block - often denoted as prediction error or prediction residual - is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy coding, quantization, transform, and prediction. Summary of the invention

[0003] According to one embodiment, a video decoding method is proposed, comprising: obtaining a first data set of a prediction block corresponding to a block of a picture; obtaining a second data set of a decoded prediction residual corresponding to the block of the picture; adjusting the second data set to form an adjusted second data set based on at least a mixing parameter; and combining the first data set and the adjusted second data set to form a decoded version of the block of the picture.

[0004] According to another embodiment, a video encoding method is proposed, comprising: obtaining a first data set of a prediction block corresponding to a block of a picture; obtaining a second data set of a reconstructed prediction residual corresponding to the block of the picture; adjusting the second data set to form an adjusted second data set based on at least a mixing parameter; and combining the first data set and the adjusted second data set to form a reconstructed version of the block of the picture.

[0005] According to another embodiment, a device for video decoding is provided, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to: obtain a first data set corresponding to a prediction block of a block of a picture; obtain a second data set corresponding to a decoded prediction residual of the block of the picture; adjust the second data set to form an adjusted second data set based on at least a mixing parameter; and combine the first data set and the adjusted second data set to form a decoded version of the block of the picture.

[0006] According to another embodiment, a device for video encoding is provided, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to: obtain a first data set corresponding to a prediction block of a block of a picture; obtain a second data set corresponding to a reconstructed prediction residual of the block of the picture; adjust the second data set to form an adjusted second data set based on at least a mixing parameter; and combine the first data set and the adjusted second data set to form a reconstructed version of the block of the picture.

[0007] One or more embodiments also provide a computer program including instructions, which, when executed by one or more processors, cause one or more processors to perform the encoding method or decoding method according to any embodiment described herein. One or more embodiments of the present embodiment also provide a computer-readable storage medium having stored thereon instructions for performing video encoding or decoding according to the method described herein.

[0008] One or more embodiments further provide a computer-readable storage medium on which video data generated according to the above method is stored. One or more embodiments further provide a method and apparatus for transmitting or receiving video data generated according to the method described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A block diagram of a system is illustrated in which aspects of the present embodiments may be implemented.

[0010] Figure 2 A block diagram of an embodiment of a video encoder is illustrated.

[0011] Figure 3 A block diagram of an embodiment of a video decoder is illustrated.

[0012] Figure 4 A block diagram of a video encoder for hybrid decoding of prediction residuals and prediction samples according to an embodiment is illustrated.

[0013] Figure 5 A block diagram of a video decoder that mixes prediction residuals and prediction samples according to an embodiment is illustrated.

[0014] Figure 6 A block diagram of a video encoder that mixes prediction residuals and prediction samples in the transform domain according to an embodiment is illustrated.

[0015] Figure 7 A block diagram of a video decoder that mixes prediction residuals and prediction samples in the transform domain according to an embodiment is illustrated.

[0016] Figure 8A clipping method when mixing is performed in a sample domain according to an embodiment is illustrated.

[0017] Fig. 9 A process of a mixing process of a block to be encoded or decoded according to an embodiment is illustrated.

[0018] Fig.10 A method for obtaining mixing parameters according to an embodiment is illustrated.

[0019] Fig.11 A method for obtaining a mixing parameter based on a quantization parameter according to an embodiment is illustrated.

[0020] Fig.12 A method for obtaining a mixing parameter based on a prediction sample according to an embodiment is illustrated. DETAILED DESCRIPTION

[0021] Figure 1 A block diagram of an example of a system in which various aspects and embodiments can be implemented is illustrated. System 100 can be embodied as a device including various components described below, and is configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 100 can be embodied in a single integrated circuit, multiple ICs and / or discrete components, either individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, system 100 is coupled to other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 100 is configured to implement one or more aspects described in this application.

[0022] The system 100 includes at least one processor 110 configured to execute instructions loaded therein for implementing, for example, various aspects described in the present application. The processor 110 may include embedded memory, input-output interfaces, and various other circuit systems as known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include a non-volatile memory and / or a volatile memory, including but not limited to an EEPROM, a ROM, a PROM, a RAM, a DRAM, an SRAM, a flash memory, a disk drive, and / or an optical drive. As a non-limiting example, the storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device.

[0023] The system 100 includes an encoder / decoder module 130, which is configured to process data, for example, to provide encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a module (one or more) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module 130 may be implemented as a separate element of the system 100, or may be incorporated into the processor 110 as a combination of hardware and software as known to those skilled in the art.

[0024] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described in this application may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during the execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0025] In several embodiments, memory within the processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be a memory 120 and / or a storage device 140, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as for MPEG-2, HEVC, or VVC.

[0026] As indicated in block 105, input may be provided to the elements of system 100 through various input devices. Such input devices include, but are not limited to, (i) an RF section that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0027] In various embodiments, the input device of block 105 has associated corresponding input processing elements as known in the art. For example, the RF part can be associated with elements suitable for the following: (i) selecting a desired frequency (also referred to as selecting a signal, or limiting a signal band to a band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower band, to select a signal band that can be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF part of various embodiments includes one or more elements that perform these functions, for example, a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF part can include a tuner that performs various functions in these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the desired frequency band to perform frequency selection by filtering, down-conversion and filtering again. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements of similar or different functions. Adding element can include inserting element between existing element, for example, inserting amplifier and analog-to-digital converter. In various embodiments, the RF part includes antenna.

[0028] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 100 to other electronic devices via USB and / or HDMI connections. It will be appreciated that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within the processor 110, as necessary. Similarly, various aspects of USB or HDMI interface processing may be implemented, for example, within a separate interface IC or within the processor 110, as necessary. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 110 and the encoder / decoder 130, which operate in combination with memory and storage elements to process the data streams as necessary for presentation on an output device.

[0029] The various elements of system 100 may be provided within an integrated housing within which the various elements may be interconnected and transmit data therebetween using a suitable connection arrangement 115, such as an internal bus known in the art, including an I2C bus, wiring, and printed circuit boards.

[0030] The system 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data through the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented in, for example, a wired and / or wireless medium.

[0031] In various embodiments, data is streamed to the system 100 using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signals of these embodiments are received through a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments provide streaming data to the system 100 using a set-top box that delivers data through an HDMI connection of an input block 105. Still other embodiments provide streaming data to the system 100 using an RF connection of an input block 105.

[0032] The system 100 can provide output signals to various output devices including a display 165, a speaker 175, and other peripherals 185. In various examples of embodiments, the other peripherals 185 include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functions based on the output of the system 100. In various embodiments, control signals are transmitted between the system 100 and the display 165, the speaker 175, or other peripherals 185 using signaling such as AV Link, CEC, or other communication protocols to enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 100 via dedicated connections through the corresponding interfaces 160, 170, and 180. Alternatively, the output devices can be connected to the system 100 using a communication channel 190 via the communication interface 150. The display 165 and the speaker 175 can be integrated into a single unit with other components of the system 100 in an electronic device (e.g., a television). In various embodiments, the display interface 160 includes a display driver, such as a timing controller (T Con) chip.

[0033] For example, if the RF portion of input 105 is part of a separate set-top box, the display 165 and speaker 175 may alternatively be separate from one or more other components. In various embodiments where the display 165 and speaker 175 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0034] Figure 2An example video encoder 200 is illustrated, such as a VVC (Versatile Video Coding) encoder. Figure 2 It is also possible to illustrate an encoder that improves the VVC standard or an encoder that uses a technique similar to the VVC standard.

[0035] In this application, the terms "reconstruction" and "decoding" may be used interchangeably, the terms "encoding" or "encoded" may be used interchangeably, and the terms "image", "picture" and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0036] Before being encoded, the video sequence may undergo pre-encoding processing (201), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to achieve a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0037] In encoder 200, a picture is encoded by encoder elements as described below. The picture to be encoded is divided (202) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When a unit is encoded in intra mode, it performs intra prediction (260). In inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which of intra mode or inter mode to use to encode the unit and indicates the intra / inter decision by, for example, a prediction mode flag. After prediction, prediction enhancement (285) is applied to the prediction block. For example, a prediction residual is calculated by subtracting (210) the prediction block from the original image block.

[0038] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients along with motion vectors and other syntax elements are entropy encoded (245) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e. the residual is directly encoded without applying the transform or quantization process.

[0039] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (255) to reconstruct the image block. An in-loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).

[0040] Figure 3 A block diagram of an example video decoder 300 is shown. In the decoder 300, the bitstream is decoded by the decoder elements as described below. The video decoder 300 generally performs the same operations as described above. Figure 2 The decoding process is the inverse of the encoding process described in . Encoder 200 also typically performs video decoding as part of encoding the video data.

[0041] In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other encoding information. Picture partition information indicates how the picture is partitioned. Thus, the decoder can partition (335) the picture according to the decoded picture partition information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (355) to reconstruct the picture block. The prediction block can be obtained (370) from intra-frame prediction (360) or motion compensated prediction (i.e., inter-frame prediction) (375). After prediction, prediction enhancement (390) is applied to the prediction block. An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380).

[0042] The decoded picture may further undergo post-decoding processing (385), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse of the remapping process performed in the pre-encoding process (201). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0043] The proposed method involves a reconstruction step, corresponding to the addition step (in Figure 2 255 on the encoder side, Figure 3 355 on the decoder side in the decoder). In common video coding standards or alliance solutions such as AVC, HEVC, VVC and AV1, for samples of a given block B, the reconstruction step comprises adding the decoded prediction residual samples Res′(p) (also called residual) obtained from the inverse transform step (250, 350) to the prediction samples Pred(p), obtained from the prediction step (205, 370) when no prediction enhancement is present, or obtained from the prediction enhancement step (285, 390), to obtain reconstructed samples Ree(p) for each sample position p in the block.

[0044] Rec(p)=Pred(p)+Res′(p) Equation 1.

[0045] The residual samples Res(p) of the transformed block are obtained at the encoder side by calculating the difference between the original samples Orig(p) and the predicted samples Pred(p). The resulting difference samples are then processed by a transform followed by a quantization. To reconstruct the block, the encoder then performs a dequantization and an inverse transform. The output is a block of decoded prediction residual samples Res'(p), which represent the sample differences corrupted by the quantization noise Frr(p).

[0046] Res′(p)=Orig(p)-Pred(p)+Err(p) Equation 2

[0047] The reconstruction step does not take this error (Err(p)) into account, nor does it correct for it. A further in-loop filtering step aims to address this problem, but it is also necessary to also try to reduce the error in the reconstruction step itself.

[0048] In this paper, we propose to replace the simple addition of a block of prediction samples and a block of decoded prediction residual samples in the reconstruction step by a mixing process. The mixing process is considered as a function f(Pred(p), Res′(p)) involving operations more complex than simple addition.

[0049] Replace the addition with a mixing process to derive the reconstructed samples

[0050] Figure 4 A modified encoder scheme (400) according to an embodiment is illustrated. In particular, the addition step (255) is replaced by a mixing step (295) which is more complex than a simple addition of predicted and decoded prediction residual samples.

[0051] Figure 5 A modified decoder scheme (500) according to an embodiment is illustrated. In particular, the addition step (355) is replaced by a mixing step (395) which is more complex than a simple addition of predicted and decoded prediction residual samples.

[0052] The mixing step is performed the same at the encoder and decoder sides (295, 395). The mixing process of the prediction block and the residual block generates a reconstructed sample based on the mixing function

[0053] Rec(p) = f(Pred(p), Res′(p)), for all positions p in the block, Equation 3

[0054] where f(a, b) is different from simple addition. The function may also depend on the sample position p. Different implementations of f(.) are described below.

[0055] Scaling correction of decoded prediction residual samples

[0056] In a first embodiment, the blending process operates on the samples of a block as follows:

[0057] Rec(p)=Pred(p)+β(p))*Res′(p) Equation 4

[0058] Where β(p) is a weighting factor (scaling factor) applied to the decoded prediction residual samples, allowing to correct the quantization error of the residual. The value of β(p) is logically constrained to be close to 1, in the interval I = [1-d, 1+d], for example d = 0.1, because it is expected that Res′(p) is close to the true (non-quantized) difference value Res(p)(Orig(p)-Pred(p)). More generally, the constrained interval can be I = [1-d1, 1+d2], where d1 and d2 can take the same or different values.

[0059] In one example, the range of the interval depends on the quantization parameter QP used to quantize the block residual: I = [1-d(QP), 1+d(QP)], where 2*d(QP) is the range of the interval.

[0060] Equation 4 is written in floating point format. However, in practice, fixed point operations will be used in actual implementations. For example, Equation 4 can be implemented as follows:

[0061] Rec(p)=Pred(p)+(β(p)*Res′(p)+md_offset)>>KEquation 5

[0062] where β(p) is a fixed-point value, K is a predefined fixed-point value, and the operator >> corresponds to a right shift that moves the bit value of the binary value, and rnd_offset is a fixed rounding offset, for example equal to 2 K-1 .

[0063] To simplify notation, in the following equations, the sample position dependency “(p)” is removed when it is not necessary.

[0064] Derivation of mixing parameters from adjacent samples

[0065] The inference of β(p) (at the encoder and decoder) can be based on neighboring reconstructed samples Rec, prediction samples and decoded prediction residual samples, such as the top N rows and left columns of neighboring samples of the current block, where N is equal to 4, for example.

[0066] For example, the values ​​of β for different reconstructed sample values ​​can be estimated based on neighboring samples (in a neighboring region ν associated with the current block), knowing that the reconstructed neighboring samples follow Equation 4.

[0067] The sample range of a 10-bit signal [0-1023] can be divided into P non-overlapping intervals I1...I PThe intervals can be of equal length (1024P for a 10-bit signal). Examples of P values ​​are 16 (where the interval length is 64) or 8 (where the interval length is 128) to reduce complexity. For each interval I k , k = 1...P, the following can be applied. For all positions p in the neighboring region v, β is derived from all p k , so that for Pred(p) in I k For example, to minimize the mean square error (MSE):

[0068]

[0069] The beta k The value of will then be used to predict the sample value Pred(p) in the current block. k For all samples in I k All p in the current block

[0070] Rec(p)=Pred(p)+β k (p)*Res′(p) Equation 7.

[0071] Therefore, each position in the current block (denoted as Blk) has a value corresponding to β k The value β(p) of (p), where k is the index of the interval to which Pred(p) belongs.

[0072] In another example, to simplify the mixing parameter derivation, a single value β will be applied to all samples in the current block (P=1) instead of several piecewise (interval-dependent) values. Similar to the above, β is derived from all positions p in the neighboring region v, for example to minimize MSE:

[0073]

[0074] In another example, to further simplify the mixing parameter derivation, a single value β is applied to all samples quantized with the same QP value. QP Instead of applying several values ​​to different blocks, all previously reconstructed samples quantized with the same QP (or restricted to a certain neighboring region v) are used to derive β QP , for example to minimize the MSE:

[0075]

[0076] Therefore, each position in the current block (denoted as Blk) has a value corresponding to β QP (p), where QP is the quantization parameter applied to the block containing the sample at position p.

[0077] In the above, the mixing parameters (β, βk , β QP ) is derived using the loss function MSE (L2-norm). Other loss functions can also be used, such as the mean absolute error (MAE, L1-norm) or the Huber loss function (for small errors, it behaves similarly to MSE, but for large errors, it behaves similarly to MAE, tuned by the hyperparameter δ, which may be related to the QP value).

[0078] Using regularization terms

[0079] In one example, a regularization term is used (added) to constrain β k (p) is the regularization term for the neighbors of pixel position p that belong to the neighborhood V(p).

[0080] For example, to calculate β k To minimize the following function:

[0081]

[0082] Where λ is a predefined regularization parameter. It can be fixed, or depend on the QP parameter, or depend on the value of Pred(p). This generally leads to the minimum mean square error minimization, since all terms in the equation involve squared functions of β. Note that this may require several iterations, since in this equation the value β(q) is considered fixed. For example, the first iteration is applied without the regularization term (λ=0). This leads to the first estimate of β(q) for any p in the block. The following iterations are applied with the regularization term. The process stops after a given number of iterations (e.g. 3), or when the change in β(p) becomes small. When the mixing parameters are derived from the QP, this may lead to

[0083]

[0084] As in the previous case, this may require several iterations, since in this equation the value β(q) is considered fixed. For example, the first iteration is applied without the regularization term (λ=0). This results in a first estimate of β(p) for any p in the block. The following iterations are applied with the regularization term. The process stops after a given number of iterations (e.g. 3), or when the change in β(p) becomes small.

[0085] Alternatively, regularization is applied after a first estimation step based on Equation 6 or Equation 9. After this first estimation step, in the block, in each position p in the block, an estimated β(p) is available. A second step is then applied to regularize (smooth) the β(p) values, for example by simply applying a low-pass filter to the block of β(p) values.

[0086] The regularization term can also be related to the prior probability Proba(rec=r) of the signal. For example, the probability can be calculated from a previously encoded frame. In one embodiment, it is calculated from a previously encoded frame of the same time ID. Alternatively, it can be encoded once in each scene switch, each intra-frame picture, each gop (group of pictures) or each intra-frame slice. The probability value can be provided in the shape of a histogram H(x) covering the signal range (for example, for a 10-bit signal, x is in 0,...,1023). Alternatively, it can be modeled by a parametric function (e.g., a polynomial) or a piecewise parametric function.

[0087] Referring to Equation 8 or Equation 10, the regularization term may include adding a term proportional to:

[0088] -log(Proba(Rec(p))

[0089] This is proportional to:

[0090] -log(Proba(Pred(p)+β(p)*Res′(p))),

[0091] The regularization term constrains β(p) to tend to maximize this probability.

[0092] Signaling Mixing Parameters

[0093] An additional syntax element (e.g., a flag) may be transmitted for each block (TU, CU, or CTU) to indicate whether the proposed mixing will be used for that particular block, rather than a simple addition. When several mixing models are used, the syntax elements may be binary or non-binary. For example, a 1-bit syntax element is used when a single segment-by-segment model (P=1) is used, an 8-bit syntax element is used when 8 segment-by-segment models (P=8) are used, and a 16-bit syntax element is used when 16 segment-by-segment models (P=16) are used. In a variant, the mixing parameters may be explicitly coded and transmitted for each applied block.

[0094] As described above, the value of β should be logically constrained to be close to 1, for example, in the interval I = [1-d, 1+d]. In order to signal the mixing parameters, many bits may be required to encode these mixing parameters. In addition, the computational complexity in the encoder may be quite high. In order to limit the bit cost and computational complexity, in one example, the mixing parameter β can be selected only from a predefined set with M values, for example M = 4 or 8. In this case, only the corresponding index needs to be inferred or encoded and transmitted. The number of possible mixing parameters in the predefined set M may also depend on the slice type. For these possible mixing parameter values, they can vary and adapt to different QPs or content via some offline learning.

[0095] If the value of β is signaled, it should be estimated at the encoder side before signaling. This can be done by rate-distortion optimization, where the distortion is estimated, for example, as:

[0096]

[0097] In the above, the decoded prediction residual is scaled and then added to the prediction samples in the blending process. Other forms of blending functions can also be used.

[0098] In one example, the decoded prediction residual Res′(p) is adjusted by the offset: Rec(p)=Pred(p)+(Res′(p)+b(p)) Eq. 11.

[0099] In another example, the decoded prediction residual Res′(p) is scaled by a scaling factor and adjusted by an offset:

[0100] Rec(p)=Pred(p)+a(p)*Res′(p)+b(p) Equation 12.

[0101] In yet another example, the prediction sample Pred(p) is also scaled by the scaling factor a(p):

[0102] Rec(p)=a(p)*Pred(p)+b(p)*Res′(p)+c(p) Equation 13.

[0103] Mixing in the transform domain

[0104] In another embodiment, the mixing is performed in the transform domain. Figure 6 As depicted in FIG. 1 , at the encoder ( 600 ), a new transform step ( 296 ) applied to the prediction signal is inserted after the prediction enhancement step ( 285 ). The blending step ( 295 ) is applied after the inverse quantization ( 240 ) and uses the output of the new transform step ( 296 ). The output of the blending step ( 295 ) is processed by an inverse transform step ( 297 ) to return to the pixel domain.

[0105] In VVC, the transform process consists of two steps, MTS (Multiple Transform Selection) followed by LFNST (Low Frequency Non-Separable Transform). MTS consists of a set of multiple transforms, and for a transform unit (TU), one transform is selected from this set (signaled or inferred). On the encoder side, once the transform from MTS has been applied to the prediction residual, a secondary transform LFNST can be applied. On the decoder side, the inverse transform applies the inverse process, i.e. inverse LFNST, followed by the inverse transform from MTS.

[0106] The MTS transform matrix used in 296 must be the same as the MTS transform matrix used in step 225 (same MTS transform size, same MTS transform matrix coefficients). In one embodiment, both the MTS transform and LFNST are applied in 296 (when LFNST is activated for a TU). In another embodiment, only the MTS transform is applied in step 296. In this case, in order to be in the same transform domain, an additional inverse LFNST step should be applied between the inverse quantization (240) before the mixing step (295).

[0107] At the decoder (700), as in Figure 7 As depicted in , a new transform step (396) applied to the prediction signal is inserted after the prediction enhancement step (390). The blending step (395) is applied after the inverse quantization (340) and uses the output of the new transform step (396). The output of the blending step (395) is processed by an inverse transform step (397) to return to the pixel domain. It should be noted that the blending is now applied to the dequantized transform coefficients, i.e. the decoded prediction residual samples in the transform domain, before the inverse transform. Similar to the blending in the sample domain (see Figure 4 and Figure 5 ), the dequantized transform coefficients may be scaled and / or shifted by an offset when combined with the (scaled) transformed prediction samples.

[0108] The MTS transform matrix 396 must be the same as the MTS transform matrix used in steps 296 and 225. The inverse MTS transform matrix 397 must be the same as the inverse MTS transform matrix used in step 297. In one embodiment, both the MTS transform and LFNST are applied in 396 (when LFNST is activated for a TU). In another embodiment, only the MTS transform is applied in step 396. In this case, in order to be in the same transform domain, an inverse LFNST step should be applied between the inverse quantization (340) and mixing steps (395).

[0109] When the hybrid process is applied in the transform domain, the adjustment (scaling or offset) of the dequantized prediction residual can be viewed as a refinement of the dequantized prediction residual, or the adjustment in the hybrid process can be viewed as an additional step of the dequantization process. The hybrid process makes it possible to reduce the distortion caused by quantization. That is, now dequantization works more actively to reduce distortion, rather than passively performing dequantization based on parameters designed for quantization.

[0110] The advantage of working in the transform domain is that for each quantized coefficient, the quantization interval is completely known. This interval is defined according to the quantization parameter that controls the quantization step size. The quantization step size depends linearly on the scaling factor "ls[x][y]", for example, as derived in equations 1141 and 1142 of the VVC specification (ITU-T H.266, Series H: Audiovisual and Multimedia Systems, Infrastructure for Audiovisual Services - Coding of Mobile Video, Generic Video Coding, 08 / 2020). The inverse quantization of the decoded coefficients dz[x][y] is achieved by equation (1145) of the VVC specification to obtain the inverse quantized coefficients dnc[x][y]:

[0111] dnc[x][y]=(dz[x][y]*ls[x][y]+bdOffset)>>bdShift

[0112] Equation 14

[0113] Where bdOffset and bdShift are parameters that depend on the signal bit depth.

[0114] Therefore, it is expected that the original coefficients coef[x][y] are in the interval

[0115] Iorig=[dnc[x][y]-Delta / 2; dnc[x][y]+Delta / 2] Equation 15

[0116] in

[0117] Delta=(ls[x][y]>>bdShift) Equation 16.

[0118] This property can be used to constrain the blending process. Typically, this means that the original coefficients (in the transform domain) are in the interval where all values ​​in the interval are equally likely.

[0119]

[0120] Clipping in the transform domain

[0121] The constraint of equation 15 can be used to perform clipping in the transform domain. If the mixing process is performed in the transform domain, steps 295 and 395, where the input to the mixing process is the transform coefficients of the prediction signal (Cpred[x][y]) and the inverse quantized residual coefficients (Cres'[x][y]), and the output from the mixing process is cr=Crec'[x][y], cf should belong to the interval [Crec[x][y]-Delta / 2, Crec[x][y]+Delta / 2], where Crec[x][y]=Cpred[x][y]+Cres[x][y], which means that cf can be clipped according to the following equation:

[0122] cf=Clip3(Crec[x][y]-Delta / 2, Crec[x][y]+Delta / 2, cf), where

[0123] If the blending process is performed in the sample domain (not in the transform domain), as in Figure 4 or Figure 5 As shown in the figure, the limiting process will follow the following steps: Figure 8 Middle diagram:

[0124] - Apply a transform to the input reconstructed sample block Rec' obtained after the mixing process to obtain a transform coefficient block Crec' and apply a transform to the prediction sample block Pred to obtain a transform coefficient block Cpred (step 810). This transform must be the same as the one used in steps 225, 296, 396.

[0125] Crec' = transform(Rec');

[0126] Cpred = transform(Pred);

[0127] - Clip each coefficient cf=Crec'[x][y] according to the following equation (step 820):

[0128] cf=Clip3(Crec[x][y]-Delta / 2, Crec[x][y]+Delta / 2, cf)

[0129] Where Crec[x][y]=Cpred[x][y]+Cres′[x][y], Cres′[x][y] is the inverse quantized residual coefficient.

[0130] Then modify Crec'[x][y] to cf: Crec'[x][y] = cf.

[0131] - Applying an inverse transform to the modified block of transform coefficients Crec' to obtain a modified block of reconstructed samples Rec (830). The inverse transform must be the same as the one used in steps 250, 350, 297, 397.

[0132] Rec'=inverse_transform(Crec').

[0133] Transformation skipping case

[0134] When the transform is skipped (from the encoder decision), the residual of each sample is in the quantization interval defined by [-Delta / 2; Delta / 2]. This property can be used to constrain the blending process. Usually, this means that the original sample (in the sample domain) is in this interval, where all values ​​in this interval are equally likely.

[0135]

[0136] Alternatively, when transform skipping is applied, the clipping process can also be applied directly in the sample domain after steps 255 and 355 or any steps after 295 and 395. The input of the mixing process is the prediction signal (Pred[x][y]) and the inverse quantized residual coefficient Res'[x][y], and the output from the mixing process is R = Rec'[x][y], R should belong to the interval [Rec[x][y]-Delta / 2, Rec[x][y]+Delta2], where Rec[x][y] = Pred[x][y] + Res'[x][y], which means that R can be clipped according to the following equation:

[0137] R=Clip3(Rec[x][y]-Delta / 2, Rec[x][y]+Delta / 2, R).

[0138] Fig. 9 The mixing process in the sample domain of a block to be encoded or decoded according to one embodiment is illustrated. The encoder or decoder can obtain (910) a prediction block from intra-frame or inter-frame prediction with or without prediction refinement. For example, after dequantizing and inverse transforming the transform coefficients, a decoded prediction residual (Res′) is also obtained (910). At 920, (one or more) mixing parameters for the block are obtained. As previously described, the mixing parameters can be a scaling factor that scales (930) the prediction residual, or an offset that adjusts (930) the prediction residual. On the encoder side, (one or more) mixing parameters can be obtained, for example, as described in equations 6-10, to minimize a loss function. For example, on the decoder side, a parameter set can be predefined or decoded, and a specific mixing parameter is selected for the block from the parameter set. The prediction block and the decoded prediction residual can then be combined (940), for example, as described in equations 11-13.

[0139] Fig.10 A method for obtaining a mixing parameter according to an embodiment is illustrated. In this embodiment, at step 1010, a mixing parameter set, β(i), i=1, ..., M, is obtained, for example, decoded from a bitstream or predefined at a decoder for a slice or picture. At step 1020, the decoder decodes a syntax element of index k in the mixing parameter set defining the current block. At step 1030, the mixing parameter is set to β(k).

[0140] Fig.11 Another method of obtaining mixing parameters according to an embodiment is illustrated. In this embodiment, at step 1110, a set of mixing parameters, β(i), i=1, ..., M, is obtained, for example, decoded from a bitstream or predefined at the decoder for a slice or picture. Each mixing parameter in the set corresponds to a different QP. At step 1120, the decoder obtains a quantization parameter QP for the current block. At step 1130, the mixing parameters are set to mixing parameters corresponding to the QP (e.g., β(QP)).

[0141] Fig.12 Another method of obtaining mixing parameters according to an embodiment is illustrated. In this embodiment, at step 1210, a set of mixing parameters, β(i), i=1, ..., P, is obtained, for example, decoded from a bitstream or predefined at a decoder for a slice or picture. Each mixing parameter in the set corresponds to a sample value interval. The decoder loops through each sample in the current block. At step 1220, the decoder obtains a sample value Pred(p) for the current sample. At step 1230, the decoder determines an interval k for Pred(p) (i.e., the sample value Pred(p) belongs to interval k). At step 1240, the mixing parameter is set to the mixing parameter β(k).

[0142] Various methods are described herein, and each method includes one or more steps or actions for realizing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Additionally, terms such as "first", "second", etc. can be used to modify elements, parts, steps, operations, etc. in various embodiments, for example, "first decoding" and "second decoding". Unless specifically required, the use of such terms does not imply the sequencing of the operation to modification. Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can occur in a time period such as before, during, or overlapping with the second decoding.

[0143] The various methods and other aspects described in this application can be used to modify Figure 2 and Figure 3 Modules of the video encoder 200 and decoder 300 shown in , such as the reconstruction module (255, 355). In addition, the present aspects are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations, and extensions of any such standards and recommendations. Unless otherwise indicated or technically excluded, the aspects described in this application can be used alone or in combination.

[0144] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0145] Various implementations involve decoding. "Decoding" as used in this application may include, for example, all or part of the processing performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. Based on the context of the specific description, whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to a broader decoding process will be clear and is considered to be well understood by those skilled in the art.

[0146] Various implementations involve encoding.In a manner similar to the discussion above regarding "decoding," "encoding" as used in this application may include, for example, all or part of the processing performed on an input video sequence to produce an encoded bitstream.

[0147] Note that the grammatical elements as used herein are descriptive terms. As such, they do not exclude the use of other grammatical element names.

[0148] The implementations and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if only discussed in the context of a single implementation form (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., devices or programs). Devices can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in, for example, a device, such as a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0149] Reference to "one embodiment" or "an embodiment" or "an implementation" or "implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations appearing in various places throughout the application are not necessarily all referring to the same embodiment.

[0150] Additionally, the present application may involve “determining” various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0151] Furthermore, the present application may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0152] Additionally, the present application may involve "receiving" various pieces of information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved in one way or another.

[0153] It will be appreciated that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," any of the following uses of " / ," "and / or," and "at least one of..." are intended to include selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to as many items as are listed, as will be apparent to one of ordinary skill in this and related arts.

[0154] In addition, as used herein, the word "signaling" refers to, among other things, indicating something to a corresponding decoder. For example, in some embodiments, the encoder signals a quantization matrix for dequantization. Thus, in one embodiment, the same parameters are used at both the encoder side and the decoder side. Thus, for example, the encoder can transmit (explicitly signal) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters as well as other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit savings are achieved in various embodiments. It will be appreciated that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signal" is mentioned above, the word "signal" can also be used as a noun in this article.

[0155] As will be clear to one of ordinary skill in the art, implementations may generate a variety of signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing the method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

Claims

1. A method for video decoding, comprising: obtaining a first data set corresponding to a prediction block of a block of a picture; obtaining a second data set of decoded prediction residuals corresponding to the block of the picture; adjusting the second data set to form an adjusted second data set based on at least a mixing parameter; as well as The first data set and the adjusted second data set are combined to form a decoded version of the block of the picture.

2. A video encoding method, comprising: obtaining a first data set corresponding to a prediction block of a block of a picture; obtaining a second data set of reconstructed prediction residuals corresponding to the block of the picture; adjusting the second data set to form an adjusted second data set based on at least a mixing parameter; as well as The first data set and the adjusted second data set are combined to form a reconstructed version of the block of the picture.

3. The method according to claim 1 or 2, wherein: The second data set is scaled to form the adjusted second data set.

4. The method according to any one of claims 1 to 3, wherein: The second data set is adjusted at least by an offset to form the adjusted second data set.

5. The method according to any one of claims 1 to 4, wherein: The first data set is scaled when combined with the adjusted second data set.

6. The method according to any one of claims 1 to 5, wherein: The obtaining of the first data set comprises dequantizing transform coefficients of the block to form dequantized transform coefficients, and inverse transforming the dequantized transform coefficients to reconstruct a prediction residual of the block, wherein the reconstructed prediction residual of the block is the second data set, and wherein the prediction block is the first data set.

7. The method according to any one of claims 1 to 5, wherein: The obtaining of the first data set comprises obtaining the prediction block of the block and transforming the prediction block to form the first data set, wherein the obtaining of the second data set comprises obtaining the transform coefficients of the block of the picture and dequantizing the transform coefficients of the block to form dequantized transform coefficients, and wherein the dequantized transform coefficients are the second data set.

8. The method according to any one of claims 1 to 7, wherein: The at least mixing parameter used for adjusting the second data set is constrained to an interval between 1-d and 1+d, wherein d depends on a quantization parameter used for dequantizing transform coefficients of the block.

9. The method according to any one of claims 1 to 8, wherein: The value of the mixing parameter for the prediction residual of the samples in the block depends on the prediction value of the samples.

10. The method according to any one of claims 1 to 8, wherein: The same mixing parameters are applied to all prediction residuals of the block.

11. The method according to any one of claims 1 to 9, wherein: The same blending parameters are applied to blocks with the same quantization parameter.

12. The method according to any one of claims 1 and 3-11, further comprising decoding the at least mixing parameters of the block.

13. The method according to claim 12, wherein: The decoding includes: Get multiple mixing parameters; decoding an index of the block; and A mixing parameter corresponding to the index is selected from the plurality of mixing parameters to adjust the second data set.

14. The method according to any one of claims 1 to 13, further comprising: In a transform domain, sample values ​​in the reconstructed version of the block are clipped between a lower limit and an upper limit based on a parameter indicating a quantization step size for the block and a sum of a dequantized transform coefficient and a transform prediction for the sample.

15. An apparatus for video decoding, comprising one or more processors and at least one memory, wherein the one or more processors are configured to: obtaining a first data set corresponding to a prediction block of a block of a picture; obtaining a second data set of decoded prediction residuals corresponding to the block of the picture; Based at least on the mixing parameter, adjusting the second data set to form an adjusted second data set; as well as The first data set and the adjusted second data set are combined to form a decoded version of the block of the picture.

16. An apparatus for video encoding, comprising one or more processors and at least one memory, wherein the one or more processors are configured to: obtaining a first data set corresponding to a prediction block of a block of a picture; obtaining a second data set of reconstructed prediction residuals corresponding to the block of the picture; Based at least on the mixing parameter, adjusting the second data set to form an adjusted second data set; as well as The first data set and the adjusted second data set are combined to form a reconstructed version of the block of the picture.

17. The device according to claim 15 or 16, wherein: The second data set is scaled to form the adjusted second data set.

18. The device according to any one of claims 15 to 17, wherein: The second data set is adjusted at least by an offset to form the adjusted second data set.

19. The device according to any one of claims 15 to 18, wherein: The first data set is scaled when combined with the adjusted second data set.

20. The device according to any one of claims 15 to 19, wherein: The obtaining of the first data set comprises dequantizing transform coefficients of the block to form dequantized transform coefficients, and inverse transforming the dequantized transform coefficients to reconstruct a prediction residual of the block, wherein the reconstructed prediction residual of the block is the second data set, and wherein the prediction block is the first data set.

21. The device according to any one of claims 15 to 20, wherein: The obtaining of the first data set comprises obtaining the prediction block of the block and transforming the prediction block to form the first data set, wherein the obtaining of the second data set comprises obtaining the transform coefficients of the block of the picture and dequantizing the transform coefficients of the block to form dequantized transform coefficients, and wherein the dequantized transform coefficients are the second data set.

22. The device according to any one of claims 15 to 21, wherein: The at least mixing parameter used for adjusting the second data set is constrained to an interval between 1-d and 1+d, wherein d depends on a quantization parameter used for dequantizing transform coefficients of the block.

23. The device according to any one of claims 15 to 22, wherein: The value of the mixing parameter for the prediction residual of the samples in the block depends on the prediction value of the samples.

24. A signal comprising a bit stream formed by performing the method of any one of claims 1-14.

25. A computer-readable storage medium having stored thereon instructions for encoding or decoding a video according to the method according to any one of claims 1-14.

Images