Inter-component prediction
The multi-component image coding technique addresses inefficiencies in existing color space transformations by using a reconstructed first component signal and correction signal to adaptively manage inter-component redundancies, improving coding efficiency and reducing signaling overhead in hybrid video compression.
Patent Information
- Application Number
- JP2024135561
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2014-01-07
- Filing Date
- 2024-08-15
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2034-04-08
AI Technical Summary
Existing image and video compression technologies face inefficiencies due to suboptimal color space transformations that introduce correlations between color components, leading to higher bit rates and non-natural signals, particularly in hybrid video compression schemes, where fixed color transforms fail to completely remove inter-component redundancies.
A multi-component image coding approach that utilizes a spatially corresponding portion of the reconstructed first component signal and a correction signal derived from the data stream to eliminate remaining inter-component redundancy, allowing adaptive control of inter-component dependencies at the granularity of residual or transform blocks, using a block-based hybrid video codec with inter-component prediction.
This method enhances coding efficiency by adaptively managing inter-component redundancies, reducing signaling overhead while maintaining high compression efficiency across various multi-component image content, without the need for lossless color space conversions.
Smart Images

Figure 2024167245000001_ABST
Abstract
Description
[Technical field]
[0001] This application relates to inter-component prediction in multi-component image coding, for example between luma and chroma. [Background technology]
[0002] In image and video signal processing, color information is typically represented primarily in three-component color spaces, such as R´G´B´ or Y´CbCr. The first component Y´ in the case of Y´CbCr is often called luma, and the remaining two components, the Cb and Cr components or planes in the case of Y´CbCr, are called chroma. The advantage of the Y´CbCr color space over the R´G´B´ color space is primarily the residual property of the chroma components, i.e., they contain less energy or amplitude compared to the chroma signals of absolute color spaces such as R´G´B´. Specifically for Y´CbCr, the luma component represents the grayscale information of the image or video, and the chroma component Cb represents the difference compared to the blue primary, and the chroma component Cr represents the difference compared to the red primary.
[0003] In the application space of image and video compression and processing, Y'CbCr signals are preferred over R'G'B' due to the fact that the color space conversion from R'G'B' to Y'CbCr reduces or removes the correlation between different color components or planes. In addition to the decorrelation, less information has to be transmitted, and therefore the color conversion also works as a compression approach. Such pre-processing in decorrelation or reduction enables higher compression efficiency while maintaining or increasing the complexity by a meaningful amount, as an example. Hybrid video compression schemes are often designed for Y'CbCr inputs, since the correlation between different color components is removed or reduced, and furthermore, the design of the hybrid compression scheme only needs to consider the separate processing of the different components. However, the conversion from R'G'B' to Y'CbCr and vice versa is not lossless, and therefore information, i.e., sample values available in the original color space, may be lost after such color conversion. This problem can be avoided by using a color space that includes a lossless transformation from and to the original color space, for example the Y´CoCg color space when having an R´G´B´ input. Nevertheless, a fixed color space transformation may yield suboptimal results depending on the application. For image and video compression, a fixed color transformation often becomes suboptimal for higher bit rates and non-natural signals with high or no correlation between color planes. In the second case, a fixed transformation introduces correlation between different signals, and furthermore, in the first case, a fixed transformation may not remove all correlation between different signals. Furthermore, for global applications of the transformation, correlation may not be completely removed from different components or planes, locally or even globally. Another problem introduced by color space transformation is in the architecture of the image or video encoder. Usually, the optimization process tries to reduce a cost function, which is often a distance metric defined over the input color space.In the case of a transformed input signal, it may be difficult to achieve optimal results for the original input signal due to the additional processing steps. As a result, the optimization process may result in a minimum cost for the transformed signal but not for the original input signal. Even though the transformation is often linear, the cost calculation in the optimization process often includes signaling overhead, and furthermore, the cost for the final decision is then calculated by the Lagrangian formula. The latter may result in different cost values and different optimization decisions. Color transformation aspects are particularly important in the area of color representation, as modern image and video displays usually use R´G´B´ color synthesis for content representation. Generally speaking, transformations are applied when correlations within or between signals should be removed or reduced. As a result, color space transformations are a special case of a more general transformation approach.
[0004] It would therefore be preferable to have a multi-component image coding concept at hand that is more efficient, i.e. achieves higher bit rates over a wider range of multi-component image content. Summary of the Invention [Problem to be solved by the invention]
[0005] It is therefore an object of the present invention to provide such a more efficient multi-component image coding concept. [Means for solving the problem]
[0006] This object is achieved by the subject matter of the pending independent claims. The invention is based on the finding that reconstructing a second component signal related to a second component of a multi-component image from a spatially corresponding portion of the reconstructed first component signal and a correction signal derived from the data stream for the second component promises increased coding efficiency over a wider range of multi-component image contents. By including the spatially corresponding portion of the reconstructed first component signal in the reconstruction of the second component signal, any remaining inter-component redundancy / correlation, possibly still present despite the a priori performed component space transformation or present because it was introduced by such a a priori performed component space transformation, can be readily removed, for example by inter-component redundancy / correlation reduction of the second component signal.
[0007] According to an embodiment of the present application, the multi-component image codec is interpreted as a block-based hybrid video codec operating on the units of code blocks, prediction blocks, residual blocks and transform blocks, and furthermore, the inter-component dependency is switched on and off at the granularity of the residual block and / or transform block by respective signaling in the data stream. The additional overhead for spending signaling is overcompensated by the coding efficiency gain, as the amount of inter-component redundancy can vary within an image. According to an embodiment of the present application, the first component signal is a prediction residual of the temporal, spatial or inter-view prediction of the first component of the multi-component image, and furthermore, the second component signal is a prediction residual of the temporal, spatial or inter-view prediction of the second component of the multi-component image. With this measure, the exploited inter-component dependency focuses on the remaining inter-component redundancy, as the inter-component prediction tends to show a smoother spatial behavior.
[0008] According to an embodiment, the first weight, α, indicated below, a spatially corresponding part of the reconstructed first component signal influences the reconstruction of the second component signal, is adaptively set at sub-picture granularity. By this measure, intra-picture changes in inter-component redundancy can be more closely followed. According to an embodiment, a mixture of a high-level syntax element structure and a first weight syntax element at sub-picture granularity is used to signal the first weight at sub-picture granularity, the high-level syntax element structure defining a mapping from a set of regions of possible bin sequences of a given binarization of the first weight syntax element to a joint region of possible values of the first weight. By this measure, the overhead for side information to control the first weight is kept low. The adaptation can be performed forward-adaptively. A syntax element can be used per block, e.g., residual or transform block, that has a limited number of signalable states that symmetrically index one of a number of weight values for α, which are symmetrically distributed around zero. In one embodiment, the number of signalable states is not equal to the number of weight values including zero, and signaling zero is used to signal non-use of inter-component prediction so that an extra flag is obsolete. Furthermore, the magnitude is signaled before the conditionally signaled code, the magnitude is mapped to the number of weight values, and further, when the magnitude is zero, the code is not signaled so that the signaling cost is further reduced.
[0009] According to an embodiment, a second weight, by which the correction signal influences the reconstruction of the second component signal, is set with sub-picture granularity in addition to or instead of the adaptive setting of the first weight. By this measure, the adaptability of the inter-component redundancy reduction can be further increased. In other words, according to an embodiment, when reconstructing the second component signal, the weight of a weighted sum of spatially corresponding parts of the correction signal and the reconstructed first component signal can be set with sub-picture granularity. The weighted sum can be used as a scalar argument of a scalar function that is a constant, at least for each image. The weights can be set in the backward direction based on a local neighborhood. The weights can be corrected in a forward driving manner.
[0010] According to an embodiment, the domain in which the reconstruction of the second component signal from the spatially corresponding parts of the reconstructed first component signal using the correction signal is performed is the spatial domain. Alternatively, the spectral domain is used. Still alternatively, the domain used is changed between the spatial and the spectral domain. The switching is performed with sub-picture granularity. It has been found that the ability to switch with sub-picture granularity the domain in which the combination of the reconstructed first component signal and the correction signal occurs increases the coding efficiency. The switching can be performed in a backward-adaptive manner or in a forward-adaptive manner.
[0011] According to an embodiment, syntax elements in the data stream are used to enable changing the roles of the first and second component signals within a component of a multi-component image, the additional overhead for signaling the syntax elements being low compared to the possible gains in coding efficiency.
[0012] According to an embodiment, the reconstruction of the second component signal can be switched, with sub-picture granularity, between a reconstruction based only on the reconstructed first component signal and a reconstruction based on the reconstructed first component signal and further reconstructed component signals of further components of the multi-component image. With a relatively low additional effort, this possibility increases the flexibility in removing residual redundancies between components of the multi-component image.
[0013] Similarly, according to an embodiment, a first syntax element in the data stream is used to enable or disable, globally or at an increased scope level, the reconstruction of a second component signal based on a reconstructed first component signal. When enabled, a sub-picture level syntax element in the data stream is used to adapt the reconstruction of the second component signal based on the reconstructed first component signal at sub-picture granularity. With this approach, spending side information for sub-picture level syntax elements is only necessary in the case of applications or multi-component image content where enablement results in coding efficiency gains.
[0014] Instead, the switching between enablement and disablement is performed locally in the opposite direction. In this case, the first syntax element does not even need to be present in the data stream. According to an embodiment, for example, the local switching is performed locally depending on whether the first and second component signals are prediction residuals of spatial prediction in intra-prediction mode of spatial prediction that match or do not deviate more than a predetermined amount. By this measure, the local switching between enablement and disablement does not consume bitrate.
[0015] According to an embodiment, the second syntax elements in the data stream are used to switch between adaptive reconstruction of the second component signal based on the first component signal reconstructed at sub-picture granularity using the sub-picture level syntax elements in the data stream in a forward-adaptive manner and non-adaptively performing reconstruction of the second component signal based on the reconstructed first component signal. The signaling overhead for the second syntax elements is low compared to the possibility of avoiding overhead for transmitting sub-picture level syntax elements for multi-component image content where non-adaptively performing reconstruction is already efficient enough.
[0016] According to an embodiment, the concept of inter-component redundancy reduction is transferred to a three chroma component image. According to an embodiment, a luma and two chroma components are used. The luma component may be selected as the first component.
[0017] According to an embodiment, sub-picture level syntax elements for accommodating the reconstruction of a second component signal from a reconstructed first component signal are coded in the data stream using a Golomb-Rice code. Bins of the Golomb-Rice code may be subjected to binary arithmetic coding. Different contexts may be used for different bin positions of the Golomb-Rice code.
[0018] According to an embodiment, the reconstruction of the second component signal from the reconstructed first component signal comprises spatially rescaling and / or bit-depth fine mapping to a spatially corresponding portion of the reconstructed first component signal. The adaptation of the spatial rescaling and / or the bit-depth fine mapping execution may be performed in a backward and / or forward adaptive manner. The adaptation of the spatial rescaling may comprise the selection of a spatial filter. The adaptation of the bit-depth fine mapping execution may comprise the selection of a mapping function.
[0019] According to an embodiment, the reconstruction of the second component signal from the reconstructed first component signal is performed indirectly via a spatially low-pass filtered version of the reconstructed first component signal.
[0020] Advantageous developments of the embodiments of the present application are the subject of the dependent claims.
[0021] Preferred embodiments of the present application are described below with reference to the figures. [Brief description of the drawings]
[0022] [Figure 1] FIG. 1 shows a block diagram of an encoder configured to encode a multi-component image according to an embodiment. [Diagram 2] FIG. 2 shows a block diagram of a decoder compatible with the encoder of FIG. 1 according to an embodiment. [Diagram 3] FIG. 3 illustrates diagrammatically an image and its subdivision / division into various blocks of different types according to an embodiment. [Figure 4] FIG. 4 illustrates a schematic representation of a reconstruction module incorporated in the encoder and decoder of FIGS. 1 and 2 according to an embodiment of the present application. [Diagram 5] FIG. 5 illustrates diagrammatically an image having two components, a current inter-component predicted block and its spatially corresponding part in the first (base) component. [Figure 6] FIG. 6 shows a schematic image with three components to illustrate an embodiment in which one component always acts as the base (first) component according to the embodiment. [Figure 7] FIG. 7 shows diagrammatically an image having three components, each of which can alternately act as a first (base) component according to an embodiment. [Figure 8] FIG. 8 illustrates diagrammatically how the domain for inter-component prediction can be adaptively varied. [Figure 9a] FIG. 9a illustrates diagrammatically a possibility for signaling inter-component prediction parameters. [Figure 9b] Figure 9b shows the embodiment of Figure 9a in more detail. [Figure 9c] FIG. 9c illustrates diagrammatically the application of the inter-component prediction parameters to which FIGS. 9a and 9b relate for illustrative purposes. [Figure 10] FIG. 10 illustrates a schematic of forward-adaptive inter-component prediction according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0023] The following carry-forward description begins with a detailed embodiment description of an encoder and a detailed embodiment description of a decoder that matches the encoder, followed later by a generalized embodiment.
[0024] Figure 1 illustrates an encoder 100 configured to encode a multi-component image 102 into a data stream 104. Although Figure 1 illustrates exemplarily the case of a multi-component image 102 including three components 106, 108 and 110, the following carryover description makes it clear that advantageous embodiments of the present application can be easily derived from the description of Figure 1 by migrating the current three-component image example to an example in which there are simply two components, e.g. components 106 and 108. Accordingly, some of the elements of the encoder 100 illustrated in Figure 1 are indicated using dotted lines and, to the extent that they are optional features with respect to more generalized embodiments as described further below.
[0025] In principle, the encoder of Figure 1 and all of the embodiments described below can operate with any type of "component." In practice, the multi-component image 102 is an image of a scene that samples the scene with spatially varying components 106, 108 and 110 with other matching conditions, e.g., with respect to timestamps, viewing conditions, etc. However, to facilitate understanding of the embodiments described further below, it is assumed that the components 106, 108 and 110 are all texture-related and furthermore color components, e.g., luma and chroma components, or color components of any other color space.
[0026] 1 is substantially configured to apply block-based hybrid video coding to each of the components 106, 108, and 110 of the image 102 separately, but using inter-component prediction to reduce inter-component redundancy. Inter-component prediction relates to various aspects as outlined in further detail below.
[0027] The encoder 100 therefore comprises, for each component 106, 108 and 110, a prediction residual former 112, exemplarily embodied here as a series of subtractors, a transformer 114 and a quantizer 116, connected in series, in the order of their mention, between an input at which the respective component 106, 108 and 110 arrives, respectively, and a respective input of a data stream former 118, configured to multiplex quantization coefficients and other coding parameters, described in more detail below, into the data stream 104. A non-inverting input of the prediction residual former 112 is arranged to receive the respective component of the image 102, while its inverting (subtractive) input receives a prediction signal 120 from a predictor 122, the input of which is connected to the output of the quantizer 116 via a reconstruction path comprising a series of an inverse quantizer 124, a retransformer 126 and a prediction / residual combiner 128, exemplarily embodied here as an adder. The output of the prediction / residual recombiner 126 is connected to the input of the predictor 122 and furthermore a first input is connected to the output of the reconverter 126, the further input of which receives the prediction signal 120 output by the predictor 122.
[0028] Elements 112, 114, 116, 124, 126, 128, 122 and 120 exist in the encoder 100 in parallel for each of the components 106-110 to form three separate block-based hybrid video coding paths 1301, 1302 and 1303. The indexes 1, 2 and 3 are used to distinguish between different components of the image 102 with index 1 relating to component 106, index 2 relating to component 108 and index 3 relating to component 110.
[0029] In the example shown in Fig. 1, component 106 represents a kind of "base component", as will become clearer from the carry-forward description below. However, it should be noted that the roles between components 106-110 can be equalized in that the configuration described and further shown in Fig. 1 has been selected merely to more easily understand the principles of the present application, as will be further described below. In any case, in order to distinguish between elements 112, 114, 116, 124, 126, 128, 122 and 120 of paths 1301-1303 described thus far, indexes are also used for these elements.
[0030] As will become clear from the above remarks, in the example of Fig. 1, the components 108 and 110 represent "dependent components" and, for this purpose, in addition to those already mentioned for encoding the paths 1301 to 1303, further elements are connected between these already mentioned elements. In particular, the spatial domain inter-component prediction residual former 132 is connected by its non-inverting input and its output between the prediction residual former 112 on the one hand and the input of the transformer 114 on the other hand, and similarly, the spatial domain inter-component prediction and residual recombiner 134 is connected between the retransformer 126 on the one hand and the prediction / residual recombiner 128 on the other hand. Fig. 1 also shows the spectral domain inter-component prediction residual former 136 as being located by its non-inverting input and its output between the transformer 114 and the quantizer 116, together with a corresponding spectral domain inter-component prediction / residual recombiner 138 connected between the inverse quantizer 124 on the one hand and the retransformer 126 on the other hand. The pair of spatial domain inter-component prediction former 132 and the corresponding recombiner 134 is fed at its further input by a respective spatial domain inter-component residual predictor 140, and similarly a further input of the spectral domain inter-view prediction former 136, together with its associated recombiner 138, is fed with a spectral domain residual prediction signal 142 provided by a spectral domain inter-component residual predictor 144. At their input interfaces, the predictors 140 and 144 are connected with any internal node of the respective previous coding path. For example, the predictor 1402 is connected to the output of the prediction residual former 1121, and further the predictor 1442 has an input connected to the output of the inverse quantizer 1241. Similarly, the predictors 1403 and 1443 have inputs connected to the outputs of the prediction residual former 1122 and the inverse quantizer 1242, respectively, but in addition, a further input of the predictor 1403 is shown as being connected to the output of the prediction residual former 1121, and a further input of the predictor 1443 is shown as being connected to the output of the inverse quantizer 1241.Predictors 140 and 144 are shown to optionally generate prediction parameters for implementing a forward-adaptive control of the inter-component prediction performed thereby, the parameters output by predictor 140 being indicated by 146 and the parameters output by predictor 144 being indicated by reference numeral 148. The prediction parameters 146 and 148 of predictors 140 and 144 are received by data stream former 118, which for example encodes all of this data in a lossless manner into data stream 104, as does the residual signal 150 output by quantizer 116.
[0031] Before describing the operation mode of the encoder of Fig. 1 in more detail below, reference is made to Fig. 2 which shows the corresponding decoder. Substantially, the decoder 200 corresponds, component by component, to the part of the encoder 100 extending from the inverse quantizer 124 to the predictor 122 and accordingly the same reference numbers are used for these corresponding elements, but increased by 100. More precisely, the decoder 200 of Fig. 2 is arranged to receive the data stream 104 at the input of a data stream extractor 218 of the decoder 200 which is arranged to extract from the data stream 104 the residual signal 1501-1503 and all the relevant coding parameters, including the prediction parameters 146 and 148. Corresponding to the encoder 100 of Fig. 1, the decoder 200 of Fig. 2 is arranged in parallel decoding paths 230, with one path per component of the multi-component image, the reconstruction of which is indicated using the reference 202 and further the reconstructed components of which are output at the output of the respective decoding paths as 206, 208 and 210. Due to quantization, the reconstruction deviates from the original image 102 .
[0032] As already mentioned above, the decoder paths 2301-2303 of the decoder 200 substantially correspond to those paths of the encoding paths 1301-1303 of the encoder 100, including the elements 124, 138, 126, 134, 128 and 122 thereof. That is, the decoding paths 2302 and 2303 of the "dependent components" 208 and 210 comprise a series connection of an inverse quantizer 224, a spectral domain inter-component prediction / residual recombiner 238, an inverse transformer 226, a spatial domain inter-component prediction / residual recombiner 234 and a prediction / residual recombiner 228, connected between the respective output of the data stream extractor 218 on the one hand and the respective output for outputting the respective components 208 and 210, respectively, in the order of their mention, between the data stream extractor 218 on the one hand and an output of the decoder 200 for outputting the multi-component image 202 on the other hand, the predictor 222 being connected in a feedback loop from the prediction / residual recombiner 228 back to its further input. Further inputs of 238 and 234 are provided by respective spatial and spectral domain inter-component predictors 240 and 244. The decoding path 2301 of the "base component" 206 differs in that elements 238, 234, 244 and 240 are not present. The predictor 2443 has its input connected to the output of the inverse quantizer 2242 and another output connected to the output of the inverse quantizer 2241, and the predictor 2403 has a first input connected to the output of the inverse transformer 2262 and another input connected to the output of the inverse transformer 2261. The predictor 2442 has an input connected to the output of the inverse quantizer 2241, and further, the predictor 2402 has its input connected to the output of the inverse transformer 2261.
[0033] It should be noted that although the predictors 1403 and 1402 in FIG. 1 are shown connected to a not yet quantized residual signal of the underlying component in the spatial domain merely for the sake of easier explanation, as will become apparent from FIG. 2, in order to avoid prediction conflicts between the encoder and the decoder, the predictors 1402 and 1403 in FIG. 1 may alternatively and advantageously be connected to the outputs of the inverse transformers 1262 and 1261, respectively, instead of an unquantized version of the spectral domain residual signal of the underlying component.
[0034] After describing the structure of both the encoder 100 and the decoder 200, their operation modes are described below. In particular, as already mentioned above, the encoder 100 and the decoder 200 are configured to use hybrid coding to encode / decode each component of a multi-component image. In fact, each component of the multi-component image 102 represents a sample array or image, each spatially sampling the same scene with respect to a different color component. The spatial resolution of the components 106, 108 and 110, i.e. the spectral resolution at which the scene is sampled for each component, can differ from one component to another.
[0035] As mentioned above, the components of the multi-component image 102 are subjected to hybrid encoding / decoding separately. "Separately" does not necessarily mean that the encoding / decoding of the components is performed completely independent of each other. First of all, inter-component prediction removes redundancy between the components, and in addition, some coding parameters may be selected in common for the components. In FIG. 3, for example, it is assumed that the predictors 122 and 222 select the same subdivision of the image 102 into coding blocks, but individual subdivision of the components of the image 102 into coding blocks by the predictors 122 and 222 is possible as well. The subdivision of the image 102 into coding blocks may be fixed or signaled in the data stream 104. In the latter case, the subdivision information may be part of the prediction parameters output by the predictor 122 to the data stream former 118 and further indicated with reference number 154, and the data stream extractor 218 extracts these prediction parameters including the subdivision information and further outputs it to the predictor 222.
[0036] 3 exemplarily illustrates that the subdivision of the image 102 into coding or code blocks may be performed according to a two stage process, whereby the multi-component image 102 is first regularly subdivided into tree root blocks, the outline of which is shown in FIG. 3 using double lines 300, followed by applying a recursive multi-tree subdivision to subdivide each tree root block 302 into code blocks 304, the outline of which is shown in FIG. 3 using simple solid lines 306. The code blocks 304 thus represent leaf blocks of the recursive multi-tree subdivision of each tree root block 302. The above-mentioned subdivision information, possibly included within the data stream 104, for the components jointly or individually, may include multi-tree subdivision information for each tree root block 302 for signaling the subdivision of each tree root block 302 to the code blocks 304, and may further optionally include subdivision information for controlling and further signaling the subdivision of the image 102 into a regular array of tree root blocks 302 in rows and columns.
[0037] On a per code block 304 basis, for example, predictors 122 and 222 change between multiple prediction modes supported by encoder 100 and decoder 200, respectively. For example, predictors 1221-1223 individually select a prediction mode for the code block and further indicate the selection to predictor 2223 via prediction parameters 1541-1543. Available prediction modes may include temporal and spatial prediction modes. Other prediction modes may be supported as well, such as inter-view prediction modes. Using recursive multi-tree subdivision, such as dual tree subdivision, the code block may be further subdivided into prediction blocks 308, the outline 310 of which is shown in FIG. 3 using dotted lines. Respective recursive subdivision information may be included in prediction parameters 1541-1543 for each code block. In an alternative embodiment, this selection of prediction mode is performed at the granularity of the prediction block. Similarly, each code block 304 may be further subdivided into residual blocks 312, the outline of which is shown in FIG. 3 using a recursive multi-tree subdivision, such as a dual-tree subdivision. Thus, each code block 304 is divided in parallel into a prediction block 308 and a residual block 312. The residual block 312 may optionally be further subdivided into a transform block 316, the outline of which is shown in FIG. 3 using a double-dotted dashed line 318. Alternatively, the transform and residual blocks form the same entity, i.e., the residual block is the transform block and vice versa. In other words, the residual block may coincide with the transform block according to an alternative embodiment. Residual block and / or transform block related subdivision information may or may not be included in the prediction parameters 154.
[0038] Depending on the prediction mode associated with the respective code block or prediction block, each prediction block 308 has a respective prediction parameter associated with it that is appropriately selected by the predictors 1221-1223 and thus inserted in the parameter information 1541-1543 and further used by the predictors 2221-2223 to control prediction within the respective prediction block 308. For example, a prediction block 308 having a temporal prediction mode associated with it may have a motion vector for motion compensated prediction associated with it and, optionally, a reference image index indicating a reference image from which the prediction of the respective prediction block 308 is derived / copied by the displacement indicated by the motion vector. A prediction block 308 having a spatial prediction mode may have a spatial prediction direction associated with it, included in the prediction information 1511-1543, the latter indicating the direction in which the already reconstructed surroundings of the respective prediction block are spatially extrapolated to the respective prediction block.
[0039] Thus, using the prediction mode and the prediction parameters, the predictors 122 and 222 derive a prediction signal 1201-1203 for each of the components, which prediction signal is then corrected using the residual signal 1561-1563 for each of the components. These residual signals are coded using transform coding, i.e. the transformers 1141-1143 perform a transform, i.e. a spectral decomposition, for example DCT, DST, etc., on each transform block 116 individually, and the inverse transformer 226 inverts it individually for the transform block 308, i.e. performs for example IDCT or IDST. That is, as far as the encoder 100 is concerned, the transformer 114 performs a transform on the not yet quantized residual signal, as formed by the prediction residual former 112. The inverse transformers 126 and 226 reverse the spectral decomposition based on the quantized residual signal 150 of the respective component, which is inserted into the data stream by the data stream former 118 in a lossless manner, for example using Huffman or arithmetic coding, and which is further extracted therefrom by the data stream extractor 218, for example using Huffman or arithmetic decoding.
[0040] However, in order to achieve a lower data rate for encoding the residual signals 2561-2563 with which the prediction signals 2201-2203 are corrected in the prediction / residual recombiners 2281-2283, the encoders 100 and 200 support inter-component prediction for encoding the component residual signals. As described in more detail below, according to an embodiment of the present application, the inter-component prediction for encoding the residual signals can be switched on / off and / or forward and / or backward adaptively adjusted with the granularity of the residual block and / or the transform block. When switched off, the inter-component prediction signals output by the predictors 140, 144, 240 and 244 are zero, and furthermore, the residual signals 2561-2563 of all components are derived solely from the quantized transform coefficients contained in their respective residual signals 1501 and 1503. However, when switched on, the inter-component redundancy / correlation is removed as far as the dependent components are concerned, i.e. the residual signals 2562 and 2563 are encoded / decoded using inter-component prediction. The base (first) component acting as the inter-component prediction source is left unchanged as far as the inter-component prediction is concerned. How this is done is outlined below.
[0041] For the time being, the description of the inter-component redundancy reduction achieved by predictors 140, 144, 240 and 244 will focus on the inter-component redundancy reduction between components 106, 108, 206 and 208, respectively. Then, for ease of understanding, the description will be extended to the three-component case shown in Figures 1 and 2.
[0042] As will become clear from the carry-forward description below, the embodiments outlined below take advantage of, for example, residual characterization of chroma components, especially in the case where absolute chroma components or planes serve as inputs. For example, in the case of three components in Figs. 1 and 2, which represent components of a luma / chroma color space, component 106 / 206 may be, for example, a luma component, while components 108 / 208 and 110 / 210 are chroma components, such as blue and red related chroma components. By keeping the distortion calculation defined over the original input space, the embodiments outlined below enable higher fidelity. In other words, the encoder of Fig. 1 together with the decoder of Fig. 2 enable it to input the original multi-component image, such that there is no color space conversion performed beforehand. The coding efficiency achievable by the embodiments of Figs. 1 and 2 is independent of the input color space, i.e. the embodiments outlined below for performing inter-component redundancy elimination act as an additional decorrelation step for the y'CbCr inputs 106-110 and as a color transformation in the case of the R'G'B' inputs 106-110. In addition, the embodiments outlined below apply redundancy reduction locally between components or planes, i.e. the encoder 100 adaptively decides for each region of the image / picture 102 or video whether inter-plane / component prediction involving residual signals is applied or not. However, it should also be noted that the embodiments of the present invention are not limited to color components or planes only, rather the techniques described herein can be applied to general planes / components as well as cases where the resulting residuals of two or more planes consist of correlations. For example, the embodiments described herein can be applied to planes from different layers in a scalable video compression scheme as well. Upon prediction, the processing order of the components or planes can be locally chanted, for example by syntax elements signaling the component order in the bitstream. This alternative is also described below.
[0043] With reference to Figures 1 and 2, the case of a hybrid block-based image and video compression scheme is shown, in which the prediction performed by the predictor 122 / 222 is performed on a log basis and furthermore a transform coding is applied to the prediction error, i.e. to the output of the prediction residual former 112, called the residual, to which, as far as the encoding side is concerned, the transformer 114, the quantizer 116 and the data stream inserter 118 contribute, and as far as the decoding side is concerned, the inverse transformer 226, the inverse quantizer 224 and the data stream extractor 218 contribute.
[0044] It should be noted that the term block shall be understood as referring to a general rectangular shape as described below, ie a block may have a rectangular shape.
[0045] In the embodiment described next, the encoder determines the application of claim / inter-component prediction for each prediction block 308 or transform block 304 .
[0046] However, as an intermediate remark, it is presented here that the embodiments of the present invention are not limited to the cases outlined with respect to Fig. 1 and Fig. 2, where inter-plane / component prediction is applied to the prediction residual of the spatial, temporal and / or inter-view prediction signal. Theoretically, the embodiments shown here can be transferred to the case where inter-plane / component prediction is performed directly on the samples of the components as well. In other words, when the prediction itself is skipped, the original samples of the input signal should be called or can be treated as the residual for the rest of the description.
[0047] For each residual block 312 (or rectangular shape), according to an embodiment, a syntax element is transmitted in the data stream 104, which further indicates whether inter-plane / component prediction by predictors 140, 144, 240, 244 should be used or not. In the case of a video compression scheme such as H.265 / EVC, also shown in FIG. 3, the residual is further divided into smaller transform blocks or shapes 216, and the encoder 100 can transmit, according to an embodiment, the just-mentioned syntax element specifying the use or non-use of inter-plane / component prediction for each transform block 316 or group of transform blocks. It is noted that the grouping, i.e., the signaling level, can be adaptively selected, and further, such grouping decision can be transmitted as a further syntax element in the bits / data stream 104, for example in a header of the data stream. At the highest signaling level, the use or non-use of inter-plane / component prediction can be signaled for a coding unit or coding block 304, a group of code blocks 304, or even for the entire image or frame.
[0048] Alternatively, reference is made to Fig. 4. Fig. 4 shows a reconstruction module 400, which is divided into several parts, namely: a spatial domain inter-component predictor 1402 and an associated spatial domain inter-component prediction / residual recombiner 1342, which receives x in the form of a data stream extracted, dequantized and inverse transformed residual signal 1561, receives y in the form of a data stream extracted, dequantized and inverse transformed residual signal 1562, and outputs z as a residual signal 156'2 replacing the residual signal 1562; a spectral domain inter-component predictor 1442 and associated spectral domain inter-component prediction / residual recombiner 1382, which receives x in the form of a data stream extracted and dequantized residual signal 1701, receives y in the form of a data stream extracted and dequantized residual signal 1702, and outputs z as a residual signal 170'2 replacing the residual signal 1702; a spatial domain inter-component predictor 1403 and associated spatial domain inter-component prediction / residual recombiner 1343, which receives x in the form of a data stream extracted, inverse quantized and inverse transformed residual signal 1561 or a data stream extracted, inverse quantized and inverse transformed residual signal 1562 (or any combination thereof or both), receives y in the form of a data stream extracted, inverse quantized and inverse transformed residual signal 1563, and outputs z as a residual signal 156'3 replacing the residual signal 1563; a spectral domain inter-component predictor 1443 and associated spectral domain inter-component prediction / residual recombiner 1383, which receives x in the form of a data stream extracted and dequantized residual signal 1701 or a data stream extracted and dequantized residual signal 1702 (or any combination thereof or both), receives y in the form of a data stream extracted and dequantized residual signal 1703, and outputs z as a residual signal 170'3 replacing the residual signal 1703; a spatial domain inter-component predictor 2402 and an associated spatial domain inter-component prediction / residual recombiner 2342, which receives x in the form of a data stream extracted, dequantized and inverse transformed residual signal 2561, receives y in the form of a data stream extracted, dequantized and inverse transformed residual signal 2562, and outputs z as a residual signal 256′2 replacing the residual signal 2562; a spectral domain inter-component predictor 2442 and associated spectral domain inter-component prediction / residual recombiner 2382, which receives x in the form of a data stream extracted and dequantized residual signal 2701, receives y in the form of a data stream extracted and dequantized residual signal 2702, and outputs z as the residual signal 170'2 replacing the residual signal 2702; a spatial domain inter-component predictor 2403 and associated spatial domain inter-component prediction / residual recombiner 2343, which receives x in the form of a data stream extracted, inverse quantized and inverse transformed residual signal 2561 or a data stream extracted, inverse quantized and inverse transformed residual signal 2562 (or any combination thereof or both), receives y in the form of a data stream extracted, inverse quantized and inverse transformed residual signal 2563, and outputs z as a residual signal 256'3 replacing the residual signal 2563; - a spectral domain inter-component predictor 2443 and associated spectral domain inter-component prediction / residual recombiner 2383, which receives x in the form of a data stream extracted and dequantized residual signal 2701 or a data stream extracted and dequantized residual signal 2702 (or any combination thereof or both), receives y in the form of a data stream extracted and dequantized residual signal 2703, and outputs z as a residual signal 270'3 replacing the residual signal 2703; For each pair, the embodiment shown in FIG. 1 and FIG. 2 is embodied.
[0049] In all of these cases of Figures 1 and 2, inter-plane / component prediction is performed as described in more detail subsequently, and furthermore, in any of these cases, the embodiments of Figures 1 and 2 may be supplemented by a more generalized inter-plane / component prediction module 400 of Figure 4. It should be noted that only some of the cases may actually be used.
[0050] As shown in Fig. 4, the inter-plane / component prediction module 400 or reconstruction module 400 has two inputs and one output, and optionally uses prediction parameters signaling between the encoder and decoder side. The first input 402 is shown as "x" and represents the received reconstructed first component signal on the basis of which the inter-plane / component redundancy reduction for the second component is performed by the reconstruction module 400. This reconstructed first component signal 402 may be a residual signal transmitted to be extracted from the data stream for the portion of the first component that is co-located in the spatial or spectral domain with the portion of the second component that is currently the subject of the inter-plane / component redundancy reduction, as shown in Figs. 1 and 2.
[0051] Another input signal 404 of the reconstruction module 400 is denoted as "y" and further represents, in the same domain as the signal 402, i.e. in the spectral or spatial domain, the transmitted residual signal of the part of the second component that is currently the subject of inter-plane / inter-component redundancy reduction by the module 400. The reconstruction module 400 also reconstructs a second component signal 406, denoted as "z" in FIG. 4 in the same domain, which component signal 406 represents the main output of the reconstruction module 400 and which further at least participates in reconstructing the dependent components of the multi-component image by replacing x. "At least" means that, as shown in FIGS. 1 and 2, the component signal 406 output by the reconstruction module 400 can represent a residual prediction and accordingly still represents the predicted signal 154 of the respective dependent component i. i This means that it must be combined with
[0052] As with the other modules of the encoder and decoder, the reconstruction module 400 operates on a block basis. A block-based operation can manifest itself, for example, in a block-wise adaptation of the inter-component redundancy reduction inversion performed by the reconstruction module 400. "Block-wise adaptation" can optionally include an explicit signaling of the prediction parameters 146 / 148 in the data stream. However, a backward adaptive setting of the parameters for controlling the inter-component redundancy reduction is possible as well. That is, with reference to FIG. 4, in the case of a reconstruction module 400 incorporated in a decoder, the prediction parameters input the regulation module and thus represent its further input, while in the case of a reconstruction module 400 incorporated in an encoder, the prediction parameters are determined internally in the manner exemplified below, including, for example, the solution of the LSE optimization problem.
[0053] TIFF2024167245000002.tif48169
[0054] 5 shows a currently reconstructed portion or block 440 of the second component 108. FIG. 5 also shows a spatially corresponding portion / block 442 of the first component 106, i.e., a portion that is spatially co-located in the image 10. The input signals 402 and 404 that the module 400 receives for the components 106 and 108 with respect to the co-located blocks 440 and 442 may represent the residual signals as transmitted for the components 106 and 108 in the data stream as outlined with respect to FIGS. 1 and 2. With respect to the second component 108, the module 400 calculates z. For each block, such as the block 440, the parameters α, β and γ, or just a subset thereof, are adapted in a manner that will be further exemplified below.
[0055] In particular, for a given block, such as block 440, it may be signaled in the data stream by a syntax element as to whether inter-component prediction is performed or not. The parameters α, β and γ in the case of inter-component prediction being switched on merely represent possible examples. For block 440 where inter-component prediction is applied, the prediction mode may be signaled in the data stream, the prediction source may be signaled in the data stream, the prediction domain may be signaled in the data stream, and furthermore parameters related to the above mentioned parameters may be signaled in the data stream. The meaning of "prediction mode", "prediction source", "prediction domain" and "related parameters" will become clear from the carry-forward explanation below. In the example described so far, inter-component prediction operates on the residual signal. That is, x and y are prediction residuals as transmitted in the data stream, and furthermore, both represent prediction residuals of hybrid prediction. As mentioned above, x and y may be prediction residuals in the spatial domain or in the frequency domain in the exemplary case of using transform coding as outlined above. Applying prediction at the encoder or decoder stage has several advantages. Firstly, additional memory is usually unnecessary, and secondly, inter-component prediction can be performed locally, i.e. without the introduction of additional intermediate steps after the analysis process from the decoder's point of view. To distinguish the prediction domain, a further syntax element can be transmitted in the bitstream, i.e. the latter further syntax element can indicate whether the inter-component prediction domain can be the spatial domain or the spectral domain. In the first case, x, y and z are in the spatial domain, and in the latter case, x, y and z are in the spectral domain. It should be noted that from the decoder's point of view, the residuals are reconstructed from the bitstream and may be different from those generated in the encoder before the quantization step. However, it is preferable to use the already quantized and reconstructed residuals as prediction source in the encoder implementing the embodiments of the present application.Furthermore, in the case of skipping the transform stage, the inter-component prediction in the spatial and frequency domains is exactly the same. For such configurations, the signaling of the prediction domain, i.e., spatial or frequency domain, can be skipped.
[0056] TIFF2024167245000003.tif39169
[0057] In order to keep the processing chain as simple as possible, an example configuration may keep the processing of the first component, e.g., the luma component, unchanged, and further use the luma reconstructed residual signal as a predictor for the component residual signal. Note that this is a possible prediction source configuration, and further that such a simple prediction source simplifies the general transform approach, where all three components or planes of the input signal are needed to generate the transform samples.
[0058] Another possible configuration is to perform prediction source adaptation, i.e., to signal that all already available residual signals or the respective reconstructed residual components are used for prediction. Alternatively, the processing order can be changed locally, e.g. the second component is reconstructed first and then used as a prediction source for the remaining components. Such a configuration takes advantage of the fact that the delta between two components using bijective (or approximately invertible) predictors is the same, but for the opposite codes the absolute cost of coding the prediction sources is different. Furthermore, the combination of several prediction sources is possible. In this case the combination weights can be transmitted in the bitstream or estimated in a backward-driving manner using the statistics of the available or respective coded neighborhoods.
[0059] The specification of the model parameters can be performed in backward driving, consisting of backward driving estimation and forward signaling, or can be fully forward signaled in the data stream. An example configuration is to use a fixed set of model parameters known to both the encoder and the decoder, and further signal a set index to the decoder for each block or shape. Another configuration is to use a dynamic set or list, where the order of the predictors is changed after a certain number of blocks or shapes. Such an approach enables a higher local adaptation to the source signal. Further detailed examples on prediction modes, prediction sources, prediction regions and parameter signaling are deferred below.
[0060] Regarding prediction modes, the following may be noted:
[0061] The prediction mode can be an affine, non-linear, or more complex function realized by approaches such as spline or support vector regression.
[0062] It should be noted that the color space conversion is mostly linear using all available input components. That is, the color space conversion tends to map a three-component vector of three color components to another vector of another three components in another color space. The encoder and decoder can operate independently of the input color space according to the embodiments of the present application, and thus the luma component can be kept unchanged to form the prediction source. However, according to alternative embodiments, the definition of the "luma component" or "first component" can be different from block to block, i.e., from block to block or shape to shape, such as a prediction or transformation block (or rectangular shape), and the component that serves as the "first component" can be adaptively selected. The adaptation can be indicated to the encoder by signaling in the data stream. For example, FIG. 5 shows the case where component 106 forms the "first component" and component 108 is the inter-component predicted component, i.e. the second component, and as far as block 440 is concerned, it can be different for another block of image 10, e.g. component signal 106 is inter-component predicted from component 108. Block-wise adaptation of the "prediction source" can be indicated in the data stream as just outlined. Additionally or alternatively, a syntax element signaling whether prediction is applied or not for each block 440 can be transmitted in the data stream. That is, in the case of always using component 106 as the "first component", such a syntax element is present only for the dependent components, i.e. the blocks of components 108 and 110 in FIG. 1 and FIG. 2. In that case, the first component, e.g. the luma component, provides the prediction source, i.e. its residual signal in the case of the embodiment outlined above.
[0063] In addition or alternatively, if prediction is enabled for a particular block 440, the prediction mode may be signaled in the data stream for that block 440. Note that prediction may be skipped for the case of a zero-valued reconstructed first component signal, i.e., a zero-valued residual luma signal in the case of using the prediction residual as a base for inter-component prediction. In that case, the above-mentioned syntax element signaling whether inter-component prediction is applied or not may be omitted, i.e., not present, in the data stream for the respective block 440.
[0064] In the case of a combined backward driving and forward signaling approach, parameters derived from the respective reconstructed data already coded can serve as starting parameters. In such a case, deltas compared to the selected prediction mode can be transmitted in the data stream. This can be achieved by calculating optimal parameters for a fixed or adapted prediction mode, and the calculated parameters are then transmitted in the bitstream. Another possibility is to transmit some deltas compared to the starting parameters derived by using the backward driving selection approach, or by always using parameters calculated and selected by the backward driving approach only.
[0065] TIFF2024167245000004.tif60169
[0066] It should be noted that a single element in y is replaced after prediction by z. In other words, the reconstruction module 400 receives the corrected signal 404 and replaces it with z 406. It should also be noted that when using such a configuration, the inter-component prediction is simplified to an addition operation: the fully reconstructed residual sample value of the first component (α=1) or its half value (α=0.5) is added to the corrected sample value 404. The halving can be generated by a simple right shift operation. In that case, for example, the configuration can be implemented by realizing a multiplication between x and α in the predictor in the reconstruction module 400 and further realizing an addition in the adder shown in FIG. 4. In other words, in the given example, on the encoder side, the operation mode is as follows:
[0067] TIFF2024167245000005.tif78169
[0068] TIFF2024167245000006.tif71170
[0069] The parameters for the first sample or group of samples can be initialized by some default values or can be calculated from adjacent blocks or shapes. Another possibility is to transmit optimal parameters for the first sample or group of samples. In order to use as many previous samples as possible, a predefined scanning pattern can be used to map the two-dimensional residual block (or rectangular shape) into a one-dimensional vector. For example, the sample or group of samples can be scanned vertically, horizontally, or in a direction similar to the scanning direction of the transform coefficients. Also, the particular scanning can be derived by a backward driving scheme or it is signaled in the bitstream.
[0070] Another extension is the combination of the already shown examples and a transformation using all three available components as input. Here, in this example, the residual signal is transformed using a transformation matrix that removes the correlation for a given transformation block. This configuration is useful when the correlation between planes is very large or extremely small. An example configuration uses a Y'CoCg transformation in the case of input color spaces such as R'G'B' or a principal component analysis approach. For the latter case, the transformation matrix must be signaled to the decoder in a forward manner or using a predefined set and rules known to the encoder and decoder to derive the matrix values. Note that this configuration requires the residual signals of all available components or planes.
[0071] Regarding the prediction domain, the following is noted:
[0072] The prediction domain can be the spatial domain, i.e., operating on the residual, as described above, or the frequency domain, i.e., operating on the residual after applying a transform such as DCT or DST. Furthermore, the domain can be both configurations, by transmitting information to the decoder. In addition to the domain for prediction, parameters related to the domain can be transmitted or derived by a backward driving scheme.
[0073] Related to the domain parameters is the additional subsampling of the chroma components, i.e. the chroma block (or rectangular shape) is scaled down horizontally, vertically or both. In such cases, the prediction source may be downsampled as well, or a set of prediction modes must be selected that considers different resolutions in the spatial domain, the frequency domain or both. Another possibility is to upscale the prediction target so that the dimensions of the prediction source and the prediction target match each other. Downscaling further improves the compression efficiency, especially for very flat regions in the image or video. For example, the prediction source contains both low and high frequencies, but the chroma block (or rectangular shape) contains only low frequencies. In this example, the subsampling of the prediction source removes the high frequencies, and furthermore, less complex prediction modes can be used, which further leads to less information having to be transmitted to the decoder. It should be noted that the signaling of downscaling use can be done for each transform block, or for each prediction block, or even for a group of prediction blocks or for the whole image.
[0074] In addition to the additive downsampling approach, bit depth adjustments can also be transmitted to the decoder. This case occurs when the precision of the samples differs along different components. One possible way is to reduce or increase the number of bits for the source. Another possible configuration could increase or reduce the target bit depth and then correct the final result back to the correct bit depth. A further option is the use of a set of predictors suitable for different bit depths. Such prediction modes consider different bit depths by their prediction parameters. The signaling level for the bit depth correction can be done for each block or shape or for the whole image or sequence, depending on the change in content.
[0075] The following forecast sources are noted:
[0076] The prediction source may be the first component or all available components. For each block (or rectangular shape), the prediction source may be signaled by the encoder. Alternatively, the prediction source may be a dictionary containing all possible blocks of prediction or transform blocks from all available components, and the prediction source to be used for prediction is signaled to the decoder by a syntax element in the bitstream.
[0077] Regarding the parameter derivation, the following is noted:
[0078] TIFF2024167245000007.tif43170
[0079] TIFF2024167245000008.tif34168
[0080] TIFF2024167245000009.tif54170
[0081] TIFF2024167245000010.tif61170
[0082] TIFF2024167245000011.tif31170
[0083] Note that in the above minimization problem, y represents the residual signal that is losslessly transmitted in the data stream for each inter-component predicted block 440. In other words, y is the correction signal for the inter-component predicted block 440. It may be determined iteratively while performing the solution of the LSE optimization problem outlined above at each iteration. In this way, the encoder can optimally decide whether to perform inter-component prediction or not when selecting inter-component prediction parameters such as prediction mode, prediction source, etc.
[0084] Regarding parameter signaling, the following is noted:
[0085] The prediction itself may be switchable, and a header flag specifying the use of residual prediction should be transmitted at the beginning of the bitstream. When prediction is possible, a syntax element specifying its local use is embedded in the bitstream for the block 440, which may be a residual block (or rectangular shape), a transform block (or rectangular shape), or a group of transform blocks (or rectangular shapes), as described above. The first bit may indicate whether prediction is enabled, and the following bits may indicate the prediction mode, the prediction source, or the prediction domain and related parameters. Note that one syntax element can be used to enable or disable prediction for both chroma components. It is also possible to signal the use of prediction and the prediction mode and source separately for each second (e.g. chroma) component. Also, to achieve high adaptation to the residual signal, a sorting criterion may be used to signal the prediction mode. For example, the most used prediction mode is 1, and the sorted list includes mode 1 at index zero. Then, only one bit is needed to signal the most probable mode 1. Furthermore, the use of prediction may be limited in the case of applying inter-component prediction to the residual, and correlation may be high if the same prediction mode for generating the residual is used in different color components. Such a restriction is useful for intra-predicted blocks (or rectangular shapes). For example, this inter-component prediction may be applied only if the same intra-prediction mode is used for the block (or rectangular shape) 440, which may be chroma, as is used for the block (or rectangular shape) 442, which may be luma.
[0086] That is, in the latter case, the block-wise adaptation of the inter-component prediction process involves checking whether the block 440 is relevant for a spatial prediction mode as far as prediction by the predictor 2222 is concerned, and whether the spatial prediction mode does not deviate more than a predefined amount from the spatial mode with which the corresponding or co-located block 442 is predicted by the predictor 2221. The spatial prediction mode may for example include a spatial prediction direction in which the adjacent blocks 440 and 442 of already reconstructed samples, respectively, are extrapolated to the blocks 440 and 442, respectively, to result in the respective prediction signals 2202 and 2201, respectively, which are combined with z and x, respectively.
[0087] In an embodiment, the prediction source is always the luma component. In this preferred embodiment, depicted in Figures 1 and 2, when the interconnection between components 108 / 208 and 110 / 210 is dropped, the luma processing is unchanged and the resulting reconstructed residual of the luma plane is used for prediction. As a result, the prediction source is not transmitted in the bitstream.
[0088] In another embodiment, a prediction source is transmitted for a l block or shape 440, such as a residual block, a group of residual blocks or shapes, for example the size for which intra or inter prediction is applied.
[0089] For example, the prediction source is luma for the first chroma component and luma for the second chroma component or the first chroma component. This preferred embodiment is similar to the configuration that allows all available planes as prediction sources and also corresponds to Figures 1 and 2.
[0090] To illustrate the above-mentioned embodiment, reference is made to FIG. 6. Three component images 102 / 202 are shown. The first component 106 / 206 is coded / decoded using hybrid (decoding) coding without any inter-component prediction. As far as the second component 108 / 208 is concerned, it is divided into blocks 440, one of which is exemplarily shown in FIG. 6, just as it was the case in FIG. 5. In units of these blocks 440, the inter-component prediction is adapted, for example, with respect to α, as it has been described above. Similarly, the image 102 / 202 is divided into blocks 450 as far as the third component 110 / 210 is concerned, one such block is exemplarily shown in FIG. 6. As described with respect to the two alternatives just outlined, it may be that the inter-component prediction of the block 450 is necessarily performed by using the first component 106 / 206 as a prediction source, i.e. the co-located part 452 of the first component 106 / 206. This is indicated using a solid arrow 454. However, according to a second alternative just described, a syntax element 456 in the data stream 104 switches between using the first component as a prediction source, i.e. 454, and using the second component as a prediction source, i.e. inter-component prediction of the block 450 based on a co-located portion 458 of the second component 108 / 208 as indicated by arrow 460. In Fig. 6, the first component 106 / 206 may be, for example, a luma component, while the other two components 108 / 208 and 110 / 210 may be chroma components.
[0091] 6, it should be noted that perhaps the division into blocks 440 for the second component 108 / 208 can be signaled in the data stream 104 in a manner independent of or allowing deviation from the division of the components 110 / 210 into blocks 450. Of course, the division can be the same as well and even be taken from one signaled form or used for the first component as is the exemplary case in FIG. 10 described later.
[0092] A further embodiment just outlined above is the possibility that all components of the image 110 / 210 can alternately serve as prediction sources. See FIG. 7. In FIG. 7, all components 106-110 / 206-210 have a common division into blocks 470 associated therewith, with one such block 470 exemplarily shown in FIG. 7. The division or subdivision into blocks 470 can be signaled in the data stream 104. As already mentioned above, the block 470 may be a residual or a transform block. However, now any component can form the "first component", i.e. the prediction source. The syntax element 472 and the data stream 104 indicate for the block 470 as to which of the components 106-110 / 206-210 forms, for example, a prediction source for the other two components. FIG. 7 shows, for example, another block 472, whose prediction source is selected differently as indicated by the arrows in FIG. 7. Within block 470, a first component serves as a prediction source for the other two components, in which case in block 472 the second component assumes the role of the first component.
[0093] With respect to Figures 6 and 7, it should be noted that the extent to which syntax elements 456 and 472 indicate a prediction source may be selected differently, i.e. for each block 450 and 470 / 72 individually, or for groups of blocks or even the entire image 102 / 202.
[0094] In a further embodiment, the prediction sources may be all available components or a subset of the available components. In this preferred embodiment, the weighting of the sources may be signaled to the decoder.
[0095] In a preferred embodiment, the prediction region is in the spatial domain. In this embodiment, the residual for the entire residual block or shape or only certain parts of the residual block or shape may be used, depending on the signaling configuration. The latter case is given when the prediction is signaled individually for each transform block or shape, and further subdivision of the residual block or shape into smaller transform blocks or shapes is possible.
[0096] In a further embodiment, the prediction domain is in the frequency domain. In this preferred embodiment, the prediction is tied to the transform block or shape size.
[0097] In a further embodiment, the prediction region may be in the spatial or frequency domain, where the prediction region is identified separately by forward signaling or backward driving estimation depending on local statistics.
[0098] TIFF2024167245000012.tif99169
[0099] Also, syntax element 490 can signal the region used for inter-component prediction for a block 440 individually, for a group of blocks 440 or for an entire image or even a larger scope, e.g., for a group of images.
[0100] In a further embodiment, both prediction domains are included in the prediction process: in this preferred embodiment of the invention, prediction is first made in the spatial domain and then a further prediction is applied in the frequency domain with both predictions using different prediction modes and sources.
[0101] In an embodiment, a chroma block or shape can be subsampled horizontally, vertically, or both by some factor. In this embodiment, the downscale factor can be equal to a power of 2. The use of a downsampler is signaled as a syntax element in the bitstream, and further, the downsampler is fixed.
[0102] In a further embodiment, a chroma block or shape can be subsampled horizontally, vertically, or both by some factor, which can be transmitted in the bitstream, and further, the downsampler is selected from a set of filters, and the exact filter can be addressed by an index that is also transmitted in the bitstream.
[0103] In a further embodiment, the selected upsampling filter is transmitted in the bitstream. In this embodiment, the chroma blocks may be subsampled first, and therefore, in order to use prediction with matching block or rectangle sizes, upsampling must be performed before prediction.
[0104] In a further embodiment, the selected downsampling filter is transmitted in the bitstream. In this embodiment, the luma is downsampled to achieve the same block or rectangle size.
[0105] In an embodiment, a syntax element is signaled to imply bit correction when the source and target bit depths are different. In this embodiment, the luma precision can be reduced or the chroma precision can be increased to have the same bit depth for prediction. In the latter case, the chroma precision is reduced back to the original bit depth.
[0106] In an embodiment, the number of prediction modes is two and the set of predictors is defined exactly as in the given example.
[0107] In a further embodiment, the number of prediction modes is 1 and further the configuration is the same as described in the previous embodiment.
[0108] In a further embodiment, the number of predictors is freely adjustable and the set of predictors is defined exactly as in the given example. This embodiment is a more general description of the example with α=1 / m, where m>0 indicates the prediction number or mode. Thus, m=0 indicates that the prediction should be skipped.
[0109] In a further embodiment, the prediction mode is fixed, i.e., prediction is always enabled. For this embodiment, it may enable adaptive inter-plane prediction and further set the number of predictors equal to zero.
[0110] In a further embodiment, prediction is always applied and furthermore prediction parameters such as α are derived from neighboring blocks or shapes. In this embodiment, the optimal α for a block or shape is calculated after full reconstruction. The calculated α serves as a parameter for the next block or shape in the local neighborhood.
[0111] In a further embodiment, a syntax element is transmitted in the bitstream indicating the use of parameters derived from a local neighborhood.
[0112] In a further embodiment, the parameters derived from the neighborhood are always used. In addition, the delta compared to the optimal parameters calculated in the encoder can be transmitted in the bitstream.
[0113] In a further embodiment, the backward driving selection scheme for parameters is disabled and the optimal parameters are transmitted in the bitstream.
[0114] In a further embodiment, the use of starting α and the presence of delta α in the bitstream are signaled separately.
[0115] In an embodiment, the signaling of prediction modes, prediction sources and prediction parameters is restricted to the same regular prediction modes, in which information related to inter-plane prediction is transmitted only when the intra prediction mode for the chroma component is the same as that used for the luma component.
[0116] In a further embodiment, the block is divided into windows of different sizes and further, parameters for a current window are derived from a reconstructed previous window within the block, hi a further embodiment, parameters for a first window are derived from a reconstructed adjacent block.
[0117] In a further embodiment, a syntax element is transmitted in the bitstream indicating the use of parameters derived from the local neighborhood used for the first window.
[0118] In a further embodiment, the window may be scanned vertically, horizontally or vertically.
[0119] In a further embodiment, the parameters of the current window are derived from a previous window, where the previous window is determined according to a scan position of the transform coefficient sub-block.
[0120] In a further embodiment, the window scanning is limited to one scan direction.
[0121] In a further embodiment, the parameters are derived using an integer implementation by using a lookup table and multiplication operations instead of division.
[0122] In an embodiment, a global flag transmitted in the bitstream header indicates the use of adaptive inter-plane prediction, in this embodiment, the flag is embedded at the sequence level.
[0123] In a further embodiment, the global flag is transmitted in the bitstream header with embedding at the picture parameter level.
[0124] In a further embodiment, the number of predictors is transmitted in the bitstream header, where the number zero indicates that prediction is always enabled and a number not equal to zero indicates that the prediction mode is adaptively selected.
[0125] In an embodiment, the set of prediction modes is derived from the number of prediction modes.
[0126] In a further embodiment, the set of prediction modes is known to the decoder and specifies all model parameters for prediction.
[0127] In a further embodiment, the prediction modes are all linear or all affine.
[0128] In an embodiment, the set of predictors is hybrid, i.e., it includes simple prediction modes that use other planes as prediction sources, as well as more complex prediction modes that use all available planes, and also transforms the input residual signal into another component or plane space.
[0129] In an embodiment, the use of prediction is specified per transform block or shape per chroma component. In this embodiment, this information may be skipped when the luma component consists of a zero-valued residual at the same spatial location.
[0130] In an embodiment, the modes are transmitted using a truncated unary decomposition. In this embodiment, a different context model is assigned per bin index, but limited to a certain number, e.g., 3. Furthermore, the same context model is used for both chroma components.
[0131] In a further embodiment, different chroma planes use different sets of context models.
[0132] In a further embodiment, different transform block or feature sizes use different sets of context models.
[0133] In a further embodiment, the mapping of bins to prediction modes is dynamic or adaptive, in which, from the decoder's point of view, a decoded mode equal to zero indicates the most used mode up to the time of decoding.
[0134] In a further embodiment, the prediction mode and the prediction source are transmitted for the residual block or shape when using a configuration that allows different prediction sources. In this embodiment, different block or shape sizes can use different context models.
[0135] The embodiments described next relate in particular to examples for methods of encoding prediction parameters for the "cross-component decorrelation" described so far.
[0136] Although not limited thereto, the following description may be considered to refer to an alternative in which the dependent (second) component is reconstructed based on the reference (first) component signal x and the residual (correction) signal y via the calculation z=αx+y with z as the prediction of the dependent component signal. For example, the prediction may be applied in the spatial domain. As in the above example, inter-component prediction may be applied to the hybrid coding residual signal, i.e. the first and second component signals may represent the hybrid coding residual signal. However, the following embodiments concentrate on the signaling of α, which is coded on a sub-picture basis, such that this parameter is, for example, in units of residual blocks into which a multi-component image is subdivided. The following is about the fact that the signaling possible state of α should preferably be variable, also to take into account the fact that the range of optimal values for α depends on the type of image content, which in turn varies in a range / unit larger than the residual block. Thus, in principle, the details described below regarding the transmission of α may be transferred to other embodiments, as outlined above as well.
[0137] Cross-component decorrelation (CCD) approaches exploit the remaining dependence between different color components enabling higher compression efficiency. An affine model can be used for such an approach, and furthermore, the model parameters are transmitted in the bitstream as side information.
[0138] To minimize the side information cost, only a limited set of possible parameters is transmitted. For example, a possible CCD implementation in High Efficiency Video Coding (HEVC) may use a linear prediction model instead of an affine prediction model, and furthermore, only the mode parameter, i.e., the slope or gradient parameter α, may be limited in the range from 0 to 1 and may be non-uniformly quantized. In particular, the limited set of values for α may be α∈{0, ±0.125, ±0.25, ±0.5, ±1}.
[0139] The choice of such quantization for the linear prediction model parameters can be based on the fact that the distribution of α is symmetrically centered around the value 0 for natural video content stored in the Y´CbCr color space, where the color components are decorrelated by using a fixed transformation matrix that converts from R´G´B´ before entering the compression stage. Due to the fact that global transformations are often suboptimal, the CCD approach can achieve higher compression efficiency by removing the remaining dependencies between the different color components.
[0140] However, such an assumption does not hold for different kinds of content, in particular for natural video content stored in the R'G'B' color space domain. In this case, the gradient parameter α is often centered around the value 1.
[0141] Similar to the case given above, the distribution becomes completely different when the CCD is extended to the first chroma component as a prediction source. Therefore, it may be beneficial to adjust the CCD parameters according to the given content.
[0142] For example, for Y´CbCr, the quantization of α can be set to (0, ±0.125, ±0.25, ±0.5, ±1), while for R´G´B´, the quantization of α can be the other way around, i.e. (0, ±1, ±0.5, ±0.25, ±0.125). However, allowing different entropy coding paths introduces additional problems. One problem is that the implementation becomes more expensive in terms of area and speed for both hardware and software. To avoid this drawback, parameter ranges can be specified at the Picture Parameter Set (PPS) level, where the use of the CCD is also indicated.
[0143] That is, the syntax element signaling α is transmitted at a sub-picture level / granularity, e.g. for each residual block individually. For example, it may be called res_scale_value. It may be coded, e.g., using a (truncated) unary binarization combined with a binary arithmetic coding of the bin sequence. The mapping of (non-binarized) values of res_scale_value to α may be performed such that the mapping is varied with pps, i.e. for a complete image, or even to a larger extent, e.g., on a per image sequence basis. The variations may vary the number of representable α values, the order of representable α values and the selection of representable α values, i.e. their actual values. While simply allowing to switch the order among representable alpha values or limiting representable alpha values to positive or negative values is one way of providing content adaptability, the embodiments outlined below further enable much increased flexibility in changing the mapping from res_scale_value signaled at sub-picture granularity to alpha values (representable set of alpha values), e.g., changing the size, members and member order of the joint region of the mapping, with only minor increased overhead; furthermore, the advantages offered by this provision in terms of bits saved to transmit res_scale_value are seen through a typical mix of video content encoded in YCC or RGB and have been found to overcompensate the need to signal mapping changes.
[0144] The specification of the range for α can be done, for example, as follows: In the case of Y′CbCr, the preferred subset can be, for example, (0,±0.125,±0.25,±0.5), while for R′G′B′ it can be (0,±0.5,±1) or (0,0.5,1) or (0,0.5,1,2). To achieve the aforementioned behavior, the range can be specified in the PPS using two syntax elements representing two values. Given the above example and the fact that the prediction is performed with three-point accuracy, i.e. the predicted sample value is multiplied by α and then shifted to the right by 3, the range configuration for Y′CbCr can be transmitted as [−3,3].
[0145] However, with such signaling, only the second and third cases for R´G´B´ can be achieved using [2,3] and [2,4]. To achieve the first example for R´G´B´, the codes have to be separated using an additional syntax. Furthermore, it is sufficient to transmit a delta for the second value with the first value serving as a starting point. For this example, the second R´G´B´ configuration is [2,1] instead of [2,3].
[0146] In the case of prediction from the first chroma component, the range values can be specified separately for each chroma component. Note that this can be done even without support for prediction from the first chroma component.
[0147] Given the constraints specified in PPS, the analysis and reconstruction of the prediction parameter α is modified as follows: For the constraints, i.e., α∈{0, ±0.125, ±0.25, ±0.5, ±1} and for the case without three-point accuracy, the final α F is rearranged as follows, where α P denotes the value parsed from the bitstream: F =1<<α P This is because α F =1<<(α P +α L), where α L indicates the offset for the smallest absolute value when the range is entirely in the positive or negative range.
[0148] Both values are α P is used to derive a limit on the number of bins parsed from the bitstream when is binarized with a truncated unary code. Note that the parsing of the sign may depend on a given range. A better way to exploit this aspect is to code the sign before the absolute value of α. After the coding of the sign, the maximum number of bins parsed from the bitstream can be derived. This case is useful when the range is asymmetric, for example when it is [-1,3].
[0149] Often a different order is desired, e.g. (0,1,0.5) for some R´G´B´ content. Such a reversal can be achieved by simply setting the range values according to [3,2]. In this case the number of bins parsed from the bitstream is still 2 (the absolute difference between the two range values is n=1, and the number of bins is always n+1 in the case of truncated unary codes). The reversal can then be achieved by two methods. The first option introduces a fixed offset, which is equal to twice the current value in case no reversal is desired, and to the maximum representable value in case of reversal. The second and more elegant way of doing this is to extend the range transmitted in the PPS to memory and access the corresponding values through a lookup operation. This approach results in a unified logic and a single path for both cases. For example, the case (0,0.25,0.5,1) is signaled in the PPS by transmitting [1,3], and furthermore, the memory is made with the following entries (0,1,2,3). In the other way, i.e., in the reversed case with the value [3,1] transmitted in the bitstream, the memory is made with the following entries (0,3,2,1). Using such an approach, the final α F is αF =1<<(LUT[α P ]), where LUT[α P ] indicates a lookup operation.
[0150] To explain the most recently mentioned embodiment in more detail, reference is made to Figures 9a, 9b, and 9c, hereinafter referred to simply as Figure 9. Figures 9a, 9b, and 9c illustratively show one multi-component image 102 / 202 in layers, illustratively with a dependent (second) component (108 / 208) followed by a first (reference) component 106 / 206.
[0151] The image 102 / 202 may be part of a video 500 .
[0152] The data stream 104 in which the image 102 / 202 is encoded includes a high-level syntax element structure 510, which relates to at least the entire image 102 / 202 or even to an image sequence from a video 500 including the image 102 / 202. This is illustrated in FIG. 9 using curly brackets 502. Furthermore, the data stream 104 includes a first weight syntax element at the sub-picture level. One such first weight syntax element 514 is exemplarily illustrated in FIG. 9 as relating to an exemplary residual block 440 of the image 102 / 202. The first weight syntax element is for individually setting a first weight, i.e., α, in inter-component predicting the dependent component 108 / 208 based on the co-located portion of the reference component 106 / 206.
[0153] The first weight syntax element is coded using a truncated unary binarization. An example for such a TU2 valuation is shown at 518 in FIG. 9. As shown, such a TU2 valuation consists of a series of bin strings of increasing length. In the above-mentioned manner, a high-level syntax element structure 510 defines how the binarization bin string 518 is mapped to possible values for α. An exemplary set of such possible values for α is shown at 520 in FIG. 9. In practice, the high-level syntax element structure 510 defines a mapping 522 from the set of bin strings 518 to a subset of the set 520. With this approach, it is possible to keep the signaling overhead for switching between different inter-component prediction weights α at sub-picture granularity with low bit consumption, as the sub-picture signaling can simply distinguish between a smaller number of possible values for α.
[0154] As mentioned above, it is possible that the syntax element structure 510 allows a decoder to derive the first and second interval boundaries 524 therefrom. Corresponding syntax elements in the structure 510 can be coded independently / separately or in relation to each other, i.e. differently. The interval boundary values 524 identify the elements from the sequence 520, i.e. with the exponential function mentioned above, which can be implemented with respective bit shifts. By this measure, the interval boundary 524 indicates to the decoder the joint region of the mapping 522.
[0155] As mentioned above, the case of a being zero may be signaled to the decoder separately using the respective zero flag 526. If the zero flag has a first state, the decoder sets a to be zero and skips reading any first weight syntax element 514 for the respective block 440. If the zero flag has another state, the decoder reads the first weight syntax element 514 from the data stream 104 and uses the mapping 522 to determine the actual value of the weight.
[0156] Furthermore, as outlined above, only the absolute part 528 of the first weight syntax element 514 may be coded using a truncated unary binarization, and the sign part of the first weight syntax element 514 may be coded before. In this way, it is possible for the encoder and decoder to appropriately set their length, i.e., the number of bin sequences, of the binarizations 518 for the absolute part 528 so as to determine whether the sign part 530 belongs to those members of the joint region 532 of the mapping 522 from the set 520 where the value α of the block 440 has a positive α value, or to the other member consisting of a negative α value. Of course, the sign part 530 of the first weight syntax element 514 may not be present, and may not be read by the decoder, in the case where the decoder derives from the high-level syntax element structure 510 where the joint region 530 contains only positive α values or only negative α values.
[0157] As will become apparent from the above discussion, the interval boundaries 524 and the order in which they are encoded in the high-level syntax element structure 510 can determine the order in which the absolute portion 528 “traverses” the members of the joint region 532.
[0158] With respect to Fig. 10, a further specific embodiment is shown that summarizes certain aspects already described above. According to the embodiment of Fig. 10, the encoder and decoder commonly subdivide the picture 102 / 202 with respect to all three components as far as hybrid prediction in the predictor 122 / 222 is concerned. In Fig. 10, the components are indicated by "1", "2" and "3", respectively, and are further described below the respectively indicated picture. The subdivision / partitioning of the picture 102 / 202 into prediction blocks 308 can be signaled in the data stream 104 via prediction-related subdivision information 600.
[0159] For each prediction block 308, prediction parameters may be signaled in the data stream 104. These prediction parameters are used for hybrid encoding / decoding of the respective components of the image 102 / 202. The prediction parameters 602 may be signaled in the data stream 104 for each component individually, commonly for all components or specifically for partial components and further component globally. For example, the prediction parameters 602 may distinguish, for example, between the spatially and / or temporally predicted blocks 308, among others, and further, for example, this indication may be common among the components, and the temporal prediction related parameters in the prediction parameters 602 may be signaled in the data stream 104, in particular within the components. With the implementation of Figs. 1 and 2, for example, the predictor 122 / 222 uses the prediction parameters 602 for each component to derive the prediction signal 120 / 220.
[0160] Furthermore, the data stream 104 signals the subdivision / splitting of the image 102 / 202 into residual or transform blocks, here indicated by reference sign 604, according to the embodiment of Fig. 10. Residual / transform related subdivision information in the data stream 104 is indicated with reference sign 606. As mentioned above with respect to Fig. 3, the subdivision / splitting of the image 102 / 202 into prediction blocks 308 on the one hand and residual / transform blocks 604 on the other hand may be at least partially combined with each other in that the subdivision into residual / transform blocks 604 at least partially forms an extension of a hierarchical multi-tree subdivision of the image 102 / 202 into prediction blocks 308 or several hierarchical intermediate subdivisions, for example into coding blocks.
[0161] For each residual / transform block, the data stream 104 may include residual data 6081, 6082, 6083, for example in the form of quantized transform coefficients. Dequantizing and then inverse transforming the residual data 6081-6083 reveals the residual signal in the spatial domain for the respective components, i.e. 6101, 6102 and 6103.
[0162] As shown in Fig. 10, the data stream 104 further includes an inter-component prediction flag 612, which in this example signals globally for the image 102 / 202 whether inter-component prediction is applied / used. If the inter-component prediction flag 612 signals that inter-component prediction is not used, the residual signals 6101-6103 are not combined with each other but are used separately for each component to correct the prediction signal 120 / 220. However, if the inter-component prediction flag 612 signals the use of inter-component prediction, the data stream 104 includes, for each residual / transform block, for each dependent component 2 and 3, a flag 6142, 6143, which signals whether inter-component prediction is applied for each component 2 / 3. When signaled to be applied, the data stream 104 includes inter-component prediction parameters 6162 and 6163, respectively, for each residual / transform block for each component i=2 / 3, corresponding, for example, to α in the description outlined above.
[0163] Thus, for example, if the flag 6142 indicates for the second component that inter-component prediction is used, the inter-component prediction parameters 6162 indicate the weights with which the residual signal 6101 is added to the residual signal 6102 in order to replace the residual signal 6102 with a new residual signal 6102', which is then used instead of the residual signal 6102 to correct the respective prediction signal 1202 / 2 202.
[0164] Similarly, when the flag 6143 indicates the use of inter-component prediction for the respective residual / transform block, the inter-component prediction parameter 6163 indicates a weight α3 by which the residual signal 6101 is added to the residual signal 6103 to result in a new residual signal 6103' that is used to replace the residual signal 6103 and correct the prediction signal 1203 / 2203 of the third component.
[0165] Inter-component prediction parameters 616 2 / 3 The first flag 614 is followed conditionally by 1 / 2 Instead of transmitting the weights α separately, another signaling may be possible. For example, 2 / 3 Its domain of possible values is symmetrically arranged around zero, so that α may be a signed value. 2 / 3 The absolute value of can be used to distinguish between the case of not using inter-component prediction as far as component 2 / 3 is concerned and using inter-component prediction for each component 2 / 3. In particular, when the absolute value is zero, this corresponds to not using inter-component prediction. Then, each parameter α 2 / 3 Any sign flag signaling for α may be suppressed in the data stream 104. As a reminder, according to the embodiment of FIG. 10, inter-component prediction is varied on a per residual / transform block basis as far as component 2 / 3 is concerned, and the variation is in contrast to not using inter-component prediction at all, α 2 / 3 =0 and α 2 / 3 By changing α 2 / 3 ≠ 0).
[0166] TIFF2024167245000013.tif21168
[0167] However, it should be noted that the scope of the flag 612 may be chosen differently. For example, the flag 612 may relate to a smaller unit, such as a slice of the image 102 / 202, or to a larger unit, such as a group of images or a series of images.
[0168] TIFF2024167245000014.tif64170
[0169] That is, the above syntax generates in the data stream 104, for example, one for each second and third component, such as a chroma component, for each residual or transform block of a second image, while the luma component forms the base (first) component.
[0170] As shown previously with respect to Figure 10, the syntax example just outlined corresponds to an alternative of the configuration of Figure 10: log2_res_scale_abs_plus1 signals the absolute value of α, and if the syntax element is zero, this corresponds to inter-component prediction not being used for the respective component c. However, if used, res_scale_sign_flag is also signaled and indicates the sign of α.
[0171] The semantics of the syntax elements presented so far can be provided as follows.
[0172] cross_component_prediction_enabled_flag equal to 1 specifies log2_res_scale_abs_plus1 and res_scale_sign_flag may be present in the transform unit syntax for images that reference a PPS. cross_component_prediction_enabled_flag equal to 0 specifies log2_res_scale_abs_plus1 and res_scale_sign_flag is not present for images that reference a PPS. When not present, the value of cross_component_prediction_enabled_flag is inferred to be equal to 0. It is a bitstream conformance requirement that the value of cross_component_prediction_enabled_flag equal to 0 when ChromaArrayType is not equal to 3.
[0173] log2_res_scale_abs_plus1[c]-1 specifies the base 2 logarithm of the magnitude of the scaling factor ResScaleVal used in cross-component residual prediction. When not present, log2_res_scale_abs_plus1 is inferred to be equal to 0.
[0174] TIFF2024167245000015.tif115170
[0175] In the above, ResScaleVal corresponds to α mentioned above.
[0176] TIFF2024167245000016.tif48170
[0177] A right shift ">>3" corresponds to a division by 8. According to the present example, the α values that can be signaled are {0, ±0.125, ±0.25, ±0.5, ±1}, as already exemplified above.
[0178] log2_res_scale_abs_plus1 may be signaled in the data stream with truncated unary binarization and binary arithmetic coding and binary arithmetic decoding and truncated unary inverse binarization, respectively. The binary arithmetic decoding / encoding may be context adaptive. A context may be selected based on a local neighborhood. For example, a different context may be selected for each bin of the binarization of log2_res_scale_abs_plus1. A different set of contexts may be used for both chroma components. Similarly, res_scale_sign_flag may be signaled in the data stream binary arithmetic coding and binary arithmetic decoding, respectively. The binary arithmetic decoding / encoding may be context adaptive. Furthermore, different contexts may be used for both chroma components. Instead, the same context is used for both chroma components.
[0179] As described, the mapping from log2_res_scale_abs_plus1 to the absolute value of α, i.e., ResScaleVal>>3, may be done arithmetically using bit shift operations, i.e., by an exponential function.
[0180] The signaling of log2_res_scale_abs_plus1 and res_scale_sign_flag for two chroma components may be skipped for a particular residual / transform block if the luma component in the latter is zero. As log2_res_scale_abs_plus1 and res_scale_sign_flag are examples for signaling 614 and 616 in FIG. 10 , this means that the decoder explicitly reads the sub-picture level syntax elements 6142, 6162, 6143, 6163 from the data stream for the currently decoded portion of the second component 208 and further retrieves the component signal 256′ from the spatially corresponding portion 442. 2,3or skip the explicit reading and, optionally, extract the second component signal 256' from the spatially corresponding portion 442. 2,3 , but instead performs a reconstruction of the second component signal 256 2,3 This means that depending on whether the spatially corresponding part 442 of the reconstructed first component signal 2561 is zero, one can possibly check whether it is zero.
[0181] Thus, FIG. 10 illustrates a method for reconstructing a first component signal 6101 associated with the first component 206 from the data stream 104, and further illustrating a reconstructed first component signal 6101 and a correction signal 610 derived from the data stream. 2,3 a second component signal 610' associated with the second (third) component 208 / 210 of the multi-component image 202 from the spatially corresponding portion 442 of 2,3 6 illustrates a decoder configured to decode a multi-component image 202 that spatially samples a scene with respect to different components 206, 208, 210 by reconstructing a portion 440 of the first component signal 6101. The first component signal 6101 is a prediction residual of a temporal, spatial or inter-view prediction of the first component 206 of the multi-component image 202, and the decoder can further perform a temporal, spatial or inter-view prediction of the first component 206 of the multi-component image 202 and further reconstruct the first component 206 of the multi-component image by correcting the temporal, spatial or inter-view prediction of the first component with the reconstructed first component signal 6101. The decoder can further reconstruct the first component 206 of the multi-component image by correcting the temporal, spatial or inter-view prediction of the first component with the reconstructed first component signal 6101 at sub-picture granularity according to a signaling in the data stream. 2,3is configured to adaptively set α2. For this purpose, the decoder is configured to read, with sub-picture granularity, the absolute value of the first weight from the data stream and further to read the sign of the first weight in a manner conditional on whether it is zero. The decoder is configured to skip reading, with sub-picture granularity, the absolute value of the first weight from the data stream and reading the sign of the first weight in a manner conditional on whether it is zero in parts where the first component signal is zero. When reconstructing the second component signal, the decoder adds the spatially corresponding part of the reconstructed first component signal weighted by the first weight (α2) to the correction signal. The addition can be performed in the spatial domain in a sample-wise manner. Alternatively, it is performed in the spectral domain. The encoder executes it in a prediction loop.
[0182] Although some aspects are described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, with blocks or apparatus corresponding to method steps or features of method steps. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of any of the most important method steps may be performed by such an apparatus.
[0183] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or in software. Implementation can be carried out using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM or FLASH memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system so that the respective method is executed. The digital storage medium may therefore be computer readable.
[0184] Some embodiments according to the invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.
[0185] Generally, embodiments of the present invention can be implemented as a computer program product having program code which operates to perform one of the methods when the computer program product is run on a computer. The program code may for example be stored on a machine-readable carrier.
[0186] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.
[0187] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
[0188] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium or computer readable medium) comprising recorded thereon the computer program for performing one of the methods described herein. The data carrier, digital storage medium or recording medium is typically tangible and / or non-transitory.
[0189] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, the data stream or the sequence of signals for example being adapted to be transferred via a data communication connection, for example via the Internet.
[0190] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.
[0191] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.
[0192] Further embodiments according to the invention include an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, include a file server for transferring the computer program to the receiver.
[0193] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.
[0194] The apparatus described herein may be implemented using a hardware apparatus, using a computer, or using a combination of a hardware apparatus and a computer.
[0195] The methods described herein may be performed using a hardware apparatus, using a computer, or using a combination of a hardware apparatus and a computer.
[0196] The above-described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. It is therefore intended that the present invention be limited only by the scope of the claims and not by the specific details shown herein as descriptions and illustrations of the embodiments.
Claims
1. A first component signal (256) associated with a first component (206) of the multi-component image (202) from the data stream (104). 1 ;270 1 ) and reconstruct The reconstructed first component signal (256 1 ;270 1 ) and a correction signal (256 2 ;270 2 ) to a second component signal (256') relating to a second component (208) of the multi-component image (202). 2 ;270' 2 ) (440) 20. A decoder configured to decode a multi-component image (202) that spatially samples a scene with respect to different components (206, 208, 210) by:
2. The decoder comprises: regularly subdividing the multi-component image (202) into treeblocks (302), individually subdividing the treeblocks using a recursive multi-tree subdivision into codeblocks (304), individually subdividing each codeblock using a recursive multi-tree subdivision into prediction blocks (308) and into residual blocks (312), and further subdividing the residual blocks into transform blocks (316); selecting a prediction mode according to the data stream with granularity according to the code block or according to the prediction block; setting prediction parameters according to the data stream at the granularity of the prediction block; A prediction signal (220) is generated using the prediction mode and prediction parameters. 1 , 220 2 , 220 3 ) and The residual signal (256 1 , 256' 2 , 256' 3 ) and further reconstructing the multi-component image (202) by correcting the prediction signal using the residual signal; The present invention relates to a block-based hybrid video decoder, the block-based hybrid video decoder being configured to: The decoder detects the signaling in the data stream (614 2 614 3 2. The decoder of claim 1 , further comprising: a decoder for decoding the first component signal from the spatially corresponding portions of the reconstructed first component signal and the correction signal; and a decoder for decoding the first component signal from the spatially corresponding portions of the reconstructed first component signal from the correction signal without regard to the spatially corresponding portions of the reconstructed first component signal, the decoder being configured to switch at a granularity of the residual block and / or transform block in response to a first decoding step.
3. The decoder, in reconstructing the second component signal, determines whether the spatially corresponding portions (442) of the reconstructed first component signal have first weights (α) that influence the reconstruction of the second component signal at sub-picture granularity. 2 3. A decoder as claimed in claim 1 or 2, adapted to adaptively set .
4. The decoder comprises: Deriving a high level syntax element structure (510) from the data stream, the syntax element structure having at least an image extent; forming a mapping (522) from a set of regions of a predetermined binarization possible bin sequence (518) to a joint region (520) of possible values of the first weight, at least in the image range; and reading a first weight syntax element (514) from the data stream at sub-picture granularity using the predetermined binarization, and further deriving the first weight by subjecting a bin sequence of the first weight syntax element to the mapping.
4. The decoder of claim 3, configured to set the first weight by:
5. 5. The decoder of claim 1, wherein the decoder is configured to, when reconstructing the second component signal (208), adaptively set a second weight by which the correction signal influences the reconstruction of the second component signal at sub-picture granularity.
6. 6. A decoder as claimed in claim 1, wherein the decoder is configured to adaptively set weights of a weighted sum of the correction signal and the spatially corresponding portions (442) of the reconstructed first component signal with sub-picture granularity when reconstructing the second component signal, and to obtain the reconstructed second component signal using the weighted sum as a scalar argument of a scalar function that is constant at least per picture.
7. 7. The decoder of claim 1, wherein the decoder is configured to adaptively set weights of the correction signal, the spatially corresponding portions (442) of the reconstructed first component signal and a constant weighted sum with sub-picture granularity when reconstructing the second component signal, and to obtain the reconstructed second component signal using the weighted sum as a scalar argument of a scalar function that is a constant at least per picture.
8. The decoder is responsive to a first syntax element (612) in the data stream by: enabling the reconstruction of the second component signal based on the reconstructed first component signal, and extracting sub-picture level syntax elements from the data stream when parsing the data stream (614); 2 , 616 2 , 614 3 , 616 3 ), and adapting the reconstruction of the second component signal based on the reconstructed first component signal with sub-picture granularity based on the sub-picture level syntax elements; and 8. A decoder as claimed in claim 1, further comprising: disabling the reconstruction of the second component signal based on the reconstructed first component signal; and altering the parsing of the data stream to address the data stream that does not include the sub-picture level syntax elements.
9. The decoder comprises: enabling the reconstruction of the second component signal based on the reconstructed first component signal; disabling the reconstruction of the second component signal based on the reconstructed first component signal; 9. A decoder as claimed in any preceding claim, arranged to locally switch between in a rear-driven manner.
10. the decoder is configured to reconstruct the first component of the multi-component image such that the first component signal is a prediction residual of a temporal, spatial or inter-view prediction of the first component of the multi-component image, and by performing the temporal, spatial or inter-view prediction of the first component of the multi-component image and correcting the temporal, spatial or inter-view prediction of the first component using the reconstructed first component signal; the decoder is configured to reconstruct the second component of the multi-component image such that the second component signal is a prediction residual of a temporal, spatial or inter-view prediction of the second component of the multi-component image, and by performing the temporal, spatial or inter-view prediction of the second component of the multi-component image, and by correcting the temporal, spatial or inter-view prediction of the multi-component image using the reconstructed second component signal; and The decoder is configured to perform the local switching by locally checking whether the first and second component signals are prediction residuals of a spatial prediction and whether an intra-prediction mode of the spatial prediction matches, or by locally checking whether the first and second component signals are prediction residuals of a spatial prediction and whether an intra-prediction mode of the spatial prediction does not deviate by more than a predetermined amount.
10. A decoder according to claim 9.
11. The decoder is responsive to a second syntax element in the data stream and, in response to the second syntax element, reading sub-picture level syntax elements from the data stream when parsing the data stream, and adapting the reconstruction of the second component signal based on the reconstructed first component signal at sub-picture granularity based on the sub-picture level syntax elements; and A decoder according to any of the preceding claims, which is adapted to perform the reconstruction of the second component signal non-adaptively on the basis of the reconstructed first component signal.
12. The decoder reads sub-picture level syntax elements from the data stream when parsing the data stream, and adapts the reconstruction of the second component signal based on the reconstructed first component signal with sub-picture granularity based on the sub-picture level syntax elements, and when parsing the data stream, checks whether for a currently decoded portion of the second component, the spatially corresponding portion (442) of the reconstructed first component signal is zero, and in response to the check: explicitly reading the sub-picture level syntax elements from the data stream and performing the reconstruction of the second component signal from the spatially corresponding portions of the reconstructed first component signal; or Skip the explicit read A decoder according to any one of claims 1 to 11, arranged to:
13. The reconstructed first component signal (256 1 ;270 1 ) based on spatially corresponding portions of the second component signal (256') associated with the second component (208) of the multi-component image (202). 2 ;270' 2 ) and further comprising a correction signal (256) for correcting said inter-component prediction. 2 ;270 2 ) into the data stream 20. An encoder configured to encode a multi-component image (202) that spatially samples a scene with respect to different components (206, 208, 210) by:
14. A method for decoding a multi-component image (202) that spatially samples a scene with respect to different components (206, 208, 210), comprising: A first component signal (256) relating to a first component (206) of the multi-component image (202) from the data stream (104). 1 ;270 1 ) and The reconstructed first component signal (256 1 ;270 1 ) and a correction signal (256 2 ;270 2 ) to obtain a second component signal (256') related to a second component (208) of the multi-component image (202). 2 ;270' 2 ) (440) of the image; A method comprising:
15. A method for encoding a multi-component image (202) that spatially samples a scene with respect to different components (206, 208, 210), comprising: The reconstructed first component signal (256 1 ;270 1 a second component signal (256') related to a second component (208) of said multi-component image (202) by inter-component prediction based on spatially corresponding portions of said first component signal (256') 2 ;270' 2 ) and a correction signal (256) for correcting said inter-component prediction. 2 ;270 2 ) into the data stream. A method comprising:
Citation Information
Patent Citations
Method and apparatus for providing low resolution update mode for multi-view video coding
JP2012213175A
Coding multiview video plus depth content
US20120229602A1
