Inter-component prediction
By reconstructing the spatial corresponding part of the first component signal in the second component signal of the multi-component image, and adjusting the weights to adaptively set the weights using the sub-image particle size, the redundancy and correlation problems among components are solved, and encoding efficiency and flexibility are improved , achieving higher coding effect.
Patent Information
- Application Number
- CN202110504840.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2014-04-08
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2034-04-08
AI Technical Summary
The prior art is difficult to effectively remove redundancy and correlation between components in multi-component image encoding, resulting in low encoding efficiency and color space transformation may introduce information loss and high bit rate.
By reconstructing the spatial corresponding portion of the first component signal in the second component signal of the multi-component image, adjusting the weights with sub-image granularity, reducing component redundancy and correlation, and turning components on and off in the data stream interdependence to improve coding efficiency.
It achieves higher coding efficiency, reduces signaling overhead, enhances the flexibility and fidelity of the encoding process, and avoids information loss.
Smart Images

Figure CN113259682B_ABST
Abstract
Description
[0001] This application is a divisional application of Chinese national phase application 201480029519.0, whose international application number is PCT / EP2014 / 057090, application date is April 8, 2014, and invention name is “Inter-component Prediction”, all contents of which are incorporated herein by reference. Technical Field
[0002] The present application relates to inter-component prediction in multi-component image coding, for example, between luma and chroma. Background Art
[0003] In image and video signal processing, color information is primarily represented in a color space that is typically composed of three components (e.g., R'G'B' or Y'CbCr). The first component Y' in the case of Y'CbCr is typically referred to as luminance, and the remaining two components (Cb and Cr components or planes in the case of Y'CbCr) are referred to as chrominance. The advantage of the Y'CbCr color space over the R'G'B' color space is primarily the residual characteristics of the chrominance components, i.e., the chrominance components contain less energy or amplitude than the chrominance signals of the absolute color space (e.g., R'G'B'). In particular, for Y'CbCr, the chrominance components represent the grayscale information of an image or video, and the chrominance component Cb represents the difference relative to the blue primary, and Cr represents the difference relative to the red primary.
[0004] In the application space of image and video compression and processing, Y'CbCr signals are preferred over R'G'B' because the color space transformation from R'G'B' to Y'CbCr reduces or removes the correlation between different color components or planes. In addition to the correlation removal, less information needs to be transmitted, so the color transformation is also used as a compression method. This set of preprocessing in correlation removal or reduction can have higher compression efficiency while maintaining or improving complexity in a meaningful amount as an example. Hybrid video compression schemes are usually designed for Y'CbCr input because the correlation between different color components is removed or reduced, and the design of the hybrid compression scheme only needs to consider the separate processing of different components. However, the transformation from R'G'B' to Y'CbCr (and vice versa) is not lossless, so after this color transformation, information may be lost, i.e., sample values that can be used in the original color space. When having R'G'B' input, using a color space that involves a lossless transformation starting from the original color space and returning to the original color space (e.g., Y'CoCg color space) can avoid this problem. However, depending on the application, a fixed color space transform may result in suboptimal results. For image and video compression, fixed color transforms are often suboptimal, used for higher bit rates and artifacts, with high or no correlation between color planes. In the second case, the fixed transform may cause correlations between different signals, and in the first case, the fixed transform may not remove all correlations between different signals. Moreover, due to the comprehensive application of the transform, the correlation may not be completely removed locally or even comprehensively from different components or planes. Another problem caused by color space transforms is the architecture of the image or video encoder. Typically, the optimization process attempts to reduce the cost function, which is usually a distance metric defined on the input color space. In the case of a transformed input signal, it is difficult to achieve the best result for the original input signal due to the additional processing steps. Therefore, the optimal process may result in the lowest cost of the transformed signal (rather than the original input signal). Although the transform is usually linear, the cost calculation in the optimal process usually involves signaling overhead, and then the cost of the final decision is calculated by the Lagrangian formula. This formula may result in different cost values and different optimal decisions. Since modern image and video displays typically use R'G'B' color components for content representation, color transform aspects are particularly important within the domain of color representation. In general, transforms are applied when correlation within or between signals should be removed or reduced. As a result, color space transforms are a special case of more general transform methods.
[0005] It would therefore be advantageous to have upcoming multi-component image coding concepts that are even more efficient, ie, achieve higher bit rates over a wider range of multi-component image content. Summary of the invention
[0006] It is therefore an object of the present invention to provide a multi-component image coding concept with such a higher efficiency.
[0007] This object is achieved by the subject matter of the pending independent claims. The invention is based on the finding that reconstructing a second component signal associated with a second component of a multi-component image from spatially corresponding parts of the reconstructed first component signal and a correction signal originating from a data stream of the second component ensures greater coding efficiency over a wider range of multi-component image contents. By including the spatially corresponding parts of the reconstructed first component signal within the reconstruction of the second component signal, any remaining inter-component redundancy / correlation present (e.g. existing even with a component spatial transformation that may have been performed a priori or introduced by such a component spatial transformation performed a priori) may be easily removed by reducing the inter-component redundancy / correlation of the second component signal.
[0008] According to one embodiment of the present application, the multi-component image codec is understood as a block-based hybrid video codec, which operates in units of code blocks, prediction blocks, residual blocks and transform blocks, and turns inter-component dependencies on and off through corresponding signaling within the data stream, through the granularity of residual blocks and / or transform blocks. Since the amount of inter-component redundancy can vary within the image, the additional overhead for consuming signaling is overcompensated by the coding efficiency gain. According to one embodiment of the present application, the first component signal is the prediction residual of the temporal, spatial or inter-view prediction of the first component of the multi-component image, and the second component signal is the prediction residual of the temporal, spatial or inter-view prediction of the second component of the multi-component image. By this measure, the exploited inter-component dependencies are focused on the remaining inter-component redundancy so that the inter-component prediction can easily display a smoother spatial behavior.
[0009] According to one embodiment of the present application, a first weight (hereinafter denoted as α) is adaptively set by sub-image granularity, by which the spatially corresponding part of the reconstructed first component signal affects the reconstruction of the second component signal. By this measure, intra-frame image changes within inter-component redundancy can be followed more closely. According to one embodiment, a mixture of a high-level syntax element structure and a sub-image granularity first weight syntax element is used to signal the first weight by sub-image granularity, wherein the high-level syntax element structure defines a mapping from a domain set of possible binary strings of a predetermined binarization on the co-domain of possible values of the first weight. By this measure, the overhead of the side information used to control the first weight is kept low. Adjustments can be made adaptively forward. One syntax element can be used for each block, for example, a residual or transform block, which has a limited number of signaling states that symmetrically index one of a plurality of weight values symmetrically distributed around 0 α. In one embodiment, the number of signaling states is uneven, and the number of weight values includes 0, wherein the signaling of 0 is used to signal that inter-component prediction is not used, so as to additionally mark the discard. Further, before the conditionally signaled symbol, the amplitude is signaled, the amplitude is mapped on the number of weight values, and if the amplitude is 0, the symbol is not signaled, so that the signaling cost is further reduced.
[0010] According to one embodiment, in addition to or instead of the adaptive setting of the first weight, a second weight is set by sub-image granularity, by which the correction signal affects the reconstruction of the second component signal. By this measure, the adaptability of the redundancy reduction between components can be further increased. In other words, according to one embodiment, when reconstructing the second component signal, the weight of the weighted sum of the spatial corresponding parts of the correction signal and the reconstructed first component signal can be set by sub-image granularity. The weighted sum can be used as a scalar parameter of a scalar function, and the scalar parameter is constant for at least each image. The weight can be set based on the local neighborhood in a backward-driven manner. The weight can be corrected in a forward-driven manner.
[0011] According to one embodiment, the domain in which the second component signal is reconstructed from the spatial pair portion of the reconstructed first component signal using the correction signal is the spatial domain. Alternatively, the spectral domain is used. Even alternatively, the domain used changes between the spatial and spectral domains. The switching is performed by sub-image granularity. Results show that the ability to switch the domain in which the reconstructed first component signal and the correction signal are combined by sub-image granularity improves coding efficiency. The switching can be performed by a backward adaptive approach or by a forward adaptive approach.
[0012] According to one embodiment, syntax elements within the data stream are used to enable changing the roles of the first and second component signals within components of a multi-component image. The overhead for signalling the syntax elements is low compared to the possible gain in coding efficiency.
[0013] According to one embodiment, the reconstruction of the second component signal is allowed to switch, with sub-image granularity, between a reconstruction based on only the reconstructed first component signal and a reconstruction of the second component signal based on the reconstructed first component signal and further reconstructed component signals of further components of the multi-component image. This possibility increases the flexibility of removing residual redundancy between components of the multi-component image with low additional effort.
[0014] Likewise, according to one embodiment, a first syntax element within the data stream is used in order to enable or disable reconstruction of a second component signal based on the reconstructed first component signal globally or by a larger scope level. If enabled, a sub-picture level syntax element within the data stream is used to reconstruct the second component signal based on the reconstructed first component signal by a sub-picture granularity. By this measure, side information needs to be consumed for sub-picture level syntax elements only in application cases or in multi-component picture content where enabling results in a coding efficiency gain.
[0015] Alternatively, switching between activation and deactivation is performed by means of a backward drive. In this case, the first syntax element does not even need to be present in the data stream. According to one embodiment, local switching is performed, for example, based on checking whether the first and second component signals are prediction residuals of spatial prediction and whether the intra-frame prediction mode of the spatial prediction is consistent or does not deviate by more than a predetermined amount. By this measure, local switching between activation and deactivation does not consume bit rate.
[0016] According to one embodiment, a second syntax element is used within the data stream in order to switch between adaptively reconstructing the second component signal based on the reconstructed first component signal forward adaptively with a sub-picture granularity using a sub-picture level syntax element within the data stream and non-adaptively reconstructing the second component signal based on the reconstructed first component signal. The signaling overhead of the second syntax element is low compared to the possibility of avoiding the overhead of sub-picture level syntax elements for transmitting multi-component picture content (non-adaptively performing the reconstruction is already efficient enough).
[0017] According to one embodiment, the concept of inter-component redundancy reduction is delivered on a three-chrominance component image. According to one embodiment, luminance and two chrominance components are used. The luminance component may be selected as the first component.
[0018] According to one embodiment, a sub-picture level syntax element for adjusting the reconstruction of the second component signal from the reconstructed first component signal is encoded in the data stream using a Golomb Rice code. The bins of the Golomb Rice code can be subjected to binary arithmetic coding. Different backgrounds can be used for different bin positions of the Golomb Rice code.
[0019] According to one embodiment, reconstructing the second component signal from the reconstructed first component signal involves spatially rescaling and / or bit depth accurate mapping on spatially corresponding parts of the reconstructed first component signal. The performance of the spatial rescaling and / or the bit depth accurate mapping may be adjusted in a backward and / or forward adaptive manner. The adjustment of the spatial rescaling may involve selecting a spatial filter. The adjustment of the performance of the bit depth accurate mapping may involve selecting a mapping function.
[0020] According to one embodiment, reconstructing the second component signal from the reconstructed first component signal is performed indirectly by spatially low-pass filtering a version of the reconstructed first component signal.
[0021] Advantageous realizations of embodiments of the application are the subject matter of the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The preferred embodiments of the present application are described below with reference to the drawings, wherein:
[0023] Figure 1 shows a block diagram of an encoder configured to encode a multi-component image according to one embodiment;
[0024] Figure 2 According to one embodiment, Figure 1 A block diagram of a decoder fitted with an encoder;
[0025] Figure 3 schematically illustrates an image and sub-segmentation / division of the image into various blocks of different types according to one embodiment;
[0026] Figure 4 Schematically shows an embodiment of the present application and embedded Figure 1 and Figure 2 The reconstruction module in the encoder and decoder;
[0027] Figure 5 Schematically showing an image with two components, a block predicted between the current component and a spatially corresponding part in the first (base) component;
[0028] Figure 6Schematically illustrates an image with 3 components according to one embodiment in order to illustrate an embodiment in which one component is continuously used as the base (first) component;
[0029] Figure 7 Schematically illustrates an image with 3 components according to one embodiment, wherein each component may alternatively be used as a first (basis) component;
[0030] Figure 8 A possibility is schematically shown according to which the domain of the inter-component prediction is adaptively changed;
[0031] Fig.9A One possibility for signaling inter-component prediction parameters is schematically shown;
[0032] Fig. 9B Shown in more detail Fig.9A the implementation method;
[0033] Fig. 9C Schematically shown for illustration purposes Fig.9A and Fig. 9B the application of the inter-component prediction parameters involved; and
[0034] Fig.10 The forward adaptive inter-component prediction according to one embodiment is schematically illustrated. DETAILED DESCRIPTION
[0035] The following proposed description starts with a detailed description of an encoder and a detailed description of a decoder adapted to the encoder, after which a general implementation is presented.
[0036] Figure 1 An encoder 100 configured to encode a multi-component image 102 into a data stream 104 is shown. Figure 1 The case of a multi-component image 102 including three components 106, 108, and 110 is exemplarily shown, but by passing the present three-component image example on an example having only two components (e.g., components 106 and 108), the description presented below shows the Figure 1 The advantageous implementation of the present application can be easily obtained from the description of Figure 1 Some elements of the encoder 100 shown in FIG. 1 are shown using dashed lines and are heretofore optional functions with respect to the more general implementation set forth further below.
[0037] In principle, Figure 1The encoders of all embodiments described hereinafter may operate with any type of "component". In practice, the multi-component image 102 is an image of a scene that samples the scene spatially with respect to various components 106, 108, and 110 under consistent conditions such as timestamps, viewing conditions, etc. For ease of understanding of the embodiments described further below, however, it is assumed that the components 106, 108, and 110 all relate to textures and are, for example, color components (e.g., luminance and chrominance components) or color components of any other color space.
[0038] Figure 1 The encoder is basically configured to apply block-based hybrid video coding to each component 106, 108 and 110 of the image 102 separately, however, inter-component prediction is used in order to reduce inter-component redundancy. Inter-component prediction involves various aspects, which are outlined in more detail below.
[0039] Thus, for the components 106, 108 and 110, each encoder 100 comprises a series of prediction residual shapers 112 (here exemplarily embodied as subtractors), transformers 114 and quantizers 116, which are connected in series in the order mentioned between the inputs arriving from the respective components 106, 108 and 110 and the respective inputs of a data stream shaper 118, which is configured to multiplex the quantized coefficients and other coding parameters mentioned in more detail below into the data stream 104. While the non-inverting input of the prediction residual shaper 112 is arranged to receive the respective component of the image 102, its inverting (subtracted) input receives the prediction signal 120 from a predictor 122, the input of which is connected to the output of the quantizer 116 via a reconstruction path, which comprises a series of dequantizers 124, retransformers 126 and prediction / residual combiners 128 (here exemplarily embodied as adders). While the output of the prediction / residual recombiner 126 is connected to the input of the prediction 122 and a first input is connected to the output of the retransformer 126 , a further input thereof receives the prediction signal 120 output by the predictor 122 .
[0040] For each component 106 to 110, there are components 112, 114, 116, 124, 126, 128, 122 and 120 in the encoder 100 in a parallel manner to form three separate block-based hybrid video coding paths 1301, 1302 and 1303. The indices 1, 2 and 3 are used to distinguish between different components of the image 102 with index 1 associated with component 106, index 2 associated with component 108 and index 3 associated with component 110.
[0041] exist Figure 1In the example shown in FIG. 1 , component 106 represents a "basic component", which will be more apparent from the description presented below. However, it should be noted, and discussed in more detail below, that the roles among components 106 to 110 can be balanced, because only for easier understanding of the principles of the embodiments of the present application described in more detail below, the selection of Figure 1 Configurations described and shown. In any case, in order to distinguish between components 112, 114, 116, 124, 126, 128, 122, and 120 of the presently described paths 1301 to 1303, indices are also used relative to these components.
[0042] It is evident from the comments presented above that Figure 1 In the example of , components 108 and 110 represent "dependent components" and for this reason, in addition to the components already discussed for the encoding paths 1301 to 1303, further components are connected between these already discussed components. In particular, the spatial domain component inter prediction residual shaper 132 is connected via its non-inverting input and its output between the prediction residual shaper 112 on the one hand and the input of the transformer 114 on the other hand, and likewise the spatial domain component inter prediction and residual recombiner 134 is connected between the retransformer 126 on the one hand and the prediction / residual recombiner 128 on the other hand. Figure 1Also shown is a spatial domain component inter prediction residual shaper 136, which is placed between the transformer 114 and the quantizer 116 via its non-inverting input and its output, a corresponding spatial domain component inter prediction / residual recombiner 134 connected between the dequantizer 124 on the one hand and the retransformer 126 on the other hand, on its further input the pair of spatial domain component inter prediction residual shaper 132 and the corresponding recombiner 134 being fed by a corresponding spatial domain component inter residual predictor 140, and likewise a further input of the spatial domain component inter prediction residual shaper 136 and its associated recombiner 138 being fed with a spectral domain residual prediction signal 142 provided by the spatial domain component inter residual predictor 144. On their input interfaces, the predictors 140 and 144 are connected to any corresponding internal node of the aforementioned coding path. For example, the predictor 1402 is connected to the output of the prediction residual shaper 1121, and the predictor 1442 has an input connected to the output of the dequantizer 1241. Likewise, predictors 1403 and 1443 have inputs connected to the outputs of prediction residual shaper 1121 and dequantizer 1241 respectively, but in addition a further input of predictor 1403 is shown connected to the output of prediction residual shaper 1121 and a further input of predictor 1443 is shown connected to the output of dequantizer 1241. Predictors 140 and 144 are shown as optionally generating prediction parameters for implementing forward adaptive control of the inter-component prediction performed thereby, the parameters output by predictor 140 being indicated by 146 and the parameters output by predictor 144 being indicated by reference numeral 148. Just as quantizer 116 outputs residual signal 150, the prediction parameters 146 and 148 of predictors 140 and 144 are received by data stream shaper 118 which, for example, encodes all such data into data stream 104 in a lossless manner.
[0043] Described in more detail below Figure 1 Before the encoder operation mode, refer to Figure 2 , which shows the corresponding decoder. Essentially, the decoder 200 corresponds to the portion of the encoder 100 extending from the dequantizer 124 to the predictor 122 for each component, and therefore these corresponding parts use the same parameter notation, but increased by 100. More precisely, Figure 2 The decoder 200 is configured to receive the data stream 104 at an input of a data stream extractor 218 of the decoder 200, which is configured to extract the residual signals 1501 to 1503 and all relevant coding parameters including the prediction parameters 146 and 148 from the data stream 104. Figure 1 The encoder 100, Figure 2The decoder 200 is structured into parallel decoding paths 230, one for each component of the multi-component image, the reconstruction of the multi-component image being denoted by reference numeral 200, and the reconstructed components of the multi-component image being output at the output of the respective decoding paths 206, 208 and 210. Due to the quantization, the reconstruction deviates from the original image 102.
[0044] As mentioned above, the decoder paths 2301 to 2303 of the decoder 200 correspond substantially to those of the encoding paths 1301 to 1303 of the encoder 100, including its components 124, 138, 126, 134, 128 and 122. That is, the decoding paths 2302 and 2303 of the "dependent components" 208 and 210 respectively comprise, between the respective output of the data stream extractor 218 on the one hand and the respective output for outputting the respective components 208 and 210, a series of a dequantizer 224, a spectral domain inter-component prediction / residual recombiner 238, an inverse transformer 226, a spatial domain inter-component prediction / residual recombiner 234 and a prediction / residual recombiner 228, connected in the order mentioned therein between the data stream extractor 218 on the one hand and the output of the decoder 200 for outputting the multi-component image 202 on the other hand, wherein the predictor 222 is connected to a feedback loop which returns from the prediction / residual recombiner 228 to its other input. The respective spatial and spectral domain inter-component predictors 240 and 244 feed further inputs 238 and 234. The decoding path 2301 of the "base component" 206 differs in that there are no components 238, 234, 244 and 240. The input of the predictor 2443 is connected to the output of the dequantizer 2242 and its other output is connected to the output of the dequantizer 2241, the first input of the predictor 2403 is connected to the output of the inverse transformer 2262 and its other input is connected to the output of the inverse transformer 2261. The input of the predictor 2442 is connected to the output of the dequantizer 2241 and the input of the predictor 2402 is connected to the output of the inverse transformer 2261.
[0045] It should be noted that, just for ease of illustration, Figure 1 The predictors 1403 and 1402 are shown as being connected to the unquantized residual signal of the underlying component in the spatial domain, but from Figure 2 It is obvious that in order to avoid inconsistent predictions between the encoder and decoder, Figure 1 The predictors 1402 and 1403 may alternatively and advantageously be connected to the outputs of the inverse transformers 1262 and 1261 , respectively, replacing the spectral domain residual signal of the not yet quantized version of the underlying component.
[0046] After describing the structure of encoder 100 and decoder 200, its operation mode is described below. In particular, as described above, encoder 100 and decoder 200 are configured to use hybrid coding to encode / decode each component of a multi-component image. In fact, each component of multi-component image 102 represents a sample array or image, each component spatially sampling the same scene relative to different color components. The spatial resolution of components 106, 108 and 110 (i.e., the spectral resolution at which the scene is sampled relative to the corresponding component) can be different between components.
[0047] As described above, the components of the multi-component image 102 are individually subjected to hybrid encoding / decoding. "Individually" does not necessarily mean that the encoding / decoding of the components is performed completely independently of each other. First, inter-component prediction removes redundancy between components, and in addition, some encoding parameters are selected to be used in common for the components. Figure 3 In the embodiment, for example, it is assumed that the predictors 122 and 222 choose to sub-partition the image 102 into coding blocks as well, but the predictors 122 and 222 are also capable of sub-partitioning the components of the image 102 into coding blocks individually. The sub-partitioning of the image 102 into coding blocks may be fixed or may be signaled within the data stream 104. In the latter case, the sub-partitioning information may be part of the prediction parameters output by the predictor 122 to the data stream shaper 118 and denoted by reference symbol 154, wherein the data stream extractor 218 extracts these prediction parameters, including the sub-partitioning information, and outputs these prediction parameters to the predictor 222.
[0048] Figure 3 The exemplary embodiment shows that the image 102 can be subdivided into coding blocks or code blocks according to a two-stage process, according to which the multi-component image 102 is first regularly subdivided into tree root blocks, Figure 3 In the example, double lines 300 are used to represent the outline of the root block. Subsequently, recursive multitree subdivision is applied to subdivide each root block 302 into code blocks 304. Figure 3 The outline 306 of the code blocks is illustrated using simple continuous lines 306. Thus, the code blocks 304 represent leaf blocks of a recursive multitree subdivision of each root block 302. The above-mentioned share information (generally for components or individually for components) that may be included in the data stream 104 may include multitree subdivision information for each root block 302, used to signal the subdivision of the corresponding root block 302 into code blocks 304, and optionally the subdivision information controls and signals the regular arrangement of the subdivision of the image 102 into the root blocks 302 in rows and columns.
[0049] In units of code blocks 304, predictors 122 and 222 change between a plurality of prediction modes supported by encoder 100 and decoder 200, respectively. For example, predictors 1221 to 1223 select prediction modes for code blocks individually, and item predictor 2223 indicates the selection through prediction parameters 1541 to 1543. Available prediction modes may include temporal and spatial prediction modes. Other prediction modes may also be supported, such as inter-view prediction mode, etc. Using recursive multi-tree subdivision, such as dual-tree subdivision, code blocks may be further subdivided into prediction blocks 308, in Figure 3 The prediction block outline 310 is indicated by a dashed line in FIG. The corresponding recursive sub-partitioning information may be included in the prediction parameters 1541 to 1543 for each code block. In an alternative embodiment, this selection of prediction mode is performed by the granularity of the prediction block. Similarly, each code block 304 may be further sub-partitioned into residual blocks 312 using recursive multi-tree sub-partitioning (e.g., dual-tree sub-partitioning), Figure 3 The dashed line 314 is used to indicate the outline of the residual block. Therefore, each code block 304 is simultaneously divided into a prediction block 308 and a residual block 312. The residual block 312 can be optionally further subdivided into a transform block 316. Figure 3 The outline of the transform block is indicated by a dashed line 318 in FIG. Alternatively, the transform and residual blocks form the same entity, i.e., the residual block is the transform block and vice versa. In other words, according to an alternative embodiment, the residual block is consistent with the transform block. The sub-division information related to the residual block and / or the transform block may or may not be included in the prediction parameters 154.
[0050] Depending on the prediction mode associated with the corresponding code block or prediction block, each prediction block 308 has associated therewith corresponding prediction parameters, which are appropriately selected by the predictors 1221 to 1223, inserted into the parameter information 1541 to 1543, and used by the predictors 2221 to 2223 to control the prediction accordingly within the corresponding prediction block 308. For example, a prediction block 308 having a temporal prediction mode associated therewith may have a motion vector for motion compensated prediction associated therewith and optionally a reference picture index representing a reference picture from which the prediction of the corresponding prediction block 308 is obtained / copied by the displacement represented by the motion vector. A prediction block 308 of a spatial prediction mode may have associated therewith a spatial prediction direction contained in the prediction information 1511 to 1543, which indicates the direction along which the reconstructed surroundings of the corresponding prediction block are spatially extrapolated into the corresponding prediction block.
[0051] Thus, using the prediction mode and the prediction parameters, the predictors 122 and 222 obtain a prediction signal 1201 to 1203 for each component, and for each component, correct this prediction signal using the residual signal 1561 to 1563. These residual signals are encoded using transform coding. That is, the transformers 1141 to 1143 perform a transform, i.e., a spectral decomposition, e.g., DCT, DST, etc., on each transform block 116 individually, and the inverse transformer 226 reverses the transform for the transform block 308 individually, i.e., performs (e.g.) DCT or IDST. That is, as far as the encoder 100 is concerned, the transformer 114 performs a transform on the residual signal that has not yet been quantized formed by the prediction residual shaper 112. The inverse transformers 126 and 226 reverse the spectral decomposition based on the quantized residual signal 150 of the corresponding component in a lossless manner, e.g., using Huffman or arithmetic coding, inserted into the data stream by the data stream shaper 118, and extracted from it by the data stream extractor 218 using (e.g., Huffman or arithmetic decoding).
[0052] However, in order to reduce the data rate of the encoded residual signals 2561 to 2563 (using which the prediction signals 2201 to 2203 are corrected on the prediction / residual recombination units 2281 to 2283), the encoders 100 and 200 support inter-component prediction with respect to the encoding of the residual signals of the components. As described in more detail below, according to an embodiment of the present application, the inter-component prediction for encoding the residual signals can be turned on and off, and / or adaptively adjusted forward and / or backward by the granularity of the residual block and / or the transform block. If turned off, the inter-component prediction signal output by the predictors 140, 144 and 240 and 244 is 0, and the residual signals 2561 to 2563 of all components are derived only from the quantized transform coefficients contained in their corresponding residual signals 1501 to 1503. However, if turned on, the redundancy / correlation between components is removed with respect to the subordinate components, that is, the residual signals 2562 and 2563 are encoded / decoded using inter-component prediction. In the case of inter-component prediction, the basis (first component) used as the source for inter-component prediction is unchanged. The manner in which this is done is outlined below.
[0053] Currently, the description of the inter-component redundancy reduction achieved by predictors 140, 144, 240, and 244 focuses on the inter-component redundancy reduction between components 106 and 108 and 206 and 208, respectively. Subsequently, for ease of understanding, the description is expanded to Figure 1 and Figure 2 The three-component case described in .
[0054] As will be apparent from the description presented below, the embodiments outlined below exploit residual features of, for example, chrominance components, especially when absolute chrominance components or planes are used as input. Figure 1 and Figure 2 In the case of three components, for example, components representing a luminance / chrominance color space, for example, component 106 / 206 may be a luminance component, while components 108 / 208 and 110 / 210 are chrominance components, for example, chrominance components associated with blue and red. By maintaining at least a limited distortion calculation in the original input space, the embodiments outlined below can have higher fidelity. In other words, Figure 1 The encoder and Figure 2 The decoder is able to input the original multi-component image since there is no (for example) color space transformation performed in advance. Figure 1 and Figure 2 The achievable coding efficiency of the embodiments is independent of the color space of the input, i.e., the embodiments for inter-component redundancy removal outlined below are used as an additional correlation removal step for y'CbCr inputs 106 to 110 and, in the case of R'G'B' inputs 106 to 110, as a color transform. In addition, the embodiments outlined further below apply easy reduction locally between components or planes, i.e., the encoder 100 adaptively decides for each region of the image / picture 102 or video whether to apply inter-plane / component prediction involving the residual signal. However, it should be noted again that embodiments of the present invention are not limited to color components or planes, and in particular, the techniques set out in this document can be applied to general planes / components and where the generated residuals of two or more planes consist of correlations. For example, the embodiments described in this application can also be applied to planes from different layers in a scalable video compression scheme. On top of the prediction, the processing order of components or planes can be locally repeated, for example, by signaling the component order in the bitstream through syntax elements. This alternative is also described below.
[0055] Reference Figure 1 and Figure 2 , illustrating the case of a hybrid block-based image and video compression scheme, wherein the prediction performed by the predictor 122 / 222 is based on a record and the transformer 114, quantizer 116 and data stream inserter 118 on the encoding side and the inverse transformer 226, dequantizer 224 and data stream extractor 218 on the decoding side transform the coding, contributing to the prediction error application, i.e. the output of the prediction residual former 112, called residual.
[0056] Please note that the term block is to be understood as describing a generally rectangular shape in the following, ie a block may have a rectangular shape.
[0057] In the implementation described below, the encoder decides to apply inter-plane / component prediction for each prediction block 308 or transform block 304 .
[0058] As an intermediate note, however, it is suggested that embodiments of the present invention are not limited to Figure 1 and Figure 2 The case outlined in , where inter-plane / component prediction is applied to the prediction residuals of the spatial, temporal and / or inter-view prediction signals. In theory, the embodiments proposed in this article can also be transferred to the following case: the inter-plane / component prediction is performed directly on the samples of the components. In other words, when skipping the prediction itself, the original samples of the input signal shall be called or can be regarded as the residual description.
[0059] For each residual block 312 (or rectangular shape), according to one embodiment, a syntax element is transmitted within the data stream 104 and indicates whether plane / component inter-prediction should be used by the predictor 140, 144, 240, 244. In the case of a video compression scheme (e.g., H.265 / EVC), also as in Figure 3 , where the residual is further divided into smaller transform blocks or shapes 216, according to one embodiment, the encoder 100 may transmit the syntax element just mentioned, specifying the use or non-use of inter-plane / component prediction for each transform block 316 or even for a group of transform blocks. It is to be noted that the grouping, i.e. the signaling level, may be selected adaptively and such grouping decision may be transmitted as a further syntax element within the bit / data stream 104, e.g. within a header of the data stream. With the highest signaling level, the use or non-use of inter-plane / component prediction may be signaled for a coding unit or coding block 304, a group of coding blocks 304, or even for a whole image or frame.
[0060] In other words, refer to Figure 4 . Figure 4 Shown in Figure 1 and Figure 2 In the embodiment shown in FIG. 4 , the reconstruction module 400 is embodied in several parts, namely, each pair
[0061] - a spatial domain component inter-predictor 1402 and an associated spatial domain component inter-prediction / residual recombiner 1342, which receives x in the form of a data stream extracted, dequantized and inversely transformed residual signal 1561, receives y in the form of a data stream extracted, dequantized and inversely transformed residual signal 1562, and outputs z as a residual signal 156'2 instead of the residual signal 1562;
[0062] - a spectral domain inter-component predictor 1442 and an associated spectral domain inter-component prediction / residual recombiner 1382, which receives x in the form of a residual signal 1701 extracted from the data stream and dequantized, receives y in the form of a residual signal 1702 extracted from the data stream and dequantized, and outputs z as a residual signal 170'2 replacing the residual signal 1702;
[0063] - a spatial domain component inter-predictor 1403 and an associated spatial domain component inter-prediction / residual recombiner 1343, which receives x in the form of a data stream extracted, dequantized and inversely transformed residual signal 1561 or a data stream extracted, dequantized and inversely transformed residual signal 1562 (or any combination thereof or both), receives y in the form of a data stream extracted, dequantized and inversely transformed residual signal 1563, and outputs z as a residual signal 156'3 instead of the residual signal 1563;
[0064] - a spectral domain inter-component predictor 1443 and an associated spectral domain inter-component prediction / residual recombiner 1383, which receives x in the form of a data stream extracted and dequantized residual signal 1701 or a data stream extracted and dequantized residual signal 1702 (or any combination thereof or both), receives y in the form of a data stream extracted and dequantized residual signal 1703, and outputs z as a residual signal 170'3 instead of the residual signal 1703;
[0065] - a spatial domain component inter-predictor 2402 and an associated spatial domain component inter-prediction / residual recombiner 2342, which receives x in the form of a residual signal 2561 extracted from a data stream, dequantized and inversely transformed, receives y in the form of a residual signal 2562 extracted from a data stream, dequantized and inversely transformed, and outputs z as a residual signal 256'2 instead of the residual signal 2562;
[0066] - a spectral domain inter-component predictor 2442 and an associated spectral domain inter-component prediction / residual recombiner 2382, which receives x in the form of a residual signal 2701 extracted from the data stream and dequantized, receives y in the form of a residual signal 2702 extracted from the data stream and dequantized, and outputs z as a residual signal 170'2 replacing the residual signal 2702;
[0067] - a spatial domain component inter-predictor 2403 and an associated spatial domain component inter-prediction / residual recombiner 2343, which receives x in the form of a data stream extracted, dequantized and inversely transformed residual signal 2561 or a data stream extracted, dequantized and inversely transformed residual signal 2562 (or any combination thereof or both), receives y in the form of a data stream extracted, dequantized and inversely transformed residual signal 2563, and outputs z as a residual signal 256'3 instead of the residual signal 2563;
[0068] - A spectral domain inter-component predictor 2443 and an associated spectral domain inter-component prediction / residual recombineer 2383, which receives x in the form of a residual signal 2701 extracted and dequantized from a data stream or a residual signal 2702 extracted and dequantized from a data stream (or any combination thereof or both), receives y in the form of a residual signal 2703 extracted and dequantized from a data stream, and outputs z as a residual signal 270'3 replacing the residual signal 2703.
[0069] exist Figure 1 and Figure 2 In all of these cases, inter-plane / component prediction is performed as described in more detail later, and in any of these cases, Figure 1 and Figure 2 The implementation method can be Figure 4 Note that only some cases are actually used.
[0070] like Figure 4 As shown in , the plane / component inter-prediction module 400 or reconstruction module 400 has two inputs and one output, and optionally uses prediction parameter signaling between the encoder and decoder sides. The first input 402 is denoted as "x" and represents the incoming reconstructed first component signal, based on which the plane / component inter-redundancy reduction of the second component is performed by the reconstruction module 400. Figure 1 and Figure 2 As shown in , this reconstructed first component signal 402 can be a transmitted residual signal extracted from the data stream of the first component part, co-located in the spatial or spectral domain to the part of the second component that is currently the subject of inter-plane / component redundancy reduction.
[0071] The other input signal 404 of the reconstruction module 400 is denoted "y" and represents, in the same domain (i.e., spectral or spatial domain) as the signal 402, a residual signal of the transmission of part of the second component which is now the subject of the inter-plane / component redundancy reduction of the module 400. The reconstruction module 400 reconstructs a second component signal 406, again denoted "a" in the same domain, which component signal 406 represents the main output of the reconstruction module 400 and which, by replacing x, at least participates in the reconstruction of the subordinate component of the multi-component image. "At least" means that Figure 1 and Figure 2 As shown in FIG. 4 , the component signal 406 output by the reconstruction module 400 may represent a prediction residual and, accordingly, must still be compared with the prediction signal 154 of the corresponding subordinate component i. i Combine.
[0072] As with other modules of the encoder and decoder, the reconstruction module 400 operates on a block basis. For example, the block-based operation may be manifested in a block-by-block adjustment of the inter-component redundancy reduction reversal performed by the reconstruction module 400. Optionally, the "block-by-block adjustment" may involve explicit signaling of the prediction parameters 146 / 148 within the data stream. However, backward adaptive setting of the parameters for controlling the inter-component redundancy reduction is also possible. That is, referring to Figure 4 In the case where the reconstruction module 400 is embedded in the decoder, the prediction parameters enter the restriction module, thereby representing its further input, while in the case where the reconstruction module 400 is embedded in the encoder, the prediction parameters are determined internally in the following illustrative manner, for example, involving solving the LSE optimization problem.
[0073] As described in more detail below, for example, the reconstruction module 400 may operate in such a manner that z may be represented as α, β and γ are possible inter-component prediction parameters. For each of α, β and γ, it is applicable that α is constant and therefore does not change backward or forward adaptively, changes backward adaptively and therefore does not form part of the data stream, or changes forward adaptively and is signaled within the data stream. Figure 5 This is shown for the exemplary case of inter-component prediction from a first component 106 to a second component 108. In the example specifically outlined below, the operation is lifted in a way such that z = ax + y, for example.
[0074] Figure 5 A currently constituted portion or block 440 of the second component 108 is shown. Figure 5 Also shown is a spatially corresponding portion / block 442 of the first component 106, i.e., a portion that is spatially co-located within the image 10. Figure 1 and Figure 2 , the input signals 402 and 404 received by module 400 for components 106 and 108 relative to co-located blocks 440 and 442 represent the residual signals transmitted within the data stream for components 106 and 108. Module 400 calculates a relative to the second component 108. For each block, e.g., block 440, parameters α, β, and γ, or only a subset thereof, are adjusted in a manner further exemplified below.
[0075] In particular, for a specified block, for example, block 440, whether to perform inter-component prediction can be signaled in the data stream through a syntax element. The parameters α, β, and γ in the case of turning on inter-component prediction only represent possible examples. For block 440 to which inter-component prediction is applied, the prediction mode can be signaled in the data stream, the prediction source can be signaled in the data, the prediction domain can be signaled in the data stream, and the parameters related to the above parameters can be signaled in the data stream. From the description presented below, the meaning of "prediction mode", "prediction source", "prediction domain", and "related parameters" is obvious. In the example described so far, inter-component prediction operates on the residual signal. That is, x and y are prediction residuals transmitted in the data stream, and both represent prediction residuals of hybrid prediction. As also described above, in the exemplary case of using transform coding as described above, x and y can be prediction residuals in the spatial domain or in the frequency domain. Applying prediction in the encoder or decoder stage has several advantages. First, no additional memory is usually required, and second, inter-component prediction can be performed locally, that is, from the perspective of the decoder, after the parsing process, no additional intermediate steps are introduced. In order to distinguish the prediction domains, further syntax elements may be transmitted in the bitstream. That is, the following further syntax elements may indicate whether the inter-component prediction domain is the spatial domain or the spectral domain. In the first case, x, y and z are in the spatial domain, and in the latter case, x, y and z are in the spectral domain. Please note that from the decoder's point of view, before the quantization step, the residual is reconstructed by the bitstream and may be different from the residual generated in the encoder. However, in an encoder implementing an embodiment of the present application, the quantized and reconstructed residual is preferably used as a prediction source. Moreover, in the case of skipping the transform stage, the inter-component prediction in the spatial and frequency domains is exactly the same. For this configuration, the signaling of the prediction domain (i.e., spatial or frequency domain) can be skipped.
[0076] As far as the above "prediction mode" is concerned, the prediction mode can be affine, linear, non-linear or more complex. In the first case, as described above, it is possible to record the predictor, i.e., as where z is the reconstructed residual signal or sample, x is a vector containing samples of the predicted source signal, α, β and γ are model parameters, and y is a sample of the current signal, i.e., the past residual signal at the decoder side, and Can be some linear or nonlinear function.
[0077] In order to keep the processing chain as simple as possible, one example configuration might keep the processing of the first component (e.g., the luma component) unchanged and use the luma reconstructed residual signal as a predictor for the component residual signal. This is a possible prediction source configuration, and it is noted that this simple prediction source simplifies the general transform method, where all three components or planes of the input signal are required in order to generate the transformed samples.
[0078] Another possible configuration is to make the prediction source adaptive, i.e., signaling the residual signal used for prediction in all already available or individually reconstructed residual components. Alternatively, the processing order can be changed locally, for example, the second component is reconstructed first and used as the prediction source for the remaining components. This configuration exploits the fact that the Δ between the two components using a bijective (or almost reversible) predictor is the same, but with the inverted sign, the absolute cost for encoding the prediction source is different. Moreover, several prediction sources can be combined. In this case, the combination weights may be transmitted in the bitstream, or estimated backward-driven using statistics of available or correspondingly encoded neighbors.
[0079] The specification of the mode parameters can be performed backward-driven, consisting of backward-driven estimates and forward signaling, or fully forward signaled within the data stream. One example configuration is to use a fixed set of model parameters known in the encoder and decoder, and signal the set index to the decoder for each block or shape. Another configuration is to use a dynamic group or list, where the order of the predictors changes after predicting a number of blocks or shapes. This approach allows for higher local adaptation to the source signal. More detailed examples of prediction modes, prediction sources, prediction domains, and parameter signaling are presented below.
[0080] Regarding the prediction mode, note the following.
[0081] The prediction model can be an affine, linear, nonlinear or more complex function implemented by methods such as spline curves, or support vector regression.
[0082] It is to be noted that the color space transform is most linear, using all available input components. That is, the color space transform tends to map a three-component vector of three color components onto another vector of three other components in another color space. According to one embodiment of the present application, the encoder and decoder may operate independently of the input color space, and therefore, the luminance component may remain unchanged in order to form a prediction source. However, according to an alternative embodiment, the definition of "luminance component" or "first component" may be different for each block, for example, a prediction or transform block (or rectangular shape), that is, for each block or shape, the component used as the "first component" may be adaptively selected. The adjustment may be indicated to the encoder by signaling within the data stream. For example, although Figure 5The case is shown of component 106 forming a "first component", whereas component 108 is an inter-component predicted component, i.e. a second component, in relation to block 440, but the situation may be different for another block of the image 10, where, for example, component 106 is inter-component predicted from component 108. As described above, a block-by-block adjustment of the "prediction source" may be indicated within the data stream. Additionally or alternatively, for each block 440, a syntax element signaling whether prediction is applied or not may be transmitted within the data stream. That is, in the case of continuous use of component 106 as a "first component", such a syntax element is present only for blocks of dependent components, i.e. in Figure 1 and Figure 2 Components 108 and 110 in . In this case, the first component (eg the luminance component) provides the prediction source, ie, in the case of the above-described embodiments, its residual signal.
[0083] If prediction is enabled for a certain block 440, then in addition or alternatively, the prediction mode may be signaled in the data stream for this block 440. Note that for the case of a zero-valued reconstructed first component signal (i.e., a zero-valued residual luminance signal in the case where the prediction residual is used as a basis for inter-component prediction), prediction may be skipped. In this case, in the data stream of the corresponding block 440, the above-mentioned syntax element signaling whether inter-component prediction is applied may be omitted, i.e., not present.
[0084] In case of a combined backward and forward signaling method, parameters derived from the corresponding reconstructed data already encoded can be used as starting parameters. In this case, a delta relative to the selected prediction mode can be transmitted in the data stream. This can be achieved by calculating the best parameters for a fixed or adaptive prediction mode and transmitting the calculated parameters in the bitstream. Another possibility is to transmit a certain delta relative to the starting parameters obtained using the backward drive selection method or always using parameters calculated and selected by the backward drive method alone.
[0085] An example configuration of prediction modes is described below. In this example, the first prediction mode represents α=1, β=0, γ=1 and And the prediction mode of represents the same parameters, except that α=0.5, x=(x0), y=(y0), and the individual elements of x and y are the residual values for the block at the same spatial position within the first and second components (for example, within the luminance component) and the corresponding position within the chrominance component. In other words, according to one embodiment of the present application, for each block 440 to which inter-component prediction is applied, it is signaled in the data stream whether α is equal to the first value (i.e., 1) or the second value (here 0.5). It is not signaled in the data stream, as described above, a mixed backward and forward driving method can be used, or a backward selection can be made adaptively among the available values of α. β and γ simply represent constants.
[0086] Note that, after being predicted by z, a single element in y is replaced. In other words, the reconstruction module 400 receives the correction signal 404 and replaces it with z 406. Note also that, when using this configuration, the inter-component prediction is reduced to an addition operation: the fully reconstructed residual sample value of the first component (α=1) or half of its value (α=0.5) is added to the correction sample value 404. The bisection can be generated by a simple right shift operation. In this case, for example, by implementing a multiplication between x and α in the predictor within the reconstruction module 400, and Figure 4 This configuration can be realized by implementing addition in the adder shown in . In other words, in the specified example, on the decoder side, the operation mode is as follows.
[0087] At the same (spatial or frequency) position in the luma plane, the residual signal for the chroma components of the transform block (or rectangular shape) is subtracted from the reconstructed residual samples. After transformation and quantization, the reconstructed residual is added to the reconstructed residual samples in the same spatial position in the luma plane. From the decoder's point of view, only the latter operation is required. An example configuration of the method involving a backward-driven scheme can be performed as follows. For each block or shape 440, the optimal parameter α is calculated. The starting α is obtained by backward driving from a local neighborhood (e.g., from the previously encoded block (rectangular shape) remaining above). If the syntax element signaling in the bitstream does not transmit further parameters in the bitstream, then the obtained α is used. Otherwise, that is, prediction should be used, but corrected by the syntax element, indicating that α in the bitstream Δ It is also possible to transmit two flags, the first flag indicating whether α should be used, and the second flag indicating whether α is present in the bitstream. Δ .
[0088] Another possible configuration is to increase the granularity of the parameter derivation process. This approach also represents a higher signaling granularity when forward transmission is applied. Otherwise, it represents a higher granularity for the backward drive scheme. In this possible configuration, prediction parameters are obtained for each sample within a transform block (or rectangular shape) or for a group of samples or even for the prediction block itself. Please note that by coupling the signaling level and the transform block (or rectangular shape), a special case is provided. For the current sample or group of samples, a certain amount of samples within the same transform block or prediction block defined by a predefined window are used for parameter derivation. Using the above example using an affine predictor, the parameter α can be obtained from the previously reconstructed sample or group of samples as follows n , where n is the group index.
[0089]
[0090] The parameters for the first sample or the first group of samples may be initialized by some default values or calculated from neighboring blocks or shapes. Another possibility is to transmit the best parameters for the first sample or the first group of samples. In order to use as many of the aforementioned samples as possible, a predefined scanning pattern may be used to map a two-dimensional residual block (or a rectangular shape) to a one-dimensional vector. For example, the sample or sample group may be scanned vertically, horizontally, or in a direction perpendicular to the scanning direction of the transform coefficients. Again, the specific scan may be obtained by a backward drive scheme or signaled in the bitstream.
[0091] Another extension is to combine the examples already shown and transform, using all three available components as input. Here, in this example, the residual signal is transformed using a transform matrix that removes the correlation of the specified transform block. This configuration is useful when the correlation between planes is very large or very small. In the case of an input color space (e.g., R'G'B') or a principal component analysis method, an example configuration uses a Y'CoCg transform. For the latter case, the transform matrix needs to be signaled to the decoder in a forward manner or using predefined settings and rules known at the encoder and decoder to obtain the matrix values. Note that this configuration requires the residual signal for all available components or planes.
[0092] Regarding the prediction domain, note the following.
[0093] As mentioned above, the prediction domain can be the spatial domain, i.e., operating on the residual, or it can be the frequency domain, i.e., operating on the residual after applying a transform such as DCT or DST. Moreover, the domain can be a composite of the two by transmitting the information to the decoder. In addition to the domain used for prediction, parameters related to the domain can be transmitted or obtained by the backward-driven scheme.
[0094] The additional subsampling of the chroma components is related to the domain parameters, i.e. the chroma blocks (or rectangular shapes) are reduced horizontally and / or vertically. In this case, the prediction source may also be downsampled, or a set of prediction modes that take into account different resolutions in the spatial domain and / or frequency domain must be selected. Another possibility is to increase the prediction target so that the size of the prediction source and the prediction target match each other. Reduction further improves the compression efficiency, especially for very flat areas within the image or video. For example, the prediction source contains low and high frequencies, but the chroma blocks (or rectangular shapes) contain only low frequencies. In this example, the subsampling of the prediction source removes the high frequencies, less complex prediction modes can be used, and less information connected with them must be transmitted to the decoder. Please note that the signaling used for reduction may be performed for each transform block or for each prediction block or even for a group of prediction blocks or for the entire image.
[0095] In addition to additional downsampling methods, bit depth adjustments may also be transmitted to the decoder. This occurs when the precision of the samples differs along different components. One possible approach is to reduce or increase the number of bits of the source. Another possible configuration may increase or decrease the bit depth of the target and correct the final result back to the correct bit depth. A further option is to use a set of predictors adapted to different bit depths. This prediction mode takes into account different bit depths of the prediction parameters. The signaling level for bit depth correction can be done for each block or shape or for the entire image or sequence depending on the content.
[0096] As a source of prediction, note the following.
[0097] The prediction source can be the first component or all available components. For each block (or rectangular shape), the prediction source may be signaled by the encoder. Alternatively, the prediction source may be a library containing all possible information, predictions from all available components and the prediction source applied for prediction, whether the prediction or transform block is signaled to the decoder by syntax elements in the bitstream.
[0098] Regarding parameter export, note the following.
[0099] The parameters may be derived illustratively from the causal neighborhood of the block, within the decoder, in order to perform the backward adaptivity mentioned elsewhere, or within the encoder, in order to signal the results within the data stream, in order to drive the decoder in a forward adaptive manner, by solving a least squares error (LSE) optimization problem. LSE may be formulated as:
[0100]
[0101] Minimize this equation as much as possible relative to α. The parameters for this equation have a closed form solution, calculated as:
[0102] Among them, A1 is between z and A2 is the variance of z.
[0103] An integer implementation is implemented for this method to use a lookup table and multiplication instead of division. A2 is scaled down to reduce table size, while A1 is scaled down to avoid multiplication overflow. Only the most significant bit is used. and Avoid A1 and A2. A'1 and A'2 can be obtained as:
[0104] A'=[A>>r A ]<<r A
[0105] in,
[0106] r A =max(bd(A)-n A ,0)
[0107] where bd(A) is the bit depth of the value A computed by log2A.
[0108] Now, α' can be recomputed as:
[0109]
[0110] Now, the division can be done by looking up the table Represented by n table The exponent of this table is calculated as And the table size is n A2 .
[0111] Note that in the above minimization problem, y represents the residual signal, which is transmitted losslessly within the data stream for the corresponding inter-component prediction block 440. In other words, y is the correction signal for the inter-component prediction block 440. This can also be determined repeatedly, and in each iteration, the above LSE optimization problem is solved. In this way, when selecting inter-component prediction parameters (e.g., prediction mode, prediction source, etc.), the encoder can optimally decide whether to perform or not perform inter-component prediction.
[0112] Regarding parameter signaling, note the following.
[0113] The prediction itself may be switchable, and at the beginning of the bitstream, a header flag specifying the use of residual prediction should be transmitted. When prediction is allowed, a syntax element specifying its local use is embedded in the bitstream of the block 440, which, as mentioned above, can be a residual block (or rectangular shape), a transform block (or rectangular shape), or even a group of transform blocks (or rectangular shapes). The first bit may indicate whether prediction is enabled, and the following bits may indicate the prediction mode, prediction source or prediction domain and related parameters. Note that one syntax element can be used to enable or disable prediction of both chrominance components. It is also possible to signal the use of prediction and the prediction mode and source of each second (e.g., chrominance) component separately. Again, in order to achieve a highly suitable residual signal, a classification criterion may be used to signal the prediction mode. For example, the most commonly used prediction mode is 1, then, when the index is 0, the classified list contains mode 1. Then, only one bit needs to signal the most likely mode 1. Moreover, the use of prediction can be restricted. In the case of applying inter-component prediction on the residual, the correlation may be high if the same prediction mode used to generate the residual is used between different color components. This restriction is useful within intra-predicted blocks (or rectangular shapes). For example, only if the same intra-prediction mode used for block (or rectangular shape) 442 (which may be luma) is used for block (or rectangular shape) 440 (which may be chroma), then this inter-component prediction can be applied.
[0114] That is, in the latter case, the block-by-block adaptation of the inter-component prediction process involves checking whether the block 440 is associated with a spatial prediction mode (in terms of the prediction of the predictor 2222) and whether the spatial prediction mode is consistent or does not deviate from the spatial mode by more than a predetermined amount, which is used by the predictor 2221 to predict the co-located block 442. For example, the spatial prediction mode may include a spatial prediction direction along which already reconstructed samples adjacent to the blocks 440 and 442, respectively, are extrapolated into the blocks 440 and 442, respectively, in order to generate the corresponding prediction signals 2202 and 2201, respectively, which are then combined with z and x, respectively.
[0115] In one embodiment, the prediction source is always the luma component. Figure 1 and Figure 2 In this preferred embodiment described in , if the interconnection between components 108 / 208 and 110 / 210 is stopped, the processing of the luminance is unchanged and the resulting reconstructed residual of the luminance plane is used for prediction. As a result, the prediction source is not transmitted in the bitstream.
[0116] In another embodiment, a prediction source is transmitted for one block or shape 440, such as a residual block, a group of residual blocks or shapes, such as a size for applying intra or inter prediction.
[0117] For example, the prediction source is luma for the first chroma component and luma or the first chroma component for the second chroma component. This preferred embodiment is similar to the configuration that allows all available planes to be used as prediction sources and corresponds to Figure 1 and Figure 2 .
[0118] To illustrate the above embodiments, refer to Figure 6 A three-component image 102 / 202 is shown. A first component 106 / 206 is encoded / decoded using hybrid (decoding) coding without any inter-component prediction. As for the second component 108 / 208, the component is divided into blocks 440, Figure 6 In the example, a block is shown, as in Figure 5 The same is true in FIG. In units of these blocks 440, the inter-component prediction is adjusted relative to (for example) α as described above. Similarly, the image 102 / 202 is divided into blocks 450, and with respect to the third component 110 / 210, Figure 6 One such block is representatively shown in . As described in the two alternatives just outlined, using the first component 106 / 206 as a prediction source may entail performing inter-component prediction of block 450, i.e., a co-located portion 452 of the first component 106 / 206. This is indicated using a solid arrow 454. According to the second alternative just described, however, a syntax element 456 within the data stream 104 switches between using the first component as a prediction source (i.e., 454) and using the second component as a prediction source (i.e., inter-component prediction block 450) based on a co-located portion 458 of the second component 108 / 208, as indicated by arrow 460. Figure 6 For example, the first component 106 / 206 may be a luma component, while the other two components 108 / 208 and 110 / 210 may be chroma components.
[0119] It should be noted that, relative to Figure 6 , the division of the second component 108 / 208 into blocks 440 may be signaled within the data stream 104 in a manner independent of or allowing for the division of the deviating component 110 / 210 into blocks 450. Naturally, the division may also be identical and may even be adopted from a signaling form, or for the first component, as described later. Fig.10 The same as the exemplary case in .
[0120] A further embodiment of what has just been outlined above is the possibility that all components of the image 110 / 210 may and alternatively may be used as prediction sources. Figure 7 .exist Figure 7 In the embodiment, all components 106 to 110 / 206 to 210 have associated therewith a common partitioning block 470, Figure 74. The partitioning or sub-partitioning into blocks 470 may be signaled within the data stream 104. As described above, blocks 470 may be residual or transform blocks. However, here, any component may form a "first component", i.e., a prediction source. Syntax elements 472 and data stream 104 indicate to block 470 which of components 106 to 110 / 206 to 210 forms a prediction source for, for example, the other two components. For example, Figure 7 Another block 472 is shown, with its prediction source selected differently, as in Figure 7 In block 470, the first component is used as a prediction source for the other two components, and in the case of block 472, the second component takes on the role of the first component.
[0121] Relative to Figure 6 and Figure 7 It is noted that the syntax elements 456 and 472 indicating the scope of the prediction source may be chosen differently, ie for each block 450 and 470 / 72 individually, or for a group of blocks or even for the entire picture 102 / 202.
[0122] In a further embodiment, the prediction sources may be all available components or a subset of the available components.In this preferred embodiment, the weights of the sources may be signaled to the decoder.
[0123] In a preferred embodiment, the prediction domain is in the spatial domain. In this embodiment, depending on the signaling configuration, it is possible to use the residual of the entire residual block or shape or only a specific part of the residual block or shape. The latter case is provided when prediction is signaled separately for each transform block or shape and further subdivision of the residual block or shape into smaller residual blocks or shapes is allowed.
[0124] In a further embodiment, the prediction domain is in the frequency domain.In this preferred embodiment, the prediction is coupled to the residual block or shape size.
[0125] In a further embodiment, the prediction domain is in the spatial or frequency domain. In this embodiment, the prediction domain is solely defined based on local statistics, either by forward signaling or by backward driven estimation.
[0126] exist Figure 8 The latter case is shown in Figure 8 In FIG. 1 , only two components of the image 102 / 202 are shown as an example. Further, a current inter-component prediction block 440 and its spatial counterpart 442 are shown, as well as a syntax element 490 included in the data stream 104, in order to signal to the block 440 whether the prediction is to be performed in the transform domain or in the spatial domain, as described above (for example) using the function The inter-component prediction of the reconstruction module 400 is implemented by the model of Figure 8 The processing in the decoder is shown for illustrative purposes. First, from the data stream 104, the correction signal of the second component 208 associated with the block 440 is extracted from the data stream, and likewise, the residual signal is extracted from the data stream 104 of the first component 206 as far as the spatial corresponding part 442 is concerned. If the syntax element 490 indicates the use of the spatial domain for the block 440, then these signals are directly input into the reconstruction module 400, and the inter-component prediction output z replaces the correction signal of the block 440 for further reconstruction of the second component 208 of the image 202. However, if the syntax element 490 indicates the use of the spatial domain, then before being processed by the reconstruction module 400, the signals extracted from the data stream 104 for the block 440 and the spatial corresponding part 442 are subjected to an inverse transform 226. In the case of using the spectral domain, x and y are combined spectrum-component by spectrum-component in order to obtain z, and in the case of using the spatial domain, x and y are combined sample-by-sample by the module 400 in order to obtain z.
[0127] Again, the syntax element 490 may signal a domain alone for inter-component prediction for a block 440, a group of blocks 440, or an entire picture or even larger (eg, a group of pictures).
[0128] In a further embodiment, the prediction domains are involved in the prediction process. In this preferred embodiment of the invention, prediction may be first performed in the spatial domain and further prediction applied in the frequency domain, both predictions using different prediction modes and sources.
[0129] In one embodiment, the chroma blocks or shapes may be subsampled horizontally and / or vertically by some factor. In this embodiment, the downscaling factor may be equal to a power of 2. The use of a downsampler is transmitted as a syntax element within the bitstream and the downsampler is fixed.
[0130] In further embodiments, the chroma blocks or shapes may be subsampled horizontally and / or vertically by some factor. The factor may be transmitted in the bitstream, and the downsampler is selected from a set of filters, where the exact filter may be processed by an exponent transmitted in the bitstream.
[0131] In a further embodiment, the selected upsampling filter is transmitted within the bitstream.In this embodiment, the chroma blocks may initially be subsampled, so in order to use prediction with matching block or rectangle sizes, upsampling must be performed prior to prediction.
[0132] In a further embodiment, the selected downsampling filter is transmitted within the bitstream. In this embodiment, the luma is downsampled in order to achieve the same block or rectangle size.
[0133] In one embodiment, when the bit depths of the source and target are different, a syntax element is signaled indicating a bit correction. In this embodiment, the luma precision may be reduced or the chroma precision may be increased in order to have the same bit depth for prediction. In the latter case, the chroma precision is reduced back to the original bit depth.
[0134] In one embodiment, the number of prediction modes is 2, and the set of predictors is defined exactly as in the stated example.
[0135] In a further embodiment, the number of prediction modes is 1, and the configuration is the same as described in the previous embodiment.
[0136] In a further embodiment, the number of predictors is freely adjustable, defining the set of predictors exactly as in the given example. This embodiment is a more general description of the example, α=1 / m, where m>0 indicates the number or mode of predictions. Here, m=0 indicates that predictions should be skipped.
[0137] In a further embodiment, the prediction mode is fixed, ie prediction is always enabled.For this embodiment, it is possible to enable adaptive inter-plane prediction and set the number of predictors equal to zero.
[0138] In a further embodiment, prediction is always applied and prediction parameters, e.g., α, are obtained from neighboring blocks or shapes. In this embodiment, after full reconstruction, the best α for a block or shape is calculated. Within a local neighborhood, the calculated α is used as a parameter for the next block or shape.
[0139] In a further embodiment, syntax elements are transmitted within the bitstream indicating the use of parameters obtained from the local neighborhood.
[0140] In a further embodiment, parameters obtained from the neighborhood are always used. In addition, the delta relative to the best parameters calculated in the encoder may be transmitted in the bitstream.
[0141] In a further embodiment, the backward-driven selection scheme of the parameters is disabled and the best parameters are transmitted within the bitstream.
[0142] In a further embodiment, the use of start α and the presence of Δα are signaled separately within the bitstream.
[0143] In one embodiment, the signaling of prediction modes, prediction sources and prediction parameters is limited to the same regular prediction mode.In this embodiment, information related to inter-plane prediction is transmitted only when the intra prediction mode for chroma components is the same as the mode for luma components.
[0144] In a further embodiment, the block is divided into windows of different sizes and within the block, the parameters of the current window are obtained from a previously reconstructed window. In a further embodiment, the parameters of the first window are obtained from a reconstructed adjacent block.
[0145] In a further embodiment, a syntax element is transmitted within the bitstream indicating the use of parameters obtained from the local neighborhood for the first window.
[0146] In further embodiments, the window may be scanned in a vertical, horizontal, or vertical direction.
[0147] In a further embodiment, the parameters of the current window are obtained from a previous window, wherein the previous window is determined based on the scanning position of the sub-block of the transformation system.
[0148] In a further embodiment, the window scan is limited to one scan direction.
[0149] In a further embodiment, a lookup table and multiplication operations (instead of division) are used to obtain the parameters using integer implementation.
[0150] In one embodiment, a global flag transmitted in the header of the bitstream indicates the use of adaptive inter-plane prediction. In this embodiment, the flag is embedded in the sequence level.
[0151] In a further embodiment, the global flag is transmitted in a header of the bitstream, embedded within the picture parameter level.
[0152] In a further embodiment, the number of predictors is transmitted in a header of the bitstream. In this embodiment, a number of 0 means that prediction is always enabled, and a number not equal to 0 means that the prediction mode is selected adaptively.
[0153] In one embodiment, the set of prediction modes is obtained from the number of prediction modes.
[0154] In a further embodiment, in both the encoder and the decoder, a set of prediction modes is known, specifying all model parameters for the prediction.
[0155] In further embodiments, the prediction modes are either linear or affine.
[0156] In one embodiment, the set of predictors is hybrid, ie contains simple prediction modes that use other planes as prediction sources and contains more complex prediction modes that use all available planes and transform the input residual signal into another component or plane space.
[0157] In one embodiment, the use of prediction is specified for each transform block or shape for each chroma component. In this embodiment, this information may be skipped when the luma component consists of zero-valued residuals at the same spatial position.
[0158] In one embodiment, a truncated unary decomposition,transmission mode is used.In this embodiment, a different background model is assigned to each binary index, however, limited to a specified number, e.g., 3. Moreover, the same background model is used for both chrominance components.
[0159] In a further embodiment, different chroma planes use different sets of background models.
[0160] In a further embodiment, different sets of background models are used for different transform blocks or shape sizes.
[0161] In a further embodiment, the mapping of binary to prediction modes is dynamic or adaptive. In this embodiment, from the decoder's point of view, a decoding mode equal to 0 represents the most commonly used mode until decoding time.
[0162] In a further embodiment, the prediction mode is transmitted as well as the prediction source (if a configuration allowing different prediction sources is used) for the residual block or shape. In this embodiment, different background models may be used for different block or shape sizes.
[0163] The embodiments described next relate in particular to an example of a way of encoding prediction parameters for the so far described "cross-component decorrelation".
[0164] Although not limited thereto, the following description may be considered to represent an alternative, wherein the dependent (second) component is reconstructed based on the reference (first) component signal x and the residual (corrected) signal y by calculating z=αx+y, using z as a prediction of the dependent component signal. For example, the prediction may be applied in the spatial domain. As in the above example, inter-component prediction may be applied in a hybrid coded residual signal, i.e. the first and second component signals may represent a hybrid coded residual signal. However, the following embodiments focus on the signaling of α: this parameter is encoded sub-image by sub-image, e.g. in units of residual blocks into which the multi-component image is sub-divided. The following description of the signaling state of α is preferably also variable, in order to account for the fact that the range of optimal values of α depends on the kind of image content which in turn is larger in range / units than the residual blocks. Therefore, in principle, the details stated below with respect to the transmission of α may also be passed on to the other embodiments outlined above.
[0165] The cross-component decorrelation (CCD) method uses the residual dependencies between different color components and can have higher compression efficiency. Affine models can be used for this method, and the model parameters are transmitted in the bitstream as side information.
[0166] In order to reduce the side information as much as possible, only a limited set of possible parameters are transmitted. For example, a possible CCD implementation in High Efficiency Video Coding (HEVC) may use a linear prediction model instead of an affine prediction model, and only the mode parameters (i.e., the tilt or gradient parameter α) may be restricted to the range from 0 to 1 and non-uniformly quantized. In particular, this limited set of values for α may be α∈{0,±0.125,±0.25,±0.5,±1}.
[0167] The choice of this quantization for the linear prediction model parameters can be based on the fact that the distribution of α is symmetrically concentrated around the value 0, which is used to store the Y'C b C r Natural video content in color space. b C r In CCD, color components are decorrelated using a fixed transformation matrix to convert from R'G'B' before entering the compression stage. Since the full transformation is usually suboptimal, the CCD method can achieve higher compression efficiency by removing the remaining dependencies between different color components.
[0168] However, this assumption does not hold for different types of content, especially natural video content stored in the R'G'B' color space domain. In this case, the gradient parameter α is usually concentrated around the value 1.
[0169] Similar to the case provided above, when the CCD is extended to the first chrominance component as a prediction source, the distribution becomes completely different. Therefore, the CCD parameters can be advantageously adjusted according to the specified content.
[0170] For example, for Y'C b C r , the quantization of α can be set to (0, ±0.125, ±0.25, ±0.5, ±1), while for R'G'B', the quantization of α can be reversed, i.e., (0, ±1, ±0.5, ±0.25, ±0.125). However, allowing different entropy coding paths, raises additional problems. One problem is that the implementation becomes more expensive in terms of area and speed of hardware and software. To avoid this drawback, parameter ranges can be specified at the picture parameter set (PPS) level, where the use of CCD is also indicated.
[0171] That is, the syntax element signaling α is transmitted at a sub-image level / granularity, for example, separately for residual blocks. For example, it can be called res_scale_value. For example, it can be encoded using (truncated) unary binarization combined with binary arithmetic coding of binary strings. Mapping the (non-binarized) value of res_scale_value on α can be achieved so that the mapping uses pps changes, that is, for the entire image or even over a larger range, for example, on a per-image sequence basis. The changes can change the number of representable alpha values, the order of representable alpha values, and the selection of representable alpha values, that is, their actual values. Allowing switching the order among the representable alpha values or limiting the representable alpha values to positive or negative values is just one way to provide content adaptivity, but the implementation outlined further below allows even greater flexibility in changing the mapping from granularly signaled sub-image res_scale_values to alpha values (e.g., changes in size, elements, and order of elements of the mapping's upper domain (representable group of alpha values)) with only slightly increased overhead, and it is shown that the advantage provided by this provision in terms of bit savings for transmitting res_scale_values overcompensates (considered to be more than for a typical mix of video content encoded in YCC or RGB) the necessity for the change in the signaling mapping.
[0172] For example, the specification of the range of α can be performed as follows. b C r In the case of , for example, the preferred subset may be (0, ±0.125, ±0.25, ±0.5), while for R'G'B' it may be (0, ±0.5, ±1) or (0, 0.5, 1) or even (0, 0.5, 1, 2). In order to implement the mentioned behavior, two syntax elements representing two values may be used, specifying the range within the PPS. Considering the above example and the fact that the prediction is done with 3-point precision, i.e., the predicted sample values are multiplied by α and then shifted to the right by 3, it is possible to transmit Y'C b C r The range is configured as [-3,3].
[0173] However, with this signaling, only the second and third cases of R'G'B' can be realized using [2,3] and [2,4]. To realize the first instance of R'G'B', an additional syntax must be used, separating the symbols. Moreover, it is sufficient to transmit the second value of Δ, with the first value used as a starting point. For this example, the second current R'G'B' configuration is [2,1] instead of [2,3].
[0174] In case of prediction of the first chroma component, range values may be specified for each chroma component individually. Note that this may be done even without supporting prediction of the first chroma component.
[0175] Taking into account the restrictions specified in PPS, the analysis and reconstruction of the prediction parameter α are modified as follows. For the case without restrictions, i.e., α∈{0,±0.125,±0.25,±0.5,±1} and 3-point precision, the final α is reconstructed as follows F , where α P Represents the parsed value of the bit stream: α F =1<<α P This is based on α F =1<<(α P +α L ) modified, where α is completely within the positive or negative range. L Indicates the minimum absolute value of the offset.
[0176] In α P When binarizing with truncated unary codes, these two values are used to obtain a limit on the number of bins to be parsed from the bitstream. Note that the parsing of the symbol may depend on the specified range. A better way to use this aspect is to encode the symbol before the absolute value α. After encoding the symbol, the maximum number of bins to be parsed from the bitstream can be obtained. This is useful when the range is asymmetric, i.e., [-1,3].
[0177] Often, a different order is required, such as (0,1,0.5), for example, for R'G'B' content. This reversal can be achieved simply by setting the range value according to [3,2]. In this case, the number of bins parsed from the bitstream is still 2 (the absolute difference between the two range values is n=1, and the number of bins in the case of truncated unary codes is always n+1). The reversal can then be achieved in two ways. The first option is to introduce a fixed offset, which is equal to twice the current value (if reversal is not required) and equal to the maximum representable value in the case of reversal. The second and more elegant way of doing this is to expand the range transmitted to the memory in the PPS and access the corresponding value by a lookup operation. This approach results in a unified logic and a single path for both cases. For example, by transmitting [1,3], the case (0,0.25,0.5,1) is signaled in the PPS, and the memory is generated by the following entry (0,1,2,3). On the other hand, that is, in the reversal case of transmitting the value [3,1] in the bitstream, the memory is generated by the following entry (0,3,2,1). Using this method, the final α F It can be expressed as α F =1<<(LUT[α P ]), where LUT[α P ] indicates a search operation.
[0178] For a more detailed explanation of the recently mentioned aspects, see Fig.9A , Fig. 9B as well as Fig. 9C , in the following we refer only to Fig.9A , a multi-component image 102 / 202 is exemplarily shown in a layered manner: a first (reference) component 106 / 206 is located after a dependent (second) component (108 / 208).
[0179] Image 102 / 202 may be part of video 500 .
[0180] The data stream 104 into which the picture 102 / 202 is encoded comprises a high-level syntax element structure 510 which relates at least to the entire picture 102 / 202 or even in addition thereto to a sequence of pictures in the video 500 comprising the picture 102 / 202. Fig.9A 502. Further, the data stream 104 includes a first weight syntax element at the sub-picture level. One such first weight syntax element 514 is shown in FIG. Fig.9A An exemplary residual block 440 belonging to the picture 102 / 202 is exemplarily shown in FIG. The first weight syntax element is used to individually set a first weight, ie, α, when inter-component predicting the dependent component 108 / 208 based on a co-located portion of the reference component 106 / 206.
[0181] The first weighted syntax element is coded using truncated unary binarization. Fig.9A An example of such TU binarization is shown in FIG. 518. As shown in the figure, such TU binarization consists of a series of binary strings of a larger length. In the above manner, the high-level syntax element structure 510 defines the way to map the binary strings of binarization 518 on possible values α. Fig.9A An exemplary set of such possible values α is shown in 520. In practice, the high-level syntax element structure 510 defines a mapping 522 of the set of binary strings 518 onto a subset of the set 520. By this measure, the signaling overhead for switching between different inter-component prediction weights with sub-image granularity can be kept with lower bit consumption, since the sub-image signaling can only distinguish between a smaller number of possible values α.
[0182] As described above, it is feasible that the syntax element structure 510 allows the decoder to obtain the first and second interval limits 524 therefrom. Corresponding syntax elements within the structure 510 can be encoded independently / separately from each other or relative to each other (i.e., distinctly). The interval limit value 524 identifies an element from the sequence 520, i.e., using the exponential function proposed above, which is implemented using a corresponding displacement. By this measure, the interval limit 524 indicates to the decoder the upper domain of the mapping 522.
[0183] As also described above, the case where α is 0 can be signaled to the decoder using the corresponding zero flag 526. If the zero flag has a first state, the decoder sets α to 0 and skips any first weight syntax element 514 of the corresponding block 440. If the zero flag has another state, the decoder reads the first weight syntax element 514 from the data stream 104 and determines the actual value of the weight using the mapping 522.
[0184] Further, as described above, the absolute part 528 of the first weight syntax element 514 can be encoded using only truncated unary binarization, and the sign part of the first weight syntax element 514 can be pre-coded. Since the sign part 530 determines whether the value α of the block 440 belongs to those elements of the upper domain 532 of the mapping 522 in the group 520 with positive values α or to other parts consisting of negative values α, in this way, the encoder and decoder can appropriately set the length of the binarization 518 (i.e., the number of binary strings) for the absolute path 528. Naturally, if the decoder only originates from the high-level syntax element structure 510 in which the upper domain 530 only includes positive values α or only includes negative values α, then the sign part 530 of the first weight syntax element 514 does not exist and is not read by the decoder.
[0185] It is also apparent from the above discussion that the interval boundaries 524 and the order in which the interval boundaries are encoded in the high-level syntax element structure 510 may determine the order in which the elements of the decision portion 528 “traverse” the upper domain 532 .
[0186] about Fig.10 , shows a further specific implementation of some aspects that have been described above. Fig.10 In embodiments, the encoder and decoder typically partition the image 102 / 202 with respect to all three component sub-partitions with respect to hybrid prediction within the predictor 122 / 222. Fig.10 , the components are represented by "1", "2", and "3", respectively, and are written below the respectively displayed pictures. The sub-division / partitioning of the picture 102 / 202 into prediction blocks 308 can be signaled in the data stream 104 by prediction related sub-division information 600.
[0187] For each prediction block 308, prediction parameters may be signaled within the data stream 104. These prediction parameters are used for each component of the mixed coded / decoded image 102 / 202. The prediction parameters 602 may be signaled within the data stream 104, individually for each component, commonly for all components, or specifically for some components, as well as across the components. For example, the prediction parameters 602 may distinguish in particular between blocks 308 predicted spatially and / or temporally, and for example, temporal prediction related parameters among the prediction parameters 602 may be signaled within the data stream 104, specifically for a component, although this indication is common between components. Using Figure 1 and Figure 2 In an implementation, for example, the predictor 122 / 222 uses the prediction parameters 602 in order to obtain, for each component, a prediction signal 120 / 220.
[0188] Further, according to Fig.10 In an embodiment, the data stream 104 signals a sub-partitioning / division of the picture 102 / 202 into residual or transform blocks, here indicated by reference symbol 604. Reference symbol 606 is used to indicate the residual / transform related sub-partitioning information within the data stream 104. As described above with respect to 3, the sub-partitioning / division of the picture 102 / 202 into prediction blocks 308 on the one hand and into residual / transform blocks 604 on the other hand may be at least partially coupled to each other, since the at least partial division into residual / transform blocks 604 forms an extension of a hierarchical multitree sub-partitioning of the picture 102 / 202 into prediction blocks 308 or some kind of hierarchical intermediate sub-partitioning into (e.g.) coding blocks.
[0189] For each residual / transform block, the data stream 104 may include residual data 6081, 6082, 6083, for example, in the form of quantized transform coefficients. The residual data 6081 to 6083 are dequantized and inversely transformed to display a residual signal in the spatial domain for each component (i.e., 6101, 6102, and 6103).
[0190] like Fig.10As shown in , the data stream 104 further includes an inter-component prediction flag 612, which in this example globally signals whether inter-component prediction is applied / used for the image 102 / 202. If the inter-component prediction flag 612 signals that inter-component prediction is not used, then the residual data 6101 to 6103 are not combined with each other, but are used separately with respect to each component in order to correct the prediction signal 120 / 220. However, if the inter-component prediction flag 612 signals that inter-component prediction is used, then for each residual / transform block, for each dependent component 2 and 3, the data stream 104 includes flags 6142, 6143, which signal whether inter-component prediction is applied for the corresponding component 2 / 3. If the signaling is to be applied, then for the corresponding residual / transform blocks of the corresponding component i=2 / 3, the data stream 104 includes inter-component prediction parameters 6162 and 6163, respectively, which correspond to (for example) α described above.
[0191] Thus, if, for example, flag 6142 indicates to the second component that inter-component prediction is to be used, then inter-component prediction parameter 6162 represents the weight of residual signal 6101 to be added to residual signal 6102 so that a new residual signal 6102' is used to replace residual signal 6102. The new residual signal is then used to replace the residual signal so as to correct the corresponding prediction signal 1202 / 2202.
[0192] Similarly, if mark 6143 indicates that inter-component prediction is used for the corresponding residual / transform block, then the inter-component prediction parameter 6163 indicates the weight of the residual signal 6101 to be added to the residual signal 6103 so as to replace the residual signal and generate a new residual signal 6103', and then the new residual signal is used to correct the prediction signal 1203 / 2203 of the third component.
[0193] Not conditionally predicting parameters across components 616 2 / 3 Previously, the first mark 614 was transmitted separately 1 / 2 , another signaling is also possible. For example, due to the weight α 2 / 3 can be a symbolic value whose domain of possible values is set symmetrically around 0 so that the absolute value α can be used 2 / 3 , in order to distinguish between the case where inter-component prediction is not used (in terms of component 2 / 3) and the case where inter-component prediction is used for the corresponding component 2 / 3. In particular, if the absolute value is 0, then this corresponds to not using inter-component prediction. Then, the corresponding parameter α 2 / 3 Any symbol-marked signaling may be suppressed within data stream 104. As a reminder, according to Fig.10 In an embodiment, for components 2 / 3, inter-component prediction varies on a per residual / transform block basis, including not using inter-component prediction at all. 2 / 3 = 0 and use inter-component prediction, change α2 / 3 (α 2 / 3 ≠0).
[0194] According to a specific syntax example, the inter-component prediction flag 612 may be signaled within a picture parameter group of the data stream 104. The syntax element may be represented as:
[0195] cross_component_prediction_enabled_flag.
[0196] However, it is noted that the scope of the marking 612 may be selected differently. For example, the marking 612 may relate to a smaller unit (e.g., a slice of the image 102 / 202) or a larger unit (e.g., a group of images or a series of images).
[0197] For each residual / transform block, as described above, the following syntax can be used to conditionally signal syntax elements 6142, 6162, 6143, and 6163, parameter c represents the component, so it is assumed that component 2 has one value and component 3 has another value, and for example, parameters x0 and y0 represent the corresponding residual / transform block by (for example) its top left sample.
[0198]
[0199] That is, the above syntax occurs within the data stream 104, for example, twice in each residual or transform block of a picture, once for each of the second and third components (eg, chroma components), while the luma component would form the base (first) component.
[0200] As before relative to Fig.10 As shown, the above syntax example corresponds to Fig.10 Alternative configuration of: log2_res_scale_abs_plus1 signals the absolute value of α, and if the syntax element is 0, this corresponds to inter-component prediction not being used for the corresponding component c. However, if used, res_scale_sign_flag signals and indicates the sign of α.
[0201] The semantics of the currently proposed syntax elements can be provided as follows:
[0202] cross_component_prediction_enabled_flag equal to 1 specifies that for pictures called PPS, log2_res_scale_abs_plus1 and res_scale_sign_flag are present in the transform unit syntax. cross_component_prediction_enabled_flag equal to 0 specifies that for pictures called PPS, log2_res_scale_abs_plus1 and res_scale_sign_flag are not present. In the absence of cross_component_prediction_enabled_flag, the value is inferred to be equal to 0. When ChromaArrayType is not equal to 3, bitstream conformance requires that the value of cross_component_prediction_enabled_flag be equal to 0.
[0203] log2_res_scale_abs_plus1[c] minus 1 specifies the base-2 logarithm of the magnitude of the scale factor ResScaleVal used in cross-component residual prediction. When absent, log2_res_scale_abs_plus1 is inferred to be equal to 0.
[0204] res_scale_sign_flag[c] specifies the sign of the scale factor used in the cross-component residual prediction as follows:
[0205] If res_scale_sign_flag[c] is equal to 0, then the corresponding ResScaleVal has a positive value;
[0206] Otherwise (res_scale_sign_flag[c] is equal to 1) the corresponding ResScaleVal has a negative value.
[0207] The variable ResScaleVal[cIdx][x0][y0] specifies the scale factor used in the cross-component residual prediction. The array index x0, y0 specifies the position (x0, y0) of the top left luma sample of the considered transform block relative to the top left luma sample of the image. The array index cIdx specifies the indicator of the color component; for Cb, it is equal to 1, and for Cr, it is equal to 2.
[0208] The variable ResScaleVal[cIdx][x0][y0] is obtained as follows:
[0209] If log2_res_scale_abs_plus1[cIdx-1] is equal to 0, the following applies:
[0210] ResScaleVal[cIdx][x0][y0]=0;
[0211] Otherwise (log2_res_scale_abs_plus1[cIdx-1] is not equal to 0), the following applies:
[0212] ResScaleVal[cIdx][x0][y0]=(1<<(log2_res_scale_abs_plus1[cIdx-1]-1))*(1-2*res_scale_sign_flag[cIdx-1]).
[0213] In the above, ResScaleVal corresponds to the above α.
[0214] That is, the sample value within the residual / transform block 604 using inter-component prediction, i.e., log2_res_scale_abs_plus1≠0, is calculated at position x, y, i.e., r[x][y], based on the co-located residual sample value ry[x][y] of the first component, for example, according to:
[0215] r[x][y]+=(ResScaleVal[cIdx][xTbY][yTbY]*((r Y [x][y]< <BitDepth C )>>BitDepth Y ))>>3,
[0216] Among them, BitDepth C is 2 / 3 the bit depth of the dependent component, and BitDepth Y is the bit depth of the first component.
[0217] The right shift ">>3" corresponds to a division by 8. According to the present example, as already illustrated above, the values of α that may be signaled are {0, ±0.125, ±0.25, ±0.5, ±1}.
[0218] log2_res_scale_abs_plus1 can be signaled in the data stream using truncated unary binarization and binary arithmetic coding and binary arithmetic decoding and truncated unary debinarization, respectively. Binary arithmetic decoding / encoding can be context-adaptively. The background can be selected based on the local neighborhood. For example, for each binarization of log2_res_scale_abs_plus1, a different background can be selected. Different sets of backgrounds can be used for the two chrominance components. Similarly, res_scale_sign_flag can be signaled in the data stream binary arithmetic coding and binary arithmetic decoding, respectively. Binary arithmetic decoding / encoding can be context-adaptive. Different backgrounds can be used for the two chrominance components. Alternatively, the same background is used for the two chrominance components.
[0219] As described above, the mapping from log2_res_scale_abs_plus1 to the absolute value α (ie, ResScaleVal>>3) may be performed arithmetically using a bit shift operation (ie, by an exponential function).
[0220] If the luma component in a certain residual / transform block is 0, then for a certain residual / transform block, the signaling of log2_res_scale_abs_plus1 and res_scale_sign_flag for these two chroma components can be skipped. Fig.10 In the example of signaling 614 and 616, log2_res_scale_abs_plus1 and res_scale_sign_flag are used, so this means that the decoder may check whether the spatial corresponding part 442 of the reconstructed first component signal 2561 is zero for the currently decoded part of the second component 208, and based on this check, explicitly read the sub-picture level syntax elements 6142, 6162, 6143, 6163 from the data stream and perform reconstruction of the component signal 256' from the spatial corresponding part 442. 2,3 , or skip the explicit reading, and optionally, do not perform the reconstruction of the second component signal 256′ from the spatially corresponding portion 442 2,3 , but keep the second component signal 256 2,3 constant.
[0221] therefore, Fig.10A decoder is illustrated which is configured to decode a multi-component image 202 spatially sampling a scene with respect to different components 206, 208, 210 by: reconstructing a first component signal 6101 associated with the first component 206 from a data stream 104; and reconstructing from a spatially corresponding portion 442 of the reconstructed first component signal 6101 and a correction signal 610 derived from the data stream. 2,3 The second component signal 610′ associated with the second (third) component 208 / 210 of the multi-component image 202 is reconstructed. 2,3 The first component signal 6101 is a prediction residual of the temporal, spatial or inter-view prediction of the first component 206 of the multi-component image 202, and the decoder can perform the temporal, spatial or inter-view prediction of the first component 206 of the multi-component image 202 and reconstruct the first component 206 of the multi-component image by correcting the temporal, spatial or inter-view prediction of the first component using the reconstructed first component signal 6101. The decoder is configured to adaptively set the first weight α according to the signaling in the data stream by sub-image granularity. 2,3 . To this end, the decoder is configured to read the absolute value of the first weight from the data stream with a sub-image granularity and to read the sign of the first weight in a manner conditionally depending on whether the absolute value is 0. The decoder is configured to skip the absolute value of the first weight from the data stream with a sub-image granularity and to skip the sign of the first weight in the part where the first component signal is 0 in a manner conditionally depending on whether the absolute value is 0. When reconstructing the second component signal, the decoder adds the spatially corresponding part of the reconstructed first component signal weighted by the first weight (α2) to the correction signal. The addition can be performed in the spatial domain on a sample-by-sample basis. Alternatively, this is performed in the spectral domain. The encoder performs the same in the prediction loop.
[0222] Although some aspects are described in the context of equipment, it is clear that these aspects also represent the description of the corresponding method, wherein the block or device corresponds to the method step or the feature of the method step. Similarly, the aspects described in the context of the method step also represent the description of the corresponding block or article or feature of the corresponding device. Some or all of the method steps can be performed by (using) hardware devices, such as microprocessors, programmable computers or electronic circuits. In some embodiments, one or more of the most important method steps can be performed by such devices.
[0223] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or in software. Embodiments may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray disc, CD, ROM, PROM, EPROM, EEPROM or FLASH memory, on which electronically readable control signals are stored, which cooperate (or are capable of cooperating) with a programmable computer system to perform the corresponding method. Thus, the digital storage medium may be computer readable.
[0224] Some embodiments according to the invention include a data carrier wave having electronically readable control signals capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.
[0225] Generally, the embodiments of the present invention can be implemented as a computer program product with a program code, which is effectively used to perform a method when the computer program product runs on a computer. For example, the program code can also be stored on a machine-readable carrier.
[0226] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.
[0227] In other words, therefore, one embodiment of the inventive method is a computer program with a program code for performing one of the methods described herein, when the computer program runs on a computer.
[0228] Therefore, a further embodiment of the inventive method is a data carrier (or a digital storage medium or a computer-readable medium) on which is recorded the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium is typically tangible and / or non-transitory.
[0229] Therefore, a further embodiment of the method of the present invention is a data stream or a series of signals representing a computer program for executing a method described herein.For example, the data stream or this series of signals can be configured for transmission via a data communication connection, for example, via the Internet.
[0230] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.
[0231] A further embodiment comprises a computer on which is installed the computer program for performing one of the methods described herein.
[0232] A further embodiment according to the invention comprises an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing a method described herein to a receiver. For example, the receiver may be a computer, a mobile device, a memory device, etc. For example, the apparatus or system may comprise a file server for transmitting the computer program to the receiver.
[0233] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can be coordinated with a microprocessor to perform a method described herein. Typically, these methods are preferably performed by any hardware device.
[0234] The devices described in this document may be implemented using a hardware device, or using a computer, or using a combination of a hardware device and a computer.
[0235] The methods described herein may be performed using a hardware device, or using a computer, or using a combination of a hardware device and a computer.
[0236] According to an embodiment of the present disclosure, the following configuration is provided:
[0237] Item 1. A decoder configured to decode a multi-component image (202) that spatially samples a scene relative to different components (206, 208, 210) by:
[0238] reconstructing a first component signal (2561; 2701) associated with a first component (206) of the multi-component image (202) from the data stream (104);
[0239] A portion (440) of a second component signal (256'2; 270'2) associated with a second component (208) of the multi-component image (202) is reconstructed (400) from a spatially corresponding portion of the reconstructed first component signal (2561; 2701) and a correction signal (2562; 2702) obtained from the data stream.
[0240] Item 2. The decoder of item 1, wherein the decoder is configured as a block-based hybrid video decoder, the hybrid video decoder being configured to:
[0241] Regularly sub-dividing the multi-component image (202) into tree blocks (302), individually sub-dividing the tree blocks into code blocks (304) using recursive multi-tree sub-division, sub-dividing each code block into prediction blocks (308) using recursive multi-tree sub-division, and individually sub-dividing each prediction block into residual blocks (312) using recursive multi-tree sub-division, and sub-dividing the residual blocks into transform blocks (316);
[0242] Selecting a prediction mode according to the data stream with a granularity depending on the code block or depending on the prediction block;
[0243] Setting prediction parameters according to the data stream at a granularity of the prediction block;
[0244] Using the prediction model and the prediction parameters, obtaining a prediction signal (2201, 2202, 2203);
[0245] Obtaining residual signals (2561, 256'2, 256'3) within each residual block by performing inverse transforms individually within the transform blocks; and
[0246] reconstructing the multi-component image (202) by correcting the prediction signal using the residual signal,
[0247] In which, the encoder responds to signaling (6142, 6143) within the data stream so as to switch between reconstructing the second component signal from the spatially corresponding part of the reconstructed first component signal and the correction signal and reconstructing the second component signal from the correction signal without considering the spatially corresponding part of the reconstructed first component signal at the granularity of the residual block and / or the transform block.
[0248] Item 3. A decoder according to Item 1 or 2, wherein the decoder is configured so that the first component signal is a prediction residual of a temporal, spatial or inter-view prediction of the first component (206) of the multi-component image (202), and the decoder is configured to perform the temporal, spatial or inter-view prediction of the first component (206) of the multi-component image (202) and reconstruct the first component (206) of the multi-component image by correcting the temporal, spatial or inter-view prediction of the first component using the reconstructed first component signal.
[0249] Item 4. A decoder according to any of the preceding items, wherein the decoder is configured so that the second component signal is a prediction residual of a temporal, spatial or inter-view prediction of the second component (208) of the multi-component image (202), and the decoder is configured to perform the temporal, spatial or inter-view prediction of the second component (208) of the multi-component image and reconstruct the second component (208) of the multi-component image (202) by correcting the temporal, spatial or inter-view prediction of the multi-component image (202) using the reconstructed second component signal.
[0250] Item 5. A decoder according to any of the preceding items, wherein the decoder is configured to obtain the correction signal (2562) by performing an inverse spectral transform (2262) on spectral coefficients associated with the second component (208) obtained from the data stream (104) so as to obtain the correction signal in the spatial domain.
[0251] Item 6. A decoder according to any of the preceding items, wherein the decoder is configured to adaptively set a first weight (α2) at a sub-image granularity when reconstructing the second component signal, and a spatially corresponding part (442) of the reconstructed first component signal affects the reconstruction of the second component signal with the first weight.
[0252] Item 7. A decoder according to item 6, wherein the decoder is configured to adaptively set the first weight (α2) at the sub-image granularity according to signaling in the data stream.
[0253] Item 8. A decoder according to Item 6 or 7, wherein the decoder is configured to read the absolute value of the first weight from the data stream at the sub-image granularity, and to read the sign of the first weight in a manner conditionally dependent on whether the absolute value of the first weight is 0.
[0254] Item 9. A decoder according to Item 8, wherein the decoder is configured to skip the absolute value of the first weight from the data stream at the sub-image granularity, and to skip the sign of the first weight at a portion where the first component signal is 0 in a manner conditionally dependent on whether the absolute value of the first weight is 0.
[0255] Item 10. A decoder according to any one of Items 6 to 9, wherein the decoder is configured to add a spatial corresponding part (442) of the reconstructed first component signal weighted by the first weight (α2) to the correction signal when reconstructing the second component signal.
[0256] Item 11. The decoder of Item 10, wherein the decoder is configured to perform the adding in the spatial domain on a sample-by-sample basis when reconstructing the second component signal.
[0257] Item 12. A decoder according to any one of Items 6 to 11, wherein the decoder is configured to set the first weight by:
[0258] Obtaining a high-level syntax element structure having at least a picture scope from the data stream (510);
[0259] constructing a mapping (522) from a domain set of predetermined binarized possible binary strings (518) to an upper domain (520) of possible values of the first weight within the at least image range; and
[0260] The first weight is obtained by reading a first weight syntax element from the data stream at a sub-image granularity using the predetermined binarization (514) and subjecting a binary string of the first weight syntax element to the mapping.
[0261] Item 13. A decoder according to item 12, wherein the decoder is configured to obtain a lower bound and an upper bound (524) of an interval of upper domain values within a predetermined set of possible non-zero values of the first weight from the high-level syntax element structure, and when obtaining the first weight, additionally read a zero mark (526) from the data stream at a sub-image granularity, wherein the zero mark (526) indicates whether the first weight is 0, wherein the reading and the reading of the first weight syntax element are performed conditionally based on the zero mark.
[0262] Item 14. A decoder according to Item 12 or 13, wherein the decoder is configured to obtain the sign and absolute value of the lower bound integer value and the sign and absolute value of the upper bound integer value from the high-level syntax element structure when constructing the mapping, apply an integer domain exponential function to the absolute value of the lower bound integer value and the absolute value of the upper bound integer value, and grab the upper domain of possible values of the first weight from the upper domain of the integer domain exponential function that does not include 0.
[0263] Item 15. A decoder according to any one of items 12 to 14, wherein the decoder is configured to use a truncated unary binarization as the predetermined binarization of the absolute value part of the first weight syntax element, and when obtaining the first weight, read the sign part (530) of the first weight syntax element from the data stream before the absolute part (530) of the first weight syntax element, and set the length of the truncated unary binarization of the absolute part of the first weight syntax element based on the sign part of the first weight and the upper domain of the possible values.
[0264] Item 16. A decoder according to any one of items 12 to 15, wherein the decoder is configured to obtain a first interval limit and a second interval limit (524) from the high-level syntax element structure, and the decoder is configured to use a truncated unary binarization of the TU binary string as the predetermined binarization of the absolute value part of the first weight syntax element, and when constructing the mapping, reverse the order in which the TU binary string is mapped to the possible values across the upper domain of the possible values based on a comparison of the first interval limit and the second interval limit.
[0265] Item 17. A decoder according to any of the preceding items, wherein the decoder is configured to adaptively set a second weight at a sub-image granularity when reconstructing the second component signal (208), and the correction signal affects the reconstruction of the second component signal with the second weight.
[0266] Item 18. A decoder according to any of the preceding items, wherein the decoder is configured to adaptively set the weights of the weighted sum of the spatial corresponding parts (442) of the correction signal and the reconstructed first component signal at a sub-image granularity when reconstructing the second component signal, and use the weighted sum as a scalar parameter of a scalar function to obtain the reconstructed second component signal, wherein the scalar function is constant in at least each image.
[0267] Item 19. A decoder according to any of the preceding items, wherein the decoder is configured to adaptively set the weights of the correction signal, the spatial corresponding part (442) of the reconstructed first component signal, and a weighted sum of a constant at a sub-image granularity when reconstructing the second component signal, and use the weighted sum as a scalar parameter of a scalar function to obtain the reconstructed second component signal, wherein the scalar function is constant in at least each image.
[0268] Item 20. A decoder according to item 18 or 19, wherein the decoder is configured to set the weights in a backward-driven manner based on local neighborhoods.
[0269] Item 21. A decoder according to item 18 or 19, wherein the decoder is configured to set the weights in a backward-driven manner based on local neighborhoods and to correct the weights in a forward-driven manner.
[0270] Item 22. A decoder according to Item 20 or 21, wherein the decoder is configured to set the weights in a backward-driven manner based on properties of an already decoded portion of the multi-component image.
[0271] Item 23. A decoder according to any one of items 18 to 22, wherein the decoder is configured to set the weight to a default value at a first spatial granularity by:
[0272] A combined backward and forward adaptive approach, or
[0273] Forward adaptive method,
[0274] Backward adaptive method,
[0275] And the weights are improved in a backward-driven manner based on a local neighborhood at a second spatial granularity finer than the first spatial granularity.
[0276] Item 24. A decoder according to any one of items 16 to 23, wherein the decoder is configured to set the weight to one of m different states based on an m-ary sub-image level syntax element, wherein the decoder is configured to obtain m from a higher-level syntax element (510).
[0277] Item 25. A decoder according to any preceding item, wherein the decoder is configured to adaptively switch between the following steps at a sub-image granularity when reconstructing the second component signal:
[0278] performing an inverse spectral transform on spectral coefficients associated with the second component (208) obtained from the data stream in order to obtain the correction signal (x) in the spatial domain, and reconstructing (400) the second component signal (z) using the correction signal (x) in the spatial domain, and
[0279] The correction signal (x) in the spectral domain is obtained from the data stream, the second component signal (z) is reconstructed (400) in the spectral domain using the obtained correction signal (x) in the spectral domain, and the reconstructed second component signal (z) is subjected to an inverse spectral transform in the spectral domain.
[0280] Item 26. The decoder of Item 25, wherein the decoder is configured to perform adaptive switching (490) in a backward adaptive manner and / or a forward adaptive manner.
[0281] Item 27. A decoder according to any of the preceding items, wherein the decoder is configured to adaptively switch the reconstruction direction of the second component signal at a sub-image granularity between performing reconstruction of the second component signal from a spatially corresponding portion of the reconstructed first component signal and inverting the reconstruction so as to reconstruct the first component signal from the spatially corresponding portion of the reconstructed second component signal.
[0282] Item 28. A decoder according to any one of items 1 to 27, wherein the decoder is configured to adaptively switch the reconstruction direction of the second component signal between performing reconstruction of the second component signal from a spatially corresponding portion of the reconstructed first component signal and reversing the reconstruction so as to reconstruct the first component signal from the spatially corresponding portion of the reconstructed second component signal in response to a syntax element (472) signaling an order between the first component signal and the second component signal.
[0283] Item 29. A decoder according to any of the preceding items, wherein the decoder is configured to adaptively switch the reconstruction of the second component signal at a sub-image granularity between reconstructing the second component signal based only on the reconstructed first component signal and reconstructing the second component signal based on the reconstructed first component signal and a third reconstructed component signal.
[0284] Item 30. A decoder according to any preceding item, wherein the first component and the second component are color components.
[0285] Item 31. A decoder according to any preceding item, wherein the first component is a luma component and the second component is a chroma component.
[0286] Item 32. A decoder according to any preceding item, wherein the decoder is responsive to a first syntax element (612) in the data stream so as to, based on the first syntax element (612),
[0287] enabling reconstruction of the second component signal based on the reconstructed first component signal, reading sub-picture level syntax elements (6142, 6162, 6143, 6163) from the data stream when parsing the data stream, and adjusting reconstruction of the second component signal based on the reconstructed first component signal at a sub-picture granularity based on the sub-picture level syntax elements, and
[0288] Reconstruction of the second component signal is inhibited based on the reconstructed first component signal, and a parsed data stream is altered to process the data stream not including the sub-picture level syntax element.
[0289] Item 33. A decoder according to any preceding item, wherein the decoder is configured to locally switch between the following steps in a backward driven manner:
[0290] enabling reconstruction of the second component signal based on the reconstructed first component signal, and
[0291] Reconstruction of the second component signal is inhibited based on the reconstructed first component signal.
[0292] Item 34. A decoder according to item 33, wherein the decoder is configured to perform local switching in a backward driven manner.
[0293] Item 35. A decoder according to Item 33, wherein:
[0294] the decoder being configured such that the first component signal is a prediction residual of a temporal, spatial or inter-view prediction of the first component of the multi-component image, and the decoder being configured to perform a temporal, spatial or inter-view prediction of the first component of the multi-component image and to reconstruct the first component of the multi-component image by correcting the temporal, spatial or inter-view prediction of the first component using the reconstructed first component signal,
[0295] The decoder is configured to cause the second component signal to be a prediction residual of temporal, spatial or inter-view prediction of the second component of the multi-component image, and the decoder is configured to perform temporal, spatial or inter-view prediction of the second component of the multi-component image and reconstruct the second component of the multi-component image by correcting the temporal, spatial or inter-view prediction of the multi-component image using the reconstructed second component signal, and
[0296] The decoder is configured to perform local switching by locally checking whether the first component signal and the second component signal are prediction residuals of spatial prediction and whether the intra-frame prediction mode of the spatial prediction is consistent, or by locally checking whether the first component signal and the second component signal are prediction residuals of spatial prediction and whether the intra-frame prediction mode of the spatial prediction does not deviate by more than a predetermined amount.
[0297] Item 36. A decoder according to item 33, wherein the decoder is configured to decide to switch locally first in a backward-driven manner and to modify the decision in a forward-adaptive manner in response to signaling in the data stream.
[0298] Item 37. A decoder according to any preceding item, wherein the decoder is responsive to a second syntax element in the data stream so as to, according to the second syntax element,
[0299] while parsing the data stream, reading sub-picture level syntax elements from the data stream, and adjusting reconstruction of the second component signal based on the reconstructed first component signal at a sub-picture granularity based on the sub-picture level syntax elements, and
[0300] Reconstruction of the second component signal is performed non-adaptively based on the reconstructed first component signal.
[0301] Item 38. A decoder according to any of the preceding items, wherein the first component and the second component (206, 208) are two of three color components, and the decoder is configured to also reconstruct a third component signal associated with a third color component (210) of the multi-component image (202) from a spatially corresponding portion of the reconstructed first component signal or the second component signal and a correction signal obtained from a data stream of the third component, wherein the decoder is configured to adaptively perform reconstruction of the second component signal and the third component signal separately at a sub-image level.
[0302] Item 39. A decoder according to any of the preceding items, wherein the first component (206) is a luminance component, the second component (208) is a first chrominance component, and the third component (210) is a second chrominance component, and the decoder is configured to background adaptively entropy decode a first sub-image level syntax element (6142, 6162) and a second sub-image level syntax element (6143, 6163) from a spatially corresponding portion of the reconstructed first component signal or the second component signal using the same background, wherein the first sub-image level syntax element (6142, 6162) is used to adjust the reconstruction of the second component signal associated with the first color component of the multi-component image; and the second sub-image level syntax element (6143, 6163) is used to adjust the reconstruction of the third component signal associated with the first color component of the multi-component image.
[0303] Item 40. A decoder according to any one of items 1 to 38, wherein the first component is luminance, the second component is a first chrominance component, and the third component is a second chrominance component, and the decoder is configured to background adaptively use a separate background to entropy decode a first sub-image level syntax element and a second sub-image level syntax element from a spatially corresponding portion of the reconstructed first component signal or the second component signal, wherein the first sub-image level syntax element is used to adjust the reconstruction of the second component signal associated with the first color component of the multi-component image; and the second sub-image level syntax element is used to adjust the reconstruction of the third component signal associated with the first color component of the multi-component image.
[0304] Item 41. A decoder according to any of the preceding items, wherein the decoder is configured to read sub-image level syntax elements from the data stream when parsing the data stream and adjust the reconstruction of the second component signal based on the reconstructed first component signal at a sub-image granularity based on the sub-image level syntax elements, and to check whether the spatial corresponding part (442) of the reconstructed first component signal is 0 for the currently decoded part of the second component when parsing the data stream, and according to the check
[0305] explicitly reading said sub-picture level syntax elements from said data stream and performing reconstruction of said second component signal from spatially corresponding parts of said reconstructed first component signal, or
[0306] Skip explicit reads.
[0307] Item 42. A decoder according to any of the preceding items, wherein the decoder is configured to entropy decode sub-picture level syntax elements from the data stream using GolombRice codes.
[0308] Item 43. A decoder according to item 42, wherein the decoder is configured to binary arithmetic decode the binary of the Golomb Rice code when entropy decoding the sub-picture level syntax element from the data stream.
[0309] Item 44. A decoder according to item 43, wherein the decoder is configured to binary arithmetic decode the binary of the Golomb Rice code using different backgrounds at different binary positions when entropy decoding the sub-picture level syntax elements from the data stream.
[0310] Item 45. A decoder according to item 43 or 44, wherein the decoder is configured to binary arithmetic decode the binary of the Golomb Rice code without background at binary positions exceeding a predetermined value when entropy decoding the sub-picture level syntax element from the data stream.
[0311] Item 46. A decoder according to any of the preceding items, wherein the decoder is configured to spatially rescale and / or perform bit-depth accurate mapping on a spatially corresponding portion of the reconstructed first component signal when reconstructing the second component signal.
[0312] Item 47. A decoder according to item 46, wherein the decoder is configured to adjust the performance of the spatial rescaling and / or the bit-depth accurate mapping in a backward adaptive manner and / or a forward adaptive manner.
[0313] Clause 48. The decoder of clause 46, wherein the decoder is configured to adjust the spatial rescaling by:
[0314] Select the spatial filter;
[0315] In a backward and / or forward adaptive manner.
[0316] Clause 49. The decoder of clause 48, wherein the decoder is configured to adjust the performance of the bit depth accurate mapping by:
[0317] Select the mapping function;
[0318] In a backward adaptive manner and / or a forward adaptive manner.
[0319] Item 50. A decoder according to any preceding item, wherein the decoder is configured to reconstruct the second component signal from a spatially low-pass filtered version of the reconstructed first component signal when reconstructing the second component signal.
[0320] Item 51. A decoder according to item 50, wherein the decoder is configured to perform reconstruction of the second component signal from the spatially low-pass filtered version of the reconstructed first component signal in a forward adaptive manner and / or in a backward adaptive manner.
[0321] Item 52. A decoder according to Item 50, wherein the decoder is configured to adjust the reconstruction of the second component signal from the spatial low-pass filtered version of the reconstructed first component signal in a forward adaptive manner and / or in a forward adaptive manner by setting a low-pass filter for low-pass filtering.
[0322] Item 53. A decoder according to item 45, wherein the decoder is configured to use staging to perform spatial low pass filtering to produce a reconstructed low pass filtered version of the first component signal.
[0323] Item 54. An encoder configured to encode a multi-component image (202) that spatially samples a scene with respect to different components (206, 208, 210) by:
[0324] A portion (440) of a second component signal (256'2; 270'2) associated with a second component (208) of the multi-component image (202) is encoded (400) by inter-component prediction based on spatially corresponding portions of the reconstructed first component signal (2561; 2701) and inserting a correction signal (2562; 2702) for correcting the inter-component prediction into a data stream.
[0325] Item 55. A method for decoding a multi-component image (202) that spatially samples a scene with respect to different components (206, 208, 210), comprising:
[0326] reconstructing a first component signal (2561; 2701) associated with a first component (206) of the multi-component image (202) from the data stream (104); and
[0327] A portion (440) of a second component signal (256'2; 270'2) associated with a second component (208) of the multi-component image (202) is reconstructed (400) from a spatially corresponding portion of the reconstructed first component signal (2561; 2701) and a correction signal (2562; 2702) obtained from the data stream.
[0328] Item 56. A method for encoding a multi-component image (202) that spatially samples a scene with respect to different components (206, 208, 210), comprising:
[0329] A portion (440) of a second component signal (256'2; 270'2) associated with a second component (208) of the multi-component image (202) is encoded (400) by inter-component prediction based on spatially corresponding portions of the reconstructed first component signal (2561; 2701) and inserting a correction signal (2562; 2702) for correcting the inter-component prediction into a data stream.
[0330] Item 57. A computer program having a program code, which is used to execute the method according to Item 55 or 56 when the program code is run on a computer.
[0331] The above embodiments are merely illustrative of the principles of the present invention. It is to be understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. Therefore, it is intended that the description and explanation of the embodiments herein be limited only by the scope of the forthcoming patent claims and not by the specific details set forth.
Claims
1. A decoder configured to decode a multi-component image (202) in which a scene is spatially sampled with respect to different components (206, 208, 210) by the following steps: reconstructing a first component signal (2561; 2701) associated with a first component (206) of the multi-component image (202) from the data stream (104); reconstructing (400) a portion (440) of a second component signal (256'2; 270'2) associated with a second component (208) of the multi-component image (202) from spatially corresponding portions of the reconstructed first component signal (2561; 2701) and a correction signal (2562; 2702) obtained from the data stream, in, the first component and the second component are color components, and the first component signal is a prediction residual of temporal, spatial or inter-view prediction of the first component of the multi-component image, and the second component signal is a prediction residual of temporal, spatial or inter-view prediction of the second component of the multi-component image, The decoder is configured to adaptively set a first weight at a sub-image granularity according to signaling in the data stream when reconstructing the second component signal, and the spatially corresponding part (442) of the reconstructed first component signal affects the reconstruction of the second component signal with the first weight, wherein the first component is a luminance component and the second component is a chrominance component, wherein the decoder is responsive to an inter-component prediction flag (612) in the data stream so as to decode the data according to the inter-component prediction flag (612), enabling reconstruction of the second component signal based on the reconstructed first component signal, reading sub-picture level syntax elements (6142, 6162, 6143, 6163) from the data stream when parsing the data stream, and adjusting reconstruction of the second component signal based on the reconstructed first component signal at a sub-picture granularity based on the sub-picture level syntax elements, and Reconstruction of the second component signal based on the reconstructed first component signal is disabled, and parsing of the data stream is changed to process the data stream not including the sub-picture level syntax element.
2. The decoder according to claim 1, wherein: The decoder is configured as a block-based hybrid video decoder, the hybrid video decoder being configured to: Regularly subdividing the multi-component image (202) into tree blocks (302), subdividing the tree blocks individually into code blocks (304) using recursive multi-tree subdivision, subdividing each code block into prediction blocks (308) using recursive multi-tree subdivision, and into residual blocks (312) using recursive multi-tree subdivision, and subdividing the residual blocks into transform blocks (316); Selecting a prediction mode according to the data stream with a granularity depending on the code block or depending on the prediction block; Setting prediction parameters according to the data stream at a granularity of the prediction block; Using the prediction model and the prediction parameters, obtaining a prediction signal (2201, 2202, 2203); By performing inverse transform separately in the transform block, a residual signal (2561, 256'2, 256'3) in each residual block is obtained; and reconstructing the multi-component image by correcting the prediction signal using the residual signal (202), In which, the decoder responds to other signaling (6142, 6143) within the data stream so as to switch between reconstructing the second component signal from the spatially corresponding part of the reconstructed first component signal and the correction signal and reconstructing the second component signal from the correction signal without considering the spatially corresponding part of the reconstructed first component signal at the granularity of the residual block and / or the transform block.
3. The decoder according to claim 1 or 2, wherein: The decoder is configured to read an absolute value of a first weight from the data stream at the sub-image granularity, and to read a sign of the first weight in a manner conditionally dependent on whether the absolute value of the first weight is zero.
4. The decoder according to claim 3, wherein: The decoder is configured to skip reading the absolute value of the first weight from the data stream at the sub-image granularity, and skip reading the sign of the first weight at a portion where the first component signal is 0 in a manner conditionally depending on whether the absolute value of the first weight is 0.
5. The decoder according to claim 1, wherein: The decoder is configured to set the first weight by the following steps: Obtaining a high-level syntax element structure having at least a picture scope from the data stream (510); constructing a mapping (522) from a domain set of predetermined binarized possible binary strings (518) to an upper domain (520) of possible values of the first weight using the high-level syntax element structure within the at least image range; and The first weight is obtained by reading a first weight syntax element from the data stream at a sub-image granularity using the predetermined binarization (514), and subjecting a binary string of the first weight syntax element to the mapping.
6. The decoder according to claim 1, wherein: The decoder is configured to adaptively set second weights at a sub-image granularity when reconstructing the second component signal (208), and the correction signal affects the reconstruction of the second component signal with the second weights.
7. The decoder according to claim 1, wherein: The decoder is configured to adaptively switch between the following steps at a sub-image granularity when reconstructing the second component signal: performing an inverse spectral transform on spectral coefficients associated with the second component (208) obtained from the data stream in order to obtain the correction signal in the spatial domain, and reconstructing (400) the second component signal using the correction signal in the spatial domain, and The correction signal in the spectral domain is obtained from the data stream, the second component signal is reconstructed (400) in the spectral domain using the obtained correction signal in the spectral domain, and the reconstructed second component signal is subjected to an inverse spectral transform in the spectral domain.
8. The decoder according to claim 1, wherein: The decoder is configured to locally switch between the following steps in a backward driven manner: enabling reconstruction of the second component signal based on the reconstructed first component signal, and Reconstruction of the second component signal is inhibited based on the reconstructed first component signal.
9. The decoder according to claim 8, wherein: the decoder being configured such that the first component signal is a prediction residual of a temporal or spatial prediction of the first component of the multi-component image, and the decoder being configured to perform a temporal, spatial or inter-view prediction of the first component of the multi-component image and to reconstruct the first component of the multi-component image by correcting the temporal, spatial or inter-view prediction of the first component using the reconstructed first component signal, the decoder is configured to cause the second component signal to be a prediction residual of a temporal, spatial or inter-view prediction of the second component of the multi-component image, and the decoder is configured to perform a temporal, spatial or inter-view prediction of the second component of the multi-component image and to reconstruct the second component of the multi-component image by correcting the temporal or spatial prediction of the multi-component image using the reconstructed second component signal, and The decoder is configured to perform local switching by locally checking whether the first component signal and the second component signal are prediction residuals of spatial prediction and whether the intra-frame prediction mode of the spatial prediction is consistent, or by locally checking whether the first component signal and the second component signal are prediction residuals of spatial prediction and whether the intra-frame prediction mode of the spatial prediction does not deviate by more than a predetermined amount.
10. The decoder according to claim 1, wherein: The first component is a luminance component, the second component is a first chrominance component, and the third component is a second chrominance component, and the decoder is configured to background adaptively use a separated background to entropy decode a first sub-image level syntax element and a second sub-image level syntax element from a spatially corresponding portion of the reconstructed first component signal or the second component signal, wherein the first sub-image level syntax element is used to adjust the reconstruction of the second component signal associated with the first color component of the multi-component image; and the second sub-image level syntax element is used to adjust the reconstruction of the third component signal associated with the first color component of the multi-component image.
11. The decoder according to claim 1, wherein: The decoder is configured to entropy decode sub-picture level syntax elements from the data stream using Golomb Rice codes, using different backgrounds at different binary positions, and binary arithmetic decoding of the Golomb Rice codes without background at binary positions exceeding a predetermined value.
12. An encoder configured to encode a multi-component image (202) in which a scene is spatially sampled relative to different components (206, 208, 210) by the following steps: encoding (400) a portion (440) of a second component signal (256'2; 270'2) associated with a second component (208) of the multi-component image (202) by prediction based on a spatially corresponding portion of the reconstructed first component signal (2561; 2701) and inserting a correction signal (2562; 2702) for correcting the prediction into a data stream, in, the first component and the second component are color components, and the first component signal is a prediction residual of temporal, spatial or inter-view prediction of the first component of the multi-component image, and the second component signal is a prediction residual of temporal, spatial or inter-view prediction of the second component of the multi-component image, The encoder is configured to adaptively set a first weight at a sub-image granularity and a signal in the data stream when encoding the second component signal, and the spatially corresponding part (442) of the reconstructed first component signal affects the reconstruction of the second component signal with the first weight, wherein the first component is a luminance component and the second component is a chrominance component, The encoder is configured to encode an inter-component prediction flag (612) in the data stream so as to indicate, according to the inter-component prediction flag (612): capable of reconstructing the second component signal based on the reconstructed first component signal, capable of reading sub-picture level syntax elements (6142, 6162, 6143, 6163) from the data stream when parsing the data stream, and capable of adjusting reconstruction of the second component signal based on the reconstructed first component signal at a sub-picture granularity based on the sub-picture level syntax elements, and Reconstruction of the second component signal based on the reconstructed first component signal can be disabled, and parsing of the data stream can be altered to process the data stream not including the sub-picture level syntax element.
13. A method for decoding a multi-component image (202) spatially sampling a scene with respect to different components (206, 208, 210), comprising: reconstructing a first component signal (2561; 2701) associated with a first component (206) of the multi-component image (202) from the data stream (104); and reconstructing (400) a portion (440) of a second component signal (256'2; 270'2) associated with a second component (208) of the multi-component image (202) from a spatially corresponding portion of the reconstructed first component signal (2561; 2701) and a correction signal (2562; 2702) obtained from the data stream, wherein the first component and the second component are color components, and the first component signal is a prediction residual of temporal, spatial or inter-view prediction of the first component of the multi-component image, and the second component signal is a prediction residual of temporal, spatial or inter-view prediction of the second component of the multi-component image, wherein reconstructing the second component signal comprises adaptively setting a first weight at a sub-image granularity according to signaling in the data stream, and the spatially corresponding portion (442) of the reconstructed first component signal affects the reconstruction of the second component signal with the first weight, wherein the first component is a luminance component and the second component is a chrominance component, The method is responsive to an inter-component prediction flag (612) in the data stream, so as to determine the inter-component prediction flag (612). enabling reconstruction of the second component signal based on the reconstructed first component signal, reading sub-picture level syntax elements (6142, 6162, 6143, 6163) from the data stream when parsing the data stream, and adjusting reconstruction of the second component signal based on the reconstructed first component signal at a sub-picture granularity based on the sub-picture level syntax elements, and Reconstruction of the second component signal based on the reconstructed first component signal is inhibited, and parsing of the data stream is altered to process the data stream not including the sub-picture level syntax element.
14. A method for encoding a multi-component image (202) spatially sampling a scene with respect to different components (206, 208, 210), comprising: encoding (400) a portion (440) of a second component signal (256'2; 270'2) associated with a second component (208) of the multi-component image (202) by prediction based on a spatially corresponding portion of the reconstructed first component signal (2561; 2701) and inserting a correction signal (2562; 2702) for correcting the prediction into a data stream, wherein the first component and the second component are color components, and the first component signal is a prediction residual of temporal, spatial or inter-view prediction of the first component of the multi-component image, and the second component signal is a prediction residual of temporal, spatial or inter-view prediction of the second component of the multi-component image, wherein encoding the second component signal comprises adaptively setting a first weight at a sub-image granularity and signaling in the data stream, and the spatially corresponding portion (442) of the reconstructed first component signal affects the reconstruction of the second component signal with the first weight, wherein the first component is a luminance component and the second component is a chrominance component, The method comprises encoding an inter-component prediction flag (612) in the data stream so as to indicate, according to the inter-component prediction flag (612): capable of reconstructing the second component signal based on the reconstructed first component signal, capable of reading sub-picture level syntax elements (6142, 6162, 6143, 6163) from the data stream when parsing the data stream, and capable of adjusting reconstruction of the second component signal based on the reconstructed first component signal at a sub-picture granularity based on the sub-picture level syntax elements, and Reconstruction of the second component signal based on the reconstructed first component signal can be disabled, and parsing of the data stream can be altered to process the data stream not including the sub-picture level syntax element.
15. A computer-readable medium having a computer program with program code stored thereon for executing the method according to claim 13 or 14 when the program code is run on a computer.
16. A computer program product comprising a computer program for executing the method according to claim 13 or 14 when the computer program is run on a computer.
Citation Information
Patent Citations
Video encoding / decoding apparatus and method for color image
CN101616329A
Inter-component prediction
CN113259681A
Inter-component prediction
CN113259683A
Inter-component prediction
CN113259684A
