Efficient multiview coding using depth map estimation and updating

Abstract

To provide a device, a method, and a program for realizing a more efficient multi-view coding concept.SOLUTION: In a device 10 for reconfiguration of a multi-view signal, parallax data 60 to be transmitted by a signal in a multi-view data stream 14 for interview prediction is used for initializing depth map estimation for a dependent view 22, and furthermore, this main depth map estimation 64 is continuously updated during further progress of multi-view coding by using movement data and / or the parallax data to be signaled in the multi-view data stream.EFFECT: Continuously updated depth map estimation thus obtained enables execution of various subordinate methods of interview redundancy reduction by a more efficient method without accessing the depth map estimation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to multiview coding using a multiview codec. [Background technology]

[0002] In multiview video coding, two or more views of a video scene, which are captured simultaneously by multiple cameras, are coded into a single bitstream. The main goal of multiview video coding is to provide an end user with an enhanced multimedia experience by providing a 3D viewing impression. When two views are coded, the two reconstructed video sequences can be displayed (with glasses) on a conventional stereo display. However, the use of glasses is mandatory for conventional stereo displays and is often troublesome for the user. Enabling a high-quality stereo viewing impression without glasses is currently a significant challenge in research and development. A promising technology for such autostereoscopic displays is based on lenticular lens systems. In principle, an array of cylindrical lenses is attached to a conventional display such that multiple views of a video scene are displayed simultaneously. Each view is displayed in a small cone, so that each eye of the user sees a different image, and this effect forms a stereo impression without special glasses. However, such autostereoscopic displays typically require 10-30 views of the same video scene (more views may be required when the technology becomes more advanced). More than two views can also be used to provide the user with the possibility to interactively select a viewpoint for the video scene. However, the coding of multiple views of a video scene significantly increases the required bit rate compared to conventional single view (2D) video. Typically, the required bit rate increases approximately linearly with the number of coded views. A concept to reduce the amount of transmitted data for autostereoscopic displays consists in transmitting only a small number of views (perhaps 2-5 views) as well as transmitting a so-called depth map, which represents the depth (distance from real-world objects to the camera) of the image samples for one or more views.Given a small number of coded views with corresponding depth maps, high-quality intermediate views (virtual views between the coded views) and also additional views that extend somewhat to one or both ends of the camera array can be formed at the receiver side by appropriate rendering techniques.

[0003] For both stereo video coding and general-purpose multiview video coding (with or without depth maps), it is important to exploit the interdependencies between different views. Since all views represent the same video scene (from slightly different perspectives), there is considerable interdependence between multiple views. The goal for designing a highly efficient multiview video coding system is to efficiently exploit these interdependencies. In conventional approaches for multiview video coding, e.g., for the multiview video coding (MVC) extension of ITU-T Rec. H.264 | ISO / IEC 14496-10, the only technique that exploits view interdependencies is disparity-compensated prediction of image samples from already coded views, which is conceptually similar to the motion-compensated prediction used in conventional 2D video coding. However, typically only a small subset of image samples is predicted from already coded views, because temporal motion-compensated prediction is often more effective (the similarity between two temporally consecutive images is greater than the similarity between adjacent views at the same time). To further improve the effect of multiview video coding, it is needed to combine efficient motion-compensated prediction and inter-view prediction techniques. One possibility is to reuse the motion data encoded in one view to predict the motion data of other views: since all views represent the same video scene, the motion in one view is connected to the motion in other views based on the geometry of the real-world scene, which can be represented by a depth map and some camera parameters.

[0004] In state-of-the-art image and video coding, a picture or a particular set of sample arrays for a picture is usually decomposed into blocks, which are associated with specific coding parameters. A picture usually consists of multiple sample arrays (luminance and chrominance). In addition, a picture may be associated with additional auxiliary sample arrays, which can specify, for example, transparency information or depth maps. Each picture or sample array is usually decomposed into blocks. The blocks (or the corresponding blocks of the sample arrays) are predicted by inter-picture prediction or intra-picture prediction. The blocks can be of different sizes and can be square or rectangular. The division of a picture into blocks can be fixed by syntax or it can be signaled (at least partially) in the bitstream. Often syntax elements are transmitted that can signal the subdivision for blocks of a given size. Such syntax elements can specify whether and how the blocks should be subdivided into smaller blocks and further associated coding parameters, for example for prediction purposes. For every sample of a block (or a corresponding block of a sample array), the decoding of the associated coding parameters is specified in a specific manner. In an example, every sample in a block is predicted using the same set of prediction parameters, such as, for example, a reference index (identifying a reference picture in a set of already coded pictures), a motion parameter (specifying a measure for the motion of the block between the reference picture and the current picture), a parameter for specifying an interpolation filter, an intra-prediction mode, etc. The motion parameters can be represented by a displacement vector having horizontal and vertical components, or by higher-order motion parameters, such as, for example, affine motion parameters consisting of six components. It is also possible that multiple sets of specific prediction parameters (such as, for example, reference indexes and motion parameters) are associated with a single block.Then, for each set of these specific prediction parameters, a single intermediate prediction signal for the block (or the corresponding block of the sample array) is generated, and further, the final prediction signal is constructed by a combination including superposition of the intermediate prediction signals. The corresponding weighting parameters and potentially constant offsets (which are added to the weighted sum) can also be fixed for a picture or a reference picture or a set of reference pictures, or they can be included in the set of prediction parameters for the corresponding block. The difference between the original blocks (or the corresponding blocks of the sample array) and their prediction signals, also called the residual signal, is usually transformed and further quantized. Often, a two-dimensional transform is applied to the residual signal (or the corresponding sample array for the residual block). For transform coding, the block (or the corresponding block of the sample array) for which a specific set of prediction parameters is used can be further divided before applying the transform. The transform block can be equal to or smaller than the block used for prediction. It is also possible that the transform block includes more than one of the blocks used for prediction. Different transform blocks can be of different sizes, and further, the transform block can represent a square or rectangular block. After transformation, the resulting transform coefficients are quantized to obtain further so-called transform coefficient levels. The transform coefficient levels and the prediction parameters and, if present, the subdivision information are entropy coded.

[0005] State-of-the-art techniques in multiview video coding extend 2D video coding techniques in a straightforward way. Conceptually, two or more video sequences corresponding to different views are coded (or decoded) in parallel. More specifically, for each access unit (or time instant), pictures corresponding to different views are coded in a predefined view order. An MVC bitstream always contains a base view that can be decoded independently of other views. This ensures backward compatibility with the underlying 2D video coding standard / scheme. The bitstream is usually structured in such a way that sub-bitstreams corresponding to the base view (and in addition, sub-bitstreams corresponding to a specific subset of the coded views) can be extracted in a simple way by discarding some packets of all bitstreams. To exploit dependencies between views, pictures of already coded views of the current access unit can be used for prediction of blocks of the current view. This prediction is often called disparity-compensated prediction or inter-view prediction. It is basically identical to motion compensated prediction in conventional 2D video coding, the only difference being that the reference pictures represent pictures of different views (i.e., at the same time) within the current access unit, and furthermore do not represent pictures of the same view at different times. To incorporate inter-view prediction in the design of the underlying 2D video coding scheme, one or more reference picture lists are constructed for each picture. For the base view (independently decodable view), only conventional temporal reference pictures are inserted into the reference picture list. However, for all other views, inter-view reference pictures can be inserted into the reference picture list in addition to (or instead of) the temporal reference pictures. Either picture is inserted into the reference picture list determined by the video coding standard / scheme and / or signaled in the bitstream (e.g., in the parameter set and / or slice header).Then, whether a temporal or inter-view reference picture is selected for a particular block of the current view is signaled by encoding (or inferring) the reference picture index, i.e., inter-view reference pictures are used in exactly the same way as conventional temporal reference pictures, with only a slight extension to the construction of the reference picture list.

[0006] TIFF0007661385000001.tif211170

[0007] Conceptually, a similar concept as for SVC can also be used in multi-view video coding. Multiple cameras capture the same video scene from different viewpoints. However, when real-world objects move in the scene, the motion parameters in different captured views are not independent. However, in contrast to scalable coding, the object positions are the same in all layers (layers just represent different resolutions or different qualities of the same captured video), and the interrelationships of projected motions are more complex and even depend on several camera parameters and 3D relationships in real-world scenes. However, given all relevant camera parameters (e.g. focal length, camera distance and camera optical axis direction, etc.) and the projected object point distance (depth map), the motion in a particular view can be derived based on the motion of another view. In general, to code a video sequence or view, it is not necessary to know the exact motion of the object points, instead simple parameters such as motion vectors for a block of samples are sufficient. In this spirit, the relationship of the motion parameters between different views can also be simplified to some extent.

[0008] However, advantageously, the coding order in coding the multi-view signal is selected such that the pictures conveying the texture of each view are coded before the corresponding depth map in order to be able to efficiently utilize the features known from coding / decoding the pictures when coding / decoding the depth map. In other words, it turns out that the removal of redundancy between depth maps and related pictures is more efficient in the case of a coding order leading from pictures to depth maps rather than vice versa. However, following this coding order leads to a lack of depth map information available at the decoding side when the decoder decodes the pictures of the dependent view, because the depth map has not yet been reconstructed. Unfortunately, the coding parameters of the reference view may not be efficiently utilized. The situation is even more serious in the case of a multi-view data stream in which no depth map of the view exists. Summary of the Invention [Problem to be solved by the invention]

[0009] The aim of the present invention is to provide a more efficient multi-view coding concept. [Means for solving the problem]

[0010] This object is achieved by the subject matter of the pending independent claims.

[0011] According to a first aspect of the present invention, an idea is utilized whereby the absence of a depth map for a current picture of a reference view, due to its transmission not expected anyway, or due to a suitable coding order between a texture / picture and its depth map, or due to expected discarding of depth data from the bitstream during transmission or decoding, can be adequately addressed to reduce inter-view redundancy by estimating depth maps for pictures of the reference and dependent views and further updating it with motion and / or disparity data signaled in the multiview data stream. In particular, virtually all multiview data streams have a random access point defined therein, i.e. a time corresponding to a picture of a view of the multiview signal that is coded without temporal prediction and other dependencies on previously coded pictures, but using only intra prediction as far as the reference view is concerned, and further using intra prediction and disparity-based prediction as far as the dependent view is concerned. Thus, disparity data signaled in the multiview data stream for inter-view prediction can be utilized to initialize a depth map estimate for the dependent view, and this primary depth map estimate can be continuously updated during the further course of multiview encoding with motion and / or disparity data signaled in the multiview data stream. The continuously updated depth map estimate thus obtained allows dependent methods of inter-view redundancy reduction to be performed in a more efficient manner than without access to this depth map estimate.

[0012] According to another aspect, the following discovery is exploited, whereby the overhead associated with an expanded list of motion predictor candidates for a block of a picture of a dependent view is relatively small in the sense of disparity compensation compared to the gain in motion vector prediction quality resulting from the addition of motion vector candidates determined from co-located blocks of a reference view, where the disparity between both blocks may or may not be determined using the first aspect.

[0013] Preferred embodiments of the invention are the subject matter of the dependent claims.

[0014] Preferred embodiments of the present invention are described in more detail below with reference to the figures. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 shows a block diagram of an apparatus for reconstruction of a multiview signal according to an embodiment. [Diagram 2] FIG. 2 shows a block diagram of an apparatus for encoding a multi-view signal compatible with the apparatus of FIG. 1 according to an embodiment. [Diagram 3] FIG. 3 shows the general relationship between projected object points, temporal motion vectors and disparity vectors in pictures at different views and times. [Figure 4] FIG. 4 shows the basic process for deriving a motion vector for a current block given motion in a reference view (with specific sample locations within the current block) and a depth map estimate for the current picture. [Diagram 5] FIG. 5 shows the basic process for mapping a given depth map for one view to another: (left) a given depth map for the view, where grey areas represent the background and white areas represent foreground objects; (center) a transformed depth map obtained by displacing samples with a disparity vector corresponding to the depth values ​​and keeping foreground objects for places where more than one sample is projected; black areas represent non-occluded regions where no samples are projected; and (right) a transformed depth map after filling the non-occluded regions with depth values ​​for the background. [Figure 6] FIG. 6 shows the generation of a depth map (using disparity vectors) for a random access unit. [Figure 7]FIG. 7 shows the temporal prediction of a depth map estimated using the motion parameters coded in the base view. [Figure 8] FIG. 8 shows the update of the depth map using the actual coded motion and disparity vectors. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] 1 shows an embodiment for an apparatus 10 for reconstructing a multi-view signal 12 that is encoded into a multi-view data stream 14. The apparatus 10 includes an input 16 for the multi-view data stream 14 and two outputs 18a and 18b for a reference view signal 20 and a dependent view signal 22, respectively.

[0017] The apparatus 10 further includes a reference view reconstructor 24 connected between the input 16 and the output 18a, and a dependent view reconstructor 26 connected between the input 16 and the output 18b. A depth map estimator 28 of the apparatus 10 is connected between a parameter output of the reference view reconstructor 24 and a reference input of the dependent view reconstructor 26, and is further connected to a parameter output of the dependent view reconstructor 26.

[0018] As will be outlined in more detail below, the device or decoder 10 of Fig. 1 reconstructs a multiview signal 12 from a multiview data stream 14 by following an encoding / decoding order according to which a reference signal 20 is processed before a dependent view 22. The multiview signal 12 may represent not only a spatial sampling of one common scene from different view directions or viewpoints associated with respective views 20 and 22 as shown in Fig. 1, but also a temporal sampling of this scene, as exemplarily shown in Fig. 1 by showing three times T-1, T and T+1 along a time axis 30. For each time instant, each view 20 and 22 comprises a picture 32t1 and a picture 32t2, and each picture 32t 1,2 represents each texture map.

[0019] FIG. 1 shows a picture 32t in which both views 20 and 21 are temporally aligned. 1,2 Note that we assume that we have: . However, the temporal resolution between views 20 and 22 can be different. Of course, the same applies to the spatial resolution of the picture and the depth map.

[0020] Moreover, the decoder 10 is configured to process the multiview signal 12 successively in time. More precisely, the decoder 10 processes a picture 32t of a particular time instant, e.g., T−1, before continuing to process pictures and depth maps of the next time instant T. 1,2 In this regard, it should be noted that the temporal coding order in time of the multiview signal 12 may be equal to or different from the display time order of the pictures and the depth maps, respectively.

[0021] The reference view reconstructor 24 is configured to reconstruct the reference view 20 from a reference view portion 36 of the multiview data stream 14, while the dependent view reconstructor 26 is configured to reconstruct the dependent view 22 based on a dependent view portion 38 of the multiview data stream 14. In practice, the reference view reconstructor 24 and the dependent view reconstructor 26 may be configured to operate in a similar manner. For example, the reference reconstructor 24 and the dependent view reconstructor 26 may operate on a block-by-block basis. Both may each be configured, for example, as a hybrid video decoder. The reference view reconstructor 24 reconstructs the picture 32t1 of the current time T, for example, by assigning a respective one of the available coding modes to the blocks 40 into which the picture is subdivided. The subdivision of the picture 32t1 into blocks may be predefined by default or signaled in the multiview data stream 14. The subdivision may subdivide the picture 32t1 in a regular manner into blocks of the same size or into blocks of different sizes. Furthermore, a multi-tree subdivision is also possible so that the block size of the blocks 40 may be adapted locally to the picture content. The available coding modes may include one or more intra-prediction modes according to which the reference view reconstructor 24 fills each block 40 by prediction from already reconstructed samples of blocks already reconstructed before the current block in a decoding order defined among the blocks of the picture 32t1, and one or more inter-prediction modes according to which the reference view reconstructor 24 reconstructs each block by motion compensation and / or prediction using motion data such as, for example, motion vectors, reference picture indexes, etc. For example, for the sake of explanation, two blocks are exemplarily shown to be reconstructed by inter prediction.The motion data 42 for these inter-predicted blocks may include motion vectors used by the reference view reconstructor 24 to copy respective portions of a reconstructed version of the reference picture 32t1 indexed by a motion reference index and comprised in the motion data 42. The motion data 42 is comprised in the reference view portion 36 of the multiview data stream 14.

[0022] The dependent view reconstructor 26 operates exactly the same as the reference view reconstructor 24, but is configured to reconstruct the dependent view 22 from the dependent view portion 38. Thus, in reconstructing the current picture 32t2 at the current time T, the dependent view reconstructor 26 can also use a block-by-block process with the subdivision into blocks 50 that can be fixed or signaled in the multiview data stream 14. Alternatively, inter-view prediction based on a depth map of the subdivision into blocks 50 as outlined in more detail below can be used by the dependent view reconstructor 26 to derive the subdivision into blocks 50 for the view 22 from the subdivision into blocks 40 of the view 20. As far as the coding modes are concerned, the dependent view reconstructor 26 can support the coding modes as they are described with respect to the reference view reconstructor 24. Thus, exemplarily, two blocks 50 are exemplarily shown to undergo inter prediction using motion data 54, respectively, so that each is appropriately copied from a respective portion of a reconstructed version of the previously reconstructed picture 32t2. In total, this motion data 58 represents motion data for the current picture or current time of the view 22. However, in addition to these coding modes, the dependent view reconstructor 26 has the ability to support one or more inter-view prediction modes for using disparity-compensated prediction to copy the respective blocks from a portion of the view 20 that is at the same time but spatially displaced, as defined by some disparity data. In FIG. 1, one disparity predicted block in the picture 32t2 is exemplarily shown with corresponding disparity data 60. The disparity data 60 may, for example, include a disparity vector or at least a disparity component along the view offset direction between the views 20 and 22, and optionally a view index indicating the reference view to which the respective block 50 of the dependent view 22 depends, which index may be advantageous in case of the coexistence of more than two views as exemplarily shown in FIG. 1.

[0023] That is, the reference view reconstructor 24 and the dependent view reconstructor 26 operate in such a way as to reduce, as much as possible, redundancy along the time axis 30 and in the inter-view direction between the views 20 and 22. This is also true for side information such as motion data and disparity data as well as prediction of the coding mode and subdivision information mentioned above. All of this information indicates redundancy between each other and between the views in the time direction.

[0024] However, the dependent view reconstructor 26 can more efficiently exploit the redundancy between views 20 and 22 if it has access to the depth map for the currently decoded picture 32t2. Thus, the depth estimator 28 is configured to provide a depth map estimate 64 as an estimate for the depth map of the current picture 32t2 at the current time T in a manner described in further detail below, and the dependent view reconstructor 26 is further configured to use this depth map estimate 64 to reconstruct the current picture 32t2 at the current time T of the dependent view 22 from the dependent view portion 38 of the multiview data stream 14. For example, having a depth map estimate 64 nearby, the dependent view reconstructor 26 can predict the motion data 54 of the current picture of the dependent view 22 and the motion data 42 for the current picture of the reference view 20 based on the depth map estimate 64 of the current view 22, and further reconstruct the current picture of the dependent view 22 using motion compensated prediction based on the predicted motion data. For example, the current view reconstructor 24 can be configured to use the depth data estimate 64 of the dependent view 22 to determine a corresponding position in the current picture of the reference view 20 when predicting the motion data 54, and further use the motion data 42 for the current picture of the reference view 20 at the corresponding position to predict the motion data 54 of the current picture of the dependent view 22. In the following description, possible ways in which the spatial lookup using the depth data estimate 64 is performed are described in more detail below. In particular, in the following description, the fact that the motion data 42 forms a good predictor for the motion data 54 is motivated in more detail. Of course, refinement data may be configured in the dependent view portion 38 to signal prediction residuals with respect to the motion data 54 .In particular, as described in further detail below, the dependent view reconstructor 26 may be configured to apply disparity vectors derived from the depth data estimate 64 for one or more given sample positions in the current block 50 of the current time picture 32t2 of the dependent view 22, and further use these disparity vectors to determine corresponding or warped positions in the same time picture 32t1 of the view 20 using the motion data 42 of one or more blocks 40 including the one or more warped positions as predictors for the motion data 54 of the current block 50. In the case of more than one such reference sample position in the current block 50, an average or median value of the resulting one or more reference motion data of the targeted block or blocks 40 may be used as a predictor.

[0025] Furthermore, the dependent view reconstructor 26 may be configured to predict disparity data 60 for a current picture of the dependent view 22 based on the depth data estimate 64 of the dependent view 22, and further reconstruct the current picture of the dependent view 22 using disparity-compensated prediction based on the predicted current disparity data. Refinement may also be signaled in the dependent view portion 38 of the multi-view data stream 14 and further used by the dependent view reconstructor 26 to refine the predicted current disparity data. Furthermore, as outlined above, in theory, the disparity data 60 of block 50 may also be predicted similarly. As outlined in more detail below, the dependent view reconstructor 26 may be configured to predict the disparity data 60 and 62 based on the depth data estimate 64 of the current view by converting depth data into disparity vectors and further using these disparity vectors as predictors for the disparity vectors in the disparity data 60 and 62 directly, respectively.

[0026] Of course, the dependent view reconstructor 26 can support any combination of the possibilities just mentioned, such as using depth data estimation to reduce inter-view redundancy between views 20 and 22 .

[0027] To derive the just-mentioned depth data estimate 64, the depth estimator 28 works as follows.

[0028] In particular, according to the embodiment of FIG. 1, the depth estimator 28 estimates the depth of each picture 32t 1,2 has a depth map estimate 64 associated with it, which are derived successively from one another in a series of updates. As outlined in further detail below, depth estimator 28 is configured to successively update depth map estimate 64 in a ping-pong manner between views 20 and 22, primarily for the purpose of providing such a depth map estimate 64 for each picture 32t2 of dependent view 22 to serve as a basis for the improved inter-view redundancy reduction outlined above.

[0029] It is assumed that, beforehand, the depth estimator 28 already has access to such depth estimates for one or more previous pictures 32t1 of the reference view 20, such as for example at time T-1. How the depth estimator 28 can access this depth map estimate 74 for a picture 32t1 decoded before the reference view 20 will be described further below. However, it should be noted that such depth map data can be signaled intermittently in the multiview data stream 14 specifically for the first picture 32t1 of the reference view 20 in a so-called random access unit, i.e. a group of pictures 32t1 that are decodable independently of any previous part of the signal 12. To indicate this possibility, a dotted line connects the depth estimator 28 to the input 16. In the following description, it is indicated that the possibility is not necessary for an extra transmission of such a starting depth map. Rather, the disparity data in the data stream portion 38 for the first picture 32t2 of the dependent view 22 in coding order within the random access unit is used to construct a starting depth map for the first picture 32t1 of the reference view 20 in coding order within the random access unit.

[0030] In particular, the depth estimator 28 is configured to generate the depth map 64 of the current picture 32t2 of the dependent view 22, for example, by applying the motion data 42 for the current picture 32t1 of the reference view 20 at the current time T to a depth map estimate 74 of any previous picture 32t1 of the reference view 20 at time T−1. As already mentioned above, the reference view reconstructor 24 reconstructs the current picture 32t1 of the reference view 20 using motion compensated prediction based on the motion data 42 signaled in the multiview data stream 14 for the reference view 20. The depth estimator 28 has access to this motion data 42 and further uses this motion data 42 for one of the stated updates of the sequence of updates, i.e. the transition 71 from the depth map estimate 74 of the reference picture 32t1 at the previous time T−1 to the depth map estimate 64 of the current picture 32t1 at the current time T. How this may be performed is outlined in more detail below. It should be noted that the application 71 of the motion data 42 to the depth map 74 for the previous time T-1 in advance means that the co-located block 72, i.e. the part in the depth map estimate 64 of the current picture 32t1, which is co-located with the block 40 for which this motion data 42 is signaled in the stream portion 36, is updated with the contents of the depth map estimate 74 for the picture 32t1 of the previous time T-1, i.e. copied from the referenced depth map estimate, i.e. the part in the referenced depth map estimate 74 indicated by the motion data 42' that is equal to the motion data 42. The remaining holes can be filled by interpolation and / or extrapolation making use of the additional information provided by the intra-coded blocks in the block 40 of the current picture 32t1. As a result, the depth map estimate 64 has been updated (or generated by transitioning from T-1 to T).

[0031] The depth estimator 28 also performs this update / transition 71 in order to further perform a series of updates, described further below, to serve as a basis for deriving a depth map estimate 64 for the current picture 32t2 of the dependent view 22 at the same time instant T. To complete the derivation, the depth estimator 28 warps the updated depth map estimate 64 of the current picture 32t1 of the reference view 20 to the dependent view 22 to obtain a depth map estimate 64 for the current picture 32t2 of the dependent view 22. That is, since the motion data 42 is defined only at block granularity, the resulting depth map estimate 64 for the view 22 as resulting from the update / transition 71 and warping 78 represents a quite coarse estimate of depth, but as shown below, such a coarse estimate is sufficient to significantly increase the efficiency in performing inter-view redundancy reduction.

[0032] Although possible details regarding warping 76 are described further below, in brief, the dependent view reconstructor 26 may be configured to perform warping 78 by deriving a disparity vector from the depth map estimate 64 of the current picture 32t1 and then applying the derived disparity vector to the depth map estimate 64 itself to obtain a warped depth map estimate 64 of the current picture 32t2 of the dependent view 22.

[0033] Thus, as soon as the depth estimator 28 provides the dependent view reconstructor 26 with the result of warping 76, i.e., a depth map estimate 64 for the current time T for view 22, the dependent view reconstructor 26 can use this depth map estimate 64 to perform the inter-view redundancy reduction outlined above, whose possible implementations are described in more detail below.

[0034] However, the depth estimator 28 continues to update 77 this depth map estimate 64 to obtain an updated depth map estimate 74 for the current picture 32t2 of the reference view 22, thereby maintaining a series of updates resulting in an estimate for the next time instant T+1. The dependent view reconstructor 26 is thus configured to update 77 the depth map estimate 64 for the current picture 32t2 of the dependent view 22 at the current time instant T using the disparity and / or motion data 54 and 60 for the dependent view 22 in a similar manner, at least for the motion data 54, as described above with respect to the update step 71. That is, the dependent view reconstructor 26 uses the disparity / motion data for the picture 32t2 for time instant T in the stream portion 38 for reconstructing this picture 32t2. As far as the disparity data 60 is concerned, the depth estimator 28 can easily convert the disparity vectors contained in the disparity data 54 into depth values ​​and, based on these depth values, assign updated depth values ​​to samples of the updated depth map estimate 79b of the current picture 32t2 of the dependent view 22 that are co-located with the respective disparity predicted blocks 50 in the picture 32t2. The motion data 54 can be used to copy the contents of the depth map estimate 74 of the referenced previous time T-1 picture 32t2 of the dependent view 22, in a portion thereof indicated by motion data equal to the motion data 54, into the portion in the updated depth map estimate 74 of the current picture 32t2 that is co-located with the block 50 for which this motion data 42 is signaled in the stream portion 36. The remaining holes can be filled by interpolation and / or extrapolation utilizing additional information provided by the intra-coded blocks in the blocks 40 of the current picture 32t1. As a result, an updated depth map estimate 74 for the current picture 32t2 has been updated (or generated by transitioning from T-1 to T). Possibilities regarding how the depth map estimate 74 for the referenced previous time T-1 picture 32t2 of the dependent view 22 may be derived at the start of a random access unit are described further below.However, the above-mentioned possibly explicitly transmitted depth map for view 20 at the start of such a random access unit can be warped to view 22 to obtain instead a depth map estimate 74 of the referenced previous time T-1 picture 32t2 of the dependent view 22.

[0035] To reduce blocking artifacts, updates 71 and 77 may be performed by using a weighting function that reduces the effect of individual block updates at block boundaries.

[0036] That is, based on the depth map estimate 64 as obtained by warping 76, the dependent view reconstructor 26 reconstructs the current picture 32t2 of the dependent view 22 using disparity and / or motion compensated prediction based on the disparity and / or motion data 54 and 60 for the dependent view 22 constituted in the dependent view portion 38 of the multi-view data stream 14, and further, in this case, the dependent view reconstructor 26 provides the disparity and / or motion data 54, 60 to the depth estimator 28, which in turn is used by the depth estimator 68 to perform the update 77.

[0037] After this update 77, the depth estimator 28 can warp 78 the updated depth map estimate 74 of the current picture 32t2 of the dependent view 22 back to the reference view 20 to obtain an updated depth map estimate 74 of the current picture 32t1 of the reference view 20 for time T, which can serve as a basis / reference for a transition / update 79 to the next time T+1, etc.

[0038] Thereafter, the depth estimator 28 simply repeats processes 71, 76, 77 and 78 (step 79 corresponds to step 71) iteratively to model the depth map estimation along the time axis 30 to continuously support the dependent view reconstructor 26 with the depth map estimation 64.

[0039] Further details regarding all of these steps 71, 76, 77, 78, and 79 are described in further detail below, all of which are individually applicable to the explanations given with respect to FIG.

[0040] Before describing further details regarding the concepts outlined above, an embodiment for an encoder (hereinafter referred to as encoder) compatible with the decoder (hereinafter referred to as decoder) of Fig. 1 will be described with reference to Fig. 2. Fig. 2 shows an apparatus for encoding a multiview signal 12 into a multiview data stream 14, and for this purpose includes a reference view encoder 80 having an encoder generally indicated with reference numeral 90, a dependent view encoder 82 and a depth estimator 84. The reference view encoder 80 is configured to encode the reference view 20 of the multiview signal 12 into the reference view portion 36 of the data stream 14, while the dependent view encoder 82 is responsible for encoding the dependent view 22 of the multiview signal 12 into the dependent view portion 38 of the multiview data stream 14. The reference view encoder 80 and the dependent view encoder 82 may operate in a backward prediction manner, and the depth estimator 84 may be configured to perform the depth map estimation and its successive updates in the manner described above with respect to the decoder 10 by using the same information available from the reference view encoder 80 and the dependent view encoder 82. That is, the depth estimator 84 is configured to generate 71 a depth map estimate 64 for the current picture 32t2 of the dependent view 22 by applying the motion data 42 for the reference view that is being used to motion compensate predict the current picture of the reference view to a depth map estimate of the previous picture 32t1 of the reference view 20, and further warping 76 the thus obtained depth map estimate 64 for the current picture 32t2 of the reference view 20 to the dependent view 22 to obtain a depth map estimate 64 for the current picture 32t2 of the dependent view 22. Similarly, the depth estimator 84 also performs a next update step 77 and a following backwarp step 78.For this purpose, the reference view encoder 80 and the dependent view encoder 82 may be connected in parallel between an input and an output of the encoder 90, while the depth estimator 84 may be connected between a parameter output of the reference view encoder 80 and a reference input of the dependent view encoder 82, and further connected to a parameter output of the dependent view encoder 82. A reconstruction output of the reference view encoder 80 may be connected to a prediction parameter output of the reference view encoder 80, e.g. an output of an intra prediction block.

[0041] The dependent view encoder 82 may encode the current picture or current time of the dependent view 22 using the depth map estimation 64 in the manner outlined above with respect to FIG. 1, i.e. to predict motion data 58, or at least 54, or to predict disparity data 60 and 62, or at least 60, or at least portions of these options, with or without generating prediction residual data for the respective motion or disparity data to form part of the dependent view portion 38.

[0042] In the following, a more detailed embodiment is presented, which is particularly advantageous when combined with hybrid coding types using block combining of regularly arranged tree root blocks, multi-tree block partitioning, such as in HEVC.

[0043] State-of-the-art concepts for using motion data of reference views for efficient coding of further views are all developed based on the MVC extensions of ITU-T Rec. H.264 | ISO / IEC 14496-10. The new video coding standardization project of ITU-T and ISO / IEC JTC 1 / WG 11, also called HEVC, shows very promising improvements over traditional 2D video coding techniques. The current working draft of HEVC offers substantial coding gains compared to ITU-T Rec. H.264 | ISO / IEC 14496-10. To achieve these gains, several concepts have been extended compared to ITU-T Rec. H.264 | ISO / IEC 14496-10. The main improvements in the areas of motion parameter coding and motion compensated prediction include:

[0044] · While the block sizes used for motion compensation prediction in ITU-T Rec. H.264 | ISO / IEC 14496-10 range from 4x4 to 16x16 luma samples, a larger variety of block sizes are supported in HEVC, ranging from 4x4 to 64x64 luma samples. In addition, the basic coding units are not given by fixed macroblocks and sub-macroblocks, but are adaptively selected. The largest coding unit is typically a block of 64x64 luma samples, but the largest block size is actually signaled in the bitstream. Partitioning blocks into sub-blocks can establish a subdivision hierarchy of 4 or more levels. · Motion vectors are not coded by using a fixed motion vector predictor. Instead, there is a list of motion vector predictor candidates, and one of these predictors is adaptively selected on a block basis. The selected predictor is signaled in the bitstream. ·ITU-T Rec. H.264 | ISO / IEC 14496-10 provide SKIP and DIRECT where the motion parameters (number of hypotheses, reference index, motion vector) are derived entirely from already coded information without coding any additional parameters (except residual information). HEVC provides the so-called merge mode. For this mode, a list of motion parameter candidates given by the motion parameters of spatially and temporally adjacent blocks is established. The motion parameters (including number of hypotheses, reference index and motion vector) selected for a block to be coded in merge mode are signaled by sending an index to the candidate list.

[0045] The following description describes a concept for using motion data of already coded views and disparity data for already coded pictures of a current view for coding a current picture of a current view in multi-view video coding, with this concept representing a possible implementation of the above-mentioned embodiment. Furthermore, advantages resulting from the above-mentioned and following embodiments are described in more detail. By using already coded motion and disparity information for predicting the temporal motion (or disparity) of a current view, the motion data rate for the current view can be significantly reduced, which results in an overall bitrate saving for coding of a multi-view video sequence. The described concept provides the possibility to directly derive all motion data for a block (or a common set of samples), in which case no further motion information is transmitted for the block. And it also provides the possibility to derive a motion vector predictor that is added to a list of candidate motion vector predictors. For the latter possibility, an index into a list of motion vector predictors and a motion vector differential are transmitted for the block, which identifies the final motion vector used for motion compensation prediction. In a particular embodiment of the present invention, not only motion parameters for a block, but also partition information for the block (which can divide the block into smaller blocks and further assign separate motion parameters to each sub-block) can be derived based on already coded motion and disparity information. The concept can be applied to a general block-based hybrid coding approach without assuming any specific macroblock or sub-macroblock structure. General block-based motion compensation is not modified but only codes the motion parameters, so that the concept can be integrated into a general block-based hybrid video coding scheme with very little complexity increase. It can also be directly extended to a more general concept, where other sets of samples, rather than rectangular blocks, are relevant for the specific motion parameters. The concept can be applied to multi-view coding with or without additional depth maps.Disparity information for calculating motion parameters may be derived based on the encoded depth map based on the encoded disparity vector.

[0046] With this concept representing a possible implementation of the above-mentioned embodiment, the following description describes a concept for using motion data of already coded views and disparity data for already coded pictures of a current view for coding a current picture of a current view in multi-view video coding. Furthermore, advantages resulting from the above-mentioned and following embodiments are described in more detail. By using already coded motion and disparity information for predicting the temporal motion (or disparity) of a current view, the motion data rate for the current view can be significantly reduced, which results in an overall bitrate saving for coding of a multi-view video sequence. The present invention provides the possibility to directly derive all motion data for a block (or a common set of samples), in which case no further motion information is transmitted for the block. And it also provides the possibility to derive a motion vector predictor that is added to a list of candidate motion vector predictors. For the latter possibility, an index into a list of motion vector predictors and a motion vector differential are transmitted for the block, which identifies the final motion vector used for motion compensation prediction. In a particular embodiment of the present invention, not only motion parameters for a block, but also partition information for the block (which can divide the block into smaller blocks and further assign separate motion parameters to each sub-block) can be derived based on already coded motion and disparity information. The concept can be applied to a general block-based hybrid coding approach without assuming any specific macroblock or sub-macroblock structure. General block-based motion compensation does not modify but only codes the motion parameters, so that the concept can be integrated into a general block-based hybrid video coding scheme with very small complexity increase. It can also be directly extended to a more general concept, where not rectangular blocks but other sets of samples are relevant for intrinsic motion parameters. The concept can be applied to multi-view coding with additional depth maps.Disparity information for calculating motion parameters may be derived based on the encoded depth map.

[0047] One advantage of the concept to be presented compared to conventional techniques for using motion data of already coded views is that the motion / disparity predictor is derived entirely based on the coded motion and disparity / depth information, without assuming any particular structure of the disparity field. At any time, it is not assumed that the disparity field can be well approximated by a constant translation or affine parameter for the image, instead, the actual coded disparity information is used to access the motion of the already coded views. Furthermore, the disparity of a macroblock is not assumed to be similar to the disparity of adjacent blocks, which is an unsafe assumption. By using the actual coded depth / disparity information, the concept provides a proper disparity estimation for blocks at object boundaries. Furthermore, since the motion of the current block is not assumed to be similar to that of adjacent blocks, an improved motion parameter predictor is provided at object boundaries. Furthermore, the concept does not require any transmission of disparity correction. Furthermore, the concept does not require modifying the actual motion / disparity compensation process of a hybrid video coding design to be incorporated therein. Only the derivation of the motion parameters and / or the motion parameter predictors is changed, so that it can be included in conventional video coding designs without any major modifications, and furthermore, the complexity is small. In addition, it should be noted that the concept can be applied to coding with or without a depth map. The depth map does not need to be coded as part of the bitstream. Rather, the coded disparity vector can be used to derive the disparity.

[0048] The concepts described below can be broken down into the following steps:

[0049] Deriving depth / disparity data for the current picture in the current view. Deriving candidate motion or disparity data for the current block based on the derived depth / disparity data. · Encoding of motion or disparity data for the current block.

[0050] In the following, these steps including preferred embodiments are described in more detail. All steps are described for motion compensation based on blocks with translational motion vectors. However, the concept can also be applied to more general schemes in which a generalized set of samples (e.g. non-rectangular parts of blocks or any other shape) is associated with a unique set of motion parameters, and it can also be applied for coding schemes in which motion compensation is performed by using a higher order motion model (e.g. affine motion model or other N-parameter motion model).

[0051] Before describing the concept in detail, we briefly describe its advantages and underlying ideas, which are also valid for the more general embodiment described above. The basic relationship between the projections of real-world object points at different views and different times is shown in Fig. 3. Assuming we know the real motion within the views and the real disparity between the views, the corresponding motion and disparity vectors are given as follows:

[0052] TIFF0007661385000002.tif81169

[0053] TIFF0007661385000003.tif11169

[0054] TIFF0007661385000004.tif97168

[0055] TIFF0007661385000005.tif58170

[0056] Deriving candidate motion or disparity data

[0057] TIFF0007661385000006.tif100169

[0058] TIFF0007661385000007.tif69169

[0059] All cameras in the camera array are of the same type and have the same focal length The optical axes of all cameras are parallel and in the same plane The scan line of the image sensor is parallel to the plane containing the optical axis.

[0060] TIFF0007661385000008.tif28168

[0061] TIFF0007661385000009.tif84170

[0062] By using the described basic relationship between predetermined depth values ​​and actual disparity, we describe a preferred embodiment for using motion information derived based on motion information already encoded in one or more reference views and predetermined estimated depth values.

[0063] Method 1: Switched motion / disparity vector prediction

[0064] In a preferred embodiment of the present invention, the underlying multi-view video coding scheme, eg for modules 24, 26, 80 and 82, includes a mode in which the following parameters are transmitted as part of the bitstream 21:

[0065] A reference picture index that identifies a particular (temporal or inter-view) reference picture of a given list of reference pictures. If the given list of reference pictures contains a single element, this index is not transmitted but is inferred at the decoder side. The reference pictures include temporal and / or inter-view reference pictures. A motion / disparity vector predictor index identifying a motion / disparity vector predictor of a predefined list of motion / disparity vector predictor candidates. If the list of motion / disparity vector predictor candidates consists of a single element, this index is not transmitted but is inferred at the decoder side. For at least one block of the picture, the list of motion / disparity vector predictor candidates contains motion / disparity vectors derived based on predefined depth / disparity and motion information in already coded views.

[0066] In addition, a motion / disparity vector difference specifying the difference between the motion / disparity vector used for motion / disparity compensated prediction and the selected predictor (indicated by the index transmitted in the motion / disparity vector predictor candidate list) may be transmitted as part of the bitstream. In one embodiment, this motion / disparity vector difference may be coded independently of the reference index and the selected predictor. In another embodiment of the present invention, the motion / disparity vector difference is coded according to the transmitted reference index and / or the selected predictor. For example, the motion / disparity vector difference may only be coded if a particular motion / disparity predictor is selected.

[0067] The reference picture list and the motion / disparity vector predictor candidate list are derived at the encoder and decoder sides in the same way. In a particular configuration, one or more parameters are transmitted in the bitstream to specify how the reference picture list and / or the motion / disparity vector predictor candidate list are derived. For a preferred embodiment of the present invention, for at least one of the blocks of the picture in the dependent view, such as 22, the list of motion / disparity vector predictor candidates includes motion or disparity vector predictor candidates derived based on a predetermined (estimated) depth value or based on a predetermined (estimated) depth value and a motion parameter of an already coded view. Besides the motion / disparity vector predictor derived based on the predefined depth value and the motion parameters of the already coded views, the candidate list of motion / disparity vector predictors may include spatially predicted motion vectors (e.g., motion / disparity vectors of directly adjacent blocks (left or top blocks), motion / disparity vectors derived based on the motion / disparity vectors of directly adjacent blocks) and / or temporally predicted motion / disparity vectors (e.g., motion / disparity vectors derived based on the motion / disparity vectors of co-located blocks in already coded pictures of the same view). The derivation of the motion / disparity vector candidates obtained by using the predefined depth data 64 and already coded motion parameters, e.g., 42, of other views, e.g., 20, may be performed as described below.

[0068] Based on the derivation of the representative depth for the current block

[0069] TIFF0007661385000010.tif89170

[0070] TIFF0007661385000011.tif88169

[0071] TIFF0007661385000012.tif206170

[0072] TIFF0007661385000013.tif232170

[0073] TIFF0007661385000014.tif98169

[0074] Derivation based on multiple depth values ​​for a given block

[0075] TIFF0007661385000015.tif52169

[0076] TIFF0007661385000016.tif176170

[0077] Method 2: A mode in which all relevant motion parameters are derived

[0078] In another preferred embodiment of the present invention, the multi-view video coding scheme includes a coding mode in which all motion parameters (including the number of hypotheses, the reference index and the motion vector) are derived based on a predefined depth map 64 and the motion parameters 42 of the already coded view 20. In a specific embodiment of the present invention, this mode can be coded as a candidate in a list of candidate motion parameters (as it is used in the merging syntax in the current HEVC working draft). In essence, the encoder and the decoder derive a list of motion parameter candidates for a block in the same way, where one of the motion parameter candidates is a motion parameter derived based on the motion of the already coded view 20. Then, an index is coded that signals to the decoder which of these motion parameter candidates is used. In the context of the merging syntax, it can be discussed that the current block is merged with a "co-located" block (in the spirit of representing similar content) in the reference view. In another embodiment, a specific syntax element signals the use of a new coding mode. In a slightly modified version, the number of motion hypotheses used to generate the prediction signal may be explicitly signaled in the bitstream, and furthermore, only the reference index and the associated motion vector are derived. In another modified version, the motion vector difference may additionally be transmitted to refine the derived motion parameters.

[0079] Derivation based on multiple potential reference indices

[0080] In a first preferred embodiment of the present invention, the current block 50 CThe derivation of the motion parameters for uses any of the concepts described for method 1 above, and further considers more than one potential reference index. In the following, we first describe how a reference index and associated motion vector for a particular motion hypothesis (and reference list) can be derived. As a first step, an ordered set of reference indexes for a given reference list is determined. This can be, for example, just a single reference index (e.g., the first index for the reference list or the first index representing a temporal reference picture), or it can consist of the first two indexes of the reference list, or it can consist of all reference indexes of the reference list, or it can consist of the first reference index representing a temporal reference picture and the first reference index not equal to the first reference index representing a temporal reference picture (i.e., the second temporal reference picture or the first inter-view reference picture in the list). Any other defined set of reference indexes is possible. Given an ordered set of reference indexes, the first reference index is considered, and further, a motion vector for this reference index is derived by any of the embodiments described for method 1 above. If the derived motion vector is marked as unavailable, the next reference index is considered and a corresponding motion vector is further derived. This process continues until an available motion vector is returned or all reference indexes in the list have been tested. If no available motion vector is found, the final motion parameter is marked as unavailable. In one configuration, the unavailable motion parameter is not inserted into the candidate list of motion parameters. In a second configuration, the unavailable motion parameter is inserted into the candidate list of motion parameters (for parsing robustness), but the encoder is not allowed to select the unavailable motion parameter.In a third configuration, the unavailable motion parameters are replaced with specific motion parameters, which may be, for example, a zero reference index and a zero motion vector, or a reference index and a motion vector derived based on motion parameters in spatial neighbors of the current block. If a new coding mode is signaled by a specific syntax element and the derived motion parameters are unavailable, the corresponding syntax element is not transmitted (and the coding mode is not used), or the encoder is not allowed to select a value for the syntax element that specifies the use of the new coding mode, or the unavailable motion parameters are replaced with the specific motion parameters (see above).

[0081] If the number of motion hypotheses or the number of reference lists used is explicitly coded, a set of motion parameters consisting of a reference index and a motion vector is determined for each motion hypothesis or reference list, as specified above.

[0082] If the number of motion hypotheses or the number of reference lists used is not explicitly coded, the number of motion hypotheses or the reference lists used is also derived based on the motion parameters actually coded in the reference view(s). Given the maximum number of motion hypotheses or the maximum set of reference lists that can be used, for each of the motion hypotheses (reference lists), a set of motion parameters is derived as described above. Then, the number of motion hypotheses (set of reference picture lists used) is given by the hypothesis (reference list) for which the derived motion parameters are marked as available. For example, if there are two potential motion hypotheses and furthermore a valid set of motion parameters (reference index and motion vector) is derived for both motion hypotheses, the new coding mode specifies bidirectional prediction with the derived motion parameters. However, if a valid set of motion parameters is derived only for one of the hypotheses (reference lists), the new coding mode specifies unidirectional prediction (one hypothesis) with a set of valid motion parameters. If a valid set of motion parameters is not derived for any of the motion hypotheses (reference lists), the complete set of motion parameters is marked as unavailable. In this case, the set of motion parameters is not added to the list of candidate motion parameters, or it is added (to parse robustness) but not used by the encoder, or it is replaced by a specific defined set of motion parameters (e.g., one motion hypothesis, a reference index equal to 0 and a motion vector equal to 0). It is also possible to check a different set of reference indexes for one or more of the potential motion hypotheses.

[0083] Derivation based on a single representative depth value

[0084] TIFF0007661385000017.tif72169

[0085] If the reference picture list for the reference view is configured differently than the reference picture list for the current view (i.e., a particular reference index does not always refer to the same access unit for both lists), the reference index for the current block may be modified in a way that it references a picture at the same time (or a picture with the same picture order count) as the corresponding reference picture in the reference view. If the reference index in the reference view refers to an inter-view reference picture, the reference index for the current view may be modified in such a way that it also refers to the selected inter-view reference picture (e.g. the same inter-view reference picture as the current view or a reference picture represented by the reference view). In this case, the motion vector must also be replaced by a disparity vector, which may be obtained by converting the representative depth d to a disparity vector. If a corresponding picture (same time or picture order count or reference index) for all reference pictures used in a reference block is not available in the reference list for the current block, then it may be considered that there are no motion hypotheses referencing reference pictures that are not available for the current block. If the reference block is intra-coded, the motion parameters may be replaced with motion parameters for disparity-compensated prediction. For example, the reference index may be set in a way that it refers to the reference view picture, and further, the motion vector may be set equal to the disparity vector obtained by converting the representative depth to a disparity vector. Alternatively, the motion parameters may be marked as unavailable.

[0086] Combined with Method 1

[0087] In an embodiment, the encoding modes described by the embodiments for Method 1 (encoding reference indexes, derivation of motion vectors or motion vector predictors) may be supported in addition to the encoding modes described by the embodiments for Method 2 (derivation of all motion parameters including the number of motion hypotheses, reference indexes and motion vectors or motion vector predictors).

[0088] Method 3: A mode in which all relevant motion parameters and block partitions are derived

[0089] In another preferred embodiment of the present invention, the multi-view video coding scheme includes coding modes in which different motion parameters for sub-blocks of a given block are derived based on motion parameters in already coded reference views, such as 20, and estimated depth map 64. Or, in other words, the multi-view video coding scheme includes coding modes for blocks in which different motion parameters for sub-blocks of a given block are derived based on motion parameters in already coded reference views, such as 20, and estimated depth map 64. C The motion parameters associated with the partitions and sub-blocks of are derived based on the motion parameters in the reference views and the estimated depth maps already.

[0090] For this coding mode, a minimum block size is defined, which may be equal to the minimum block size supported for motion / disparity compensated prediction or may be a multiple of the minimum block size supported for motion / disparity compensated prediction. C If is less than or equal to the defined minimum block size, the current block 50 Cis not divided into smaller blocks for the purpose of motion / disparity compensation, and the associated motion parameters are derived as described for any of the embodiments for method 2 above. If a given current block is larger than the defined minimum block size, it is divided into sub-blocks having a size equal to the defined minimum block size. For each of these sub-blocks, a set of motion parameters is derived using any of the embodiments for method 2 above.

[0091] If the motion parameters for any of the sub-blocks are marked as unavailable (e.g., the corresponding reference block 40 R If a sub-block is coded in intra-mode or it only uses inter-view prediction, they may be replaced with the motion parameters of any of the neighboring sub-blocks for which motion parameters are available. Such an algorithm may work in such a way that the neighboring blocks are tested in a certain defined order (which may depend on the position of the sub-block to be replaced), and furthermore, the motion parameters of the replaced sub-block are set equal to the motion parameters of the first sub-block in the specified order that has valid motion parameters.

[0092] In a particular embodiment of the present invention, the resulting sub-blocks having a predetermined minimum block size are CIn another embodiment of the present invention, the resulting sub-blocks may be combined based on associated motion parameters to form a larger block used for motion / disparity compensated prediction. The combination of sub-blocks may proceed hierarchically. Thus, in a first hierarchical stage, a set of four adjacent blocks may be considered. If the number of motion hypotheses and associated reference pictures and motion vectors are the same for all four sub-blocks, the four sub-blocks are summarized into a larger block (with motion parameters that are identical to those of the original sub-blocks). In the next hierarchical stage, the four blocks of the next hierarchical level (consisting of the four original sub-blocks) are considered. If all four blocks were summarized into a larger block in the previous hierarchical stage, and furthermore, if the number of motion hypotheses and associated reference pictures and motion vectors are the same for all four blocks, these four blocks are summarized again into a larger block (with motion parameters that are identical to those of the original sub-blocks). This algorithm continues up to the highest possible hierarchical level for a given current block. In the extreme case (when the motion parameters of all sub-blocks are the same), every current block is not divided but is associated with a unique set of motion parameters. In a slightly modified version, the four blocks are summarized into a larger block if the motion vectors are not exactly the same, but the difference between the motion vectors (which can be defined as the maximum component difference or the absolute value of the vector difference) is smaller than or equal to a defined threshold (the number of motion hypotheses and the reference pictures used must still be the same). In this case, the motion vector associated with the larger block is determined as a function of the motion parameters of the four sub-blocks. Possible functions are the average of the motion vectors, the median of the motion vectors (or the component-wise median), the motion vector of any particular sub-block or the motion vector that occurs most often in the four sub-blocks).

[0093] In the present invention, the coding modes described by the embodiments of Method 1 (coding of reference index, derivation of motion vector or motion vector predictor) can be supported in addition to the coding modes described by the embodiments of Method 3 (block division and derivation of all motion parameters including the number of motion hypotheses, reference index and motion vector or motion vector predictor). In addition, the coding modes according to any embodiments of Method 2 can be supported.

[0094] Encoding Motion and Disparity Data

[0095] As already mentioned above, the use of the coding modes representing the embodiments of the present invention requires signaling to the decoder. This can be realized in different ways. In one version, a specific syntax element (which may be a flag) can be inserted into the syntax, which signals whether a conventionally derived motion vector predictor (or motion vector or motion parameter set) is used or whether a motion vector predictor (or motion vector or motion parameter set) derived using depth map estimation and motion parameters of already coded views is used. In another version, a motion vector predictor (or motion vector or motion parameter set) derived using depth map estimation and motion parameters of already coded views can be inserted into a candidate list of conventionally derived motion vector predictors (or motion vectors or motion parameter sets), and further an index is transmitted signaling which motion vector predictor (or motion vector or motion parameter set) is used. A particular syntax element or index into a candidate list may be transmitted using fixed length coding, variable length coding, arithmetic coding (including context-adaptive binary arithmetic coding), or PIPE coding. When context-adaptive coding is used, the context may be derived based on parameters of neighboring blocks (e.g., a particular syntax element or index into a candidate list).

[0096] In a preferred embodiment of the present invention, the multi-view video coding scheme includes an encoding mode in which one or more motion hypotheses are signaled by transmitting a reference picture index, a motion vector predictor index, and a motion vector difference for each motion hypothesis. For this encoding mode, a list of candidate motion vector predictors is derived based on the transmitted reference picture index, and the transmitted index further signals which one of the motion vector candidates is used. By using the embodiment, one of the motion vector candidates (for at least one block) is derived based on the depth map estimate and motion parameters of an already coded view (see method 1 above). In a slightly modified version, the motion vector difference is not transmitted but is inferred to be equal to 0 (for all motion vector candidates or only for those motion vector candidates that have been derived based on the depth map estimate and motion parameters of an already coded view).

[0097] In another preferred embodiment of the present invention, the multi-view video coding scheme includes a coding mode in which one or more motion hypotheses are signaled by transmitting a motion parameter index (or merge index). For this coding mode, a list of candidate sets of motion parameters (including the number of motion hypotheses, reference indexes and motion vectors) is derived. By using the embodiment, one of the candidate sets of motion parameters (for at least one block) is derived based on depth map estimates and motion parameters of already coded views (see methods 2 and 3 above). In a particular version of this embodiment, the candidate set of motion parameters (for at least one block) derived based on depth map estimates and motion parameters of already coded views includes partition information for the current block (see method 3 above). In a slightly modified version of this embodiment, a motion vector difference may be transmitted in addition (potentially depending on the selected set of motion parameters).

[0098] In another embodiment of the present invention, the multi-view video coding scheme includes a coding mode corresponding to method 2 or 3, and the syntax further includes a flag specifying whether this coding mode is used.

[0099] Deriving a Depth Map Estimate for the Current Picture

[0100] The current picture 50 based on the motion parameters of the already coded views, e.g. 20 C The derivation of the motion parameters for the blocks of the current picture requires the availability of a depth map estimate 64 for the current picture, as described so far. As mentioned above, this depth map estimate 64 can specify a per-sample depth map (depth values ​​are specified for each sample of the current picture) or a per-block depth map (depth values ​​are specified for a block of samples). The depth map estimate 64 can be derived based on already coded parameters, such as a depth map or a disparity vector and motion parameters. In principle, the possibilities for deriving the depth map estimate 64 for the current picture can be divided into two classes. For one class, the depth map estimate is derived based on the actual coded depth map. The corresponding possibilities described below include that the coded depth maps are present in the bitstream (before they are used). The concept of the second class does not require the depth map to be coded as part of the bitstream. Instead, the depth map estimate is derived based on the coded disparity vector. The second class of procedures may be applied regardless of whether the depth map is coded as part of the bitstream. This is the case described above with respect to Figures 1 and 2, where the following description previously provides transferable details on each aspect individually. It should also be noted that both classes of methods may be applied when the depth map is coded. It is also possible to select different methods for different frames.

[0101] In the following, basic concepts and preferred embodiments for deriving a depth map estimate (with or without a coded depth map) are described.

[0102] Class 1: Derivation based on encoded depth map

[0103] If the depth map associated with the current picture 32t2(T) is coded before the current picture, the reconstructed depth map can be used directly as an estimate of the actual depth map for the current picture. It is also possible to preprocess the coded depth map (e.g., by applying filtering to it) and further use the results of the prefiltering as an estimate of the depth map used to derive the motion parameters.

[0104] In most configurations, the depth map associated with a particular picture is coded after the picture 32t2(T) (often directly after the associated picture). Such a configuration allows that coding parameters (e.g., motion parameters, etc.) transmitted for coding a conventional video picture can be used to predict coding parameters used for coding the depth map, which improves overall coding efficiency. However, in such a configuration, the depth map associated with the picture cannot be used as an estimate for the depth map in deriving the motion parameters 54. However, depth maps for already coded views (of the same access unit), such as 20, are usually available and can be further used to derive an estimate of the depth map of the current picture. At least the depth map of the base view (independent view) 20 is available before coding any dependent view 22. Since the depth map of any view represents to some extent the geometry of the projected video scene (e.g., in combination with camera parameters such as focal length and distance between the cameras), it can be mapped to another view. Thus, if a depth map for the current picture 32t2(T) is not available, a coded depth map for an already coded view of the same access unit 20 is mapped to the current view, and the result of this mapping is used as a depth map estimate.

[0105] TIFF0007661385000018.tif87169

[0106] If more than one depth value is assigned to a particular sample location, it means that the foreground objects are displaced in front of the background objects. Therefore, the depth value (of the potential depth values) that represents the smallest distance to the camera is assigned to such sample location. If a depth value is not assigned to a particular sample location, it means that the foreground object has moved and the previously covered background is now visible. The best that can be done for such regions is to assume that the non-occluded background has the same depth compared to the adjacent background samples. Therefore, the region with no assigned depth value is filled with the depth value of the surrounding sample that represents the largest distance to the camera.

[0107] This algorithm is specified in more detail below. To simplify the following discussion, it has been assumed that larger depth values ​​represent smaller distances to the camera than smaller depth values ​​(although the algorithm can be easily modified for the opposite assumption).

[0108] TIFF0007661385000019.tif119170

[0109] The algorithm for mapping a depth map of a given view to a different view is further illustrated in FIG. 5 based on a very simple example. FIG. 5 illustrates a possible process for mapping a depth map, for example 32t1(T), given for one view 20 to another view 22. On the left side, the given depth map for a reference view is illustrated, in which the shaded areas represent the background and the white areas represent the foreground objects, and in the center of FIG. 5, the transformed depth map is illustrated, which is obtained by replacing the samples of the given map with the disparity vectors corresponding to the depth values ​​and by keeping the foreground objects for the positions where more than one sample is projected. The black areas represent non-occluded areas where no samples are projected. FIG. 5 illustrates on the right side the transformed depth map after filling the non-occluded areas by the depth values ​​for the background, i.e. ba background filling.

[0110] In a particular embodiment of the present invention, hole-filling can be achieved by a particularly simple algorithm that processes the lines of the transformed depth map separately. For each line segment consisting of consecutive undefined depth values, the two surrounding values ​​are considered, and furthermore, all depth samples of the line segment are replaced by the smaller of these two depth values ​​(background depth). If a line segment has only one surrounding depth value (because it is located on the image border), the depth samples of the line segment are replaced by this value. If a complete line has undefined values ​​after this process, the same process is applied to the columns of the depth map.

[0111] Although the above algorithm is described for a per-sample depth map, it may be applied to a per-block depth map (resulting in less complexity), or a given per-sample depth map for a reference view may first be converted (by downsampling) to a per-block depth map and then the algorithm may be applied for the per-block depth map.

[0112] Class 2: Derivation based on coded disparity and motion vectors

[0113] TIFF0007661385000020.tif139169

[0114] The idea of ​​generating a depth map estimate is further illustrated by several figures (showing the processing steps for multi-view coding with two views as they are performed by the depth estimator 28). The coding / decoding starts with a random access unit, for which the base view picture 32t1(0) is intra-coded and the non-base view picture 32t2(0) is coded using only intra- and inter-view prediction (but not motion-compensated prediction). After coding the second view 22 in the random access unit "0", a block-based depth map estimate for this second view 22 is generated 120 using the coded disparity vector 122 for this view 22, as shown in FIG. 6. Then, this depth map estimate 642(0) for the second view 22 is mapped 124 to the first view (base view) 20, and a depth map estimate 641(0) for the first view 20 is obtained. It should be noted that for the second view 22 of the random access unit, derivation of motion / disparity parameters based on motion parameters and disparity estimation of the base view cannot be used, because no depth map estimation is available when the second view 22 of the random access unit is encoded.

[0115] When a third view is encoded, a depth map estimate of either of the first two views (preferably the second view) may be mapped to the third view resulting in a depth map estimate for the third view, which may be used to derive motion parameters for the third view. Then, after encoding the third view, a block-based depth map may be generated using the encoded disparity vector for the third view (which may be used later to generate a depth map estimate for any following views). Essentially the same process as for the third view may be used for any following views.

[0116] In a non-random access unit, a picture of a base view is typically coded mainly by motion compensation prediction, because motion compensation prediction usually provides better coding efficiency than intra coding. After a picture of a base view is coded, a depth map estimate for this picture is generated 140 (comparable to 71 in FIG. 1) using the motion parameters 42(1) for the picture 32t1(1), as shown in FIG. 7. Thus, each block of a new depth map estimate 641(1) is created 140 by motion compensating the corresponding reference picture or depth map estimate 641(0) for the picture (comparable to 74 in FIG. 1). The reference picture and the corresponding motion vector 42(1) used are the reference picture and the motion vector coded in the data stream for the related picture. Depth samples for intra-coded blocks may be obtained by spatial prediction. This depth map estimate for the base view is mapped 142 (compare 76 in FIG. 1 ) to a coordinate system for the second view to obtain a depth map estimate 642(1) for the second view that can be used to derive motion parameters, i.e., to perform inter-view redundancy reduction.

[0117] For any further coded view, a depth map estimate may be generated by mapping the depth map estimate for any already coded view (base view, second view, etc.) to the corresponding view.

[0118] After actually encoding the picture of the second view (or any subsequent view), the associated depth map estimate may be updated with the actually encoded motion and disparity vectors 160 (comparable to 77 in FIG. 1 ) as shown in FIG. 8 . For blocks encoded with disparity compensation, the depth map samples may be obtained by converting the encoded disparity vectors 60 to depth values ​​162 as described above. For blocks encoded with a motion compensation mode, the depth samples may be obtained by motion compensating the depth map estimate for the reference frame 32t2(0). Or, alternatively, the depth correction values ​​added to the current depth map estimate 642(1) may be derived based on the encoded motion parameters 42(1) and 54(1) for the current and reference views. The depth samples of intra-coded blocks may be predicted using spatial prediction or using the motion parameters of neighboring blocks. After the updated depth map estimate 74 for the second view is generated, this depth map estimate 74 is mapped 164 (compares 78 in FIG. 1) to the base view 20 to obtain a depth map update 64'1(1) for the base view 20 (compares 74 in FIG. 1).

[0119] If more than two views are coded, the depth map update process for these views is the same as for the second view, however, the base view depth map is only updated after the second view is coded.

[0120] The motion compensation operation for the depth map may be performed using the coded sub-sample accurate motion vectors. However, it is often preferable (from the viewpoint of complexity and coding efficiency) if the motion compensation operation for the depth map is performed at sample (or block) accuracy. Therefore, the actually coded motion vectors are rounded to sample or block accuracy, and these rounded vectors are used to perform the motion compensation. Furthermore, the described concept may be applied for sample-wise and block-wise depth map estimation. The advantage of using a block-based depth map is the lower complexity and memory requirements for all processing steps. For a block-based depth map, each depth sample represents the depth for a block of samples of the associated picture (e.g., a 4x4 block or an 8x8 block). All the described operations may be performed for a block-based depth map in a straightforward manner (i.e., by simply considering the lower resolution of the depth map, where one depth sample represents multiples instead of just one texture sample).

[0121] Besides the mapping of a given depth map from one view to another, which is described above, the algorithm includes the following basic steps:

[0122] Creating a depth map based on disparity vectors for random access unit pictures. · Temporal prediction of base view depth maps using motion parameters of related pictures. · Updating the depth map estimate with the actual coded motion and disparity vectors for the associated picture.

[0123] Specific embodiments of these algorithmic steps are described below.

[0124] Creating depth maps for pictures in random access units - Patents.com

[0125] In a particular embodiment of the present invention, the creation of a depth map for a picture of a dependent view in a random access unit proceeds as follows. In general, such a picture includes blocks that are coded using disparity-compensated prediction and blocks that are intra-coded. First, all blocks that are coded using disparity-compensated prediction are considered. The disparity vectors are converted to depth values, and these depth values ​​are assigned to the corresponding samples of the depth map. If more than one motion hypothesis is used, one hypothesis is selected or the final depth value is set equal to a function of the depth values ​​for the individual motion hypotheses (e.g., the average, median, maximum or minimum). After assigning depth values ​​for all disparity-compensated blocks, the depth values ​​for the intra-coded blocks are obtained by spatial intra prediction. In one version, these samples may be obtained by using the same intra prediction mode used for the associated texture picture. In another version, the depth of an intra block may be set equal to a depth value obtained by a weighted average of the surrounding samples (or blocks), where the weighting factor may be determined based on the intra prediction mode used. In a further version, the depth for an intra block may be obtained by setting a depth sample equal to a value given by a particular function of the surrounding intra samples (e.g., average, median, maximum, minimum). Other spatial prediction algorithms are also possible. Depth assignment for intra-coded blocks may be performed in a single loop over the blocks in the image. In essence, the blocks are processed in a particular order (e.g., coding order), and furthermore, for both disparity compensation and intra blocks, depth values ​​are generated in this order (i.e., depth assignment for intra-coded blocks does not have to wait until all disparity compensation blocks have been processed).

[0126] Temporal prediction of base-view depth maps

[0127] In general, a picture of a base view includes motion compensation blocks and intra-coded blocks. Depth values ​​for motion compensation blocks are derived by motion compensation prediction of depth map estimation for the corresponding reference picture. If a block of a texture picture is coded with a single motion hypothesis, a depth sample for this block may be obtained by replacing a depth sample of a depth map estimation for the reference picture (given by a signaled reference index) by a transmitted (or estimated) motion vector. This motion compensation operation may be performed with the accuracy of the transmitted motion vector (which is usually sub-sample accuracy) or with a sample or block accurate motion vector. If motion compensation is performed with sub-sample accuracy, an interpolation filter is applied to generate samples at sub-sample positions. If motion compensation is performed with sample or block accuracy, the transmitted motion vectors are rounded before they are used. If a block of the related picture is coded with more than two motion hypotheses, one of the hypotheses may be selected for motion compensation of the depth map, or all motion hypotheses may be used by generating a final depth map estimate for the block as a weighted sum of depth prediction signals for the individual motion hypotheses. Furthermore, a depth sample for a block of a given size may be set equal to a representative depth. This representative depth may be obtained by selecting a specific position within the block and further deriving a depth value for this position using motion compensation, or it may be obtained by defining a function of the motion-compensated depth value for this block. Such a function may be the average of the depth samples, the median of the depth samples, the minimum or maximum of the depth samples, or the depth sample value that occurs most often in the block.

[0128] Depth map update using encoded motion and disparity parameters - Patents.com

[0129] As mentioned above, the depth map estimate for a picture may be updated after encoding the picture. In the following, several embodiments of such an update algorithm are described. The blocks of a picture are processed in a particular order (e.g., encoding order), and further, for each block, the following applies:

[0130] If a block is intra-coded (in an associated texture picture), the depth sample values ​​for this block may be derived by spatial prediction using samples of neighboring blocks. Some examples for such spatial prediction techniques are described above. Depth sample values ​​may also be obtained by motion compensation using motion parameters of neighboring blocks (see the description for motion compensation blocks below). It is sometimes preferable if intra blocks are processed after all motion and disparity compensation blocks have been processed, because then more neighbors are available and can be further used for spatial prediction of depth samples or motion parameters. Otherwise, if the block is coded with one or more disparity vectors (disparity hypotheses), the depth sample is derived by converting the disparity vector into a depth value. If only one disparity vector (disparity hypothesis) is used for the block, the depth value is given by the corresponding disparity vector. If more than one disparity vector is used for the block, one of the disparity vectors may be selected to derive the depth value, or for each disparity vector a depth value may be derived, and the finally assigned depth value is obtained by applying a function of the individually derived depth values. Possible functions are, among others, the minimum or maximum of the individual depth values, the median of the depth values, the average of the depth values, or the depth value that occurs most often.

[0131] It should be noted that a block coded using a disparity vector may additionally be associated with a temporal motion vector, in which case the temporal motion vector may be ignored, or the derived depth value may be combined with a depth value derived for a temporal motion hypothesis (see below) in any particular way (e.g., by averaging these two signals). Otherwise, the block is coded using only the temporal motion hypotheses, which are then used to update the depth samples for the block.

[0132] In a first embodiment of the present invention, the depth map samples are derived by direct motion compensation prediction using a depth map estimate associated with a reference picture for the current view. This motion compensation operation can be realized by any of the embodiments for temporal prediction of the base view depth map described above.

[0133] In a second embodiment of the present invention, the depth map samples are not simply motion compensated, but instead, a depth correction value is derived based on the motion vector coded for the current view and the motion vector coded for the corresponding block in the reference view, and this depth correction value is added to the estimated depth map to obtain an updated depth map estimate. The advantage of such an approach is that the depth change between the two instances can be taken into account.

[0134] TIFF0007661385000021.tif102170

[0135] TIFF0007661385000022.tif32169

[0136] A basic concept for obtaining an updated depth value using the motion vectors of the current and reference blocks has been shown. This basic concept may be used in various practical implementations. For example, if more than one motion vector is used for the current block and / or the reference block, a single motion vector may be selected, or different depth values ​​may be derived for different motion vectors, and the final depth is obtained by using the average (or any other function) of the individually derived depth values. Furthermore, only those motion hypotheses must be considered for the current and reference blocks that use the same reference access unit (or reference time or reference picture order count or reference index). If no such motion hypothesis exists, the conventional motion compensation process described above may be used, or the depth sample may be marked as unavailable and then later replaced by an appropriate post-processing step. Furthermore, the new depth value may be derived based on a single sample for the block, or for all samples of the block, or for a selected subset of samples of the block. When different depth values ​​for different samples are derived, separate depth values ​​may be assigned for different regions of the current block (e.g., for all 4x4 blocks in the current block), or the different depth values ​​may be used to derive a representative depth for all blocks, for example, by using the average, median, minimum, maximum, or most frequently occurring depth value. The depth map update process may be combined with some pre-processing steps. For example, unavailable depth values ​​may be replaced with depth values ​​of adjacent samples or blocks.

[0137] With respect to the above-outlined description of methods 1-3, each block 40 of the reference view R In another type of method, the disparity vector used to determine the disparity vector is used to verify the reference view. RIt should be noted that these different alternatives of refining the motion or motion / disparity vector / parameter candidate list by adding respective information obtained from blocks such as: According to this embodiment, the depth estimator 28 in the decoder of FIG. 1 is optional just like the depth estimator 84 in the encoder of FIG.

[0138] In particular, according to the latter aspect, the above description of methods 1-3 reveals an apparatus for reconstructing a multiview signal into a multiview data stream, which includes a dependent view reconstructor 26 configured to do the following to reconstruct the dependent view 22. To describe the functionality, reference is made to the schematic diagrams of the multiview signal in FIG. 1 and FIG. 4. In particular, the dependent view reconstructor 26 derives a list of motion vector predictor candidates for the current block 50c of the current picture 32t2(T) and the dependent view 22 by first determining a disparity vector 102 for the current block 50c representing the disparity between the current picture 32t2(T) of the dependent view 22 and the current picture 32t1(T) of the reference view 20 of the multiview signal at the block 50c. To do so, the dependent view reconstructor 26 uses motion and disparity vectors associated with previously decoded parts of the multiview signal, such as, for example, motion / disparity vectors of pictures 32t1(T) and 32t2(T-1). In the other embodiments outlined above, the estimated depth map associated with the current picture 32t1(T) of the reference view is used as a basis for determining the disparity vector 102, together with updating the depth map estimation using the motion and disparity vectors of the previously coded picture of the dependent view as well as the estimation and reference views described above, and in this respect the above description is incorporated for the current embodiment as well, although in principle other possibilities exist as well. For example, the dependent view reconstructor 26 can spatially / temporally predict the disparity vector for the current block 50c and further use this predicted disparity vector as the disparity vector 102.

[0139] Then, the dependent view reconstructor 26 reconstructs the block 40 in the current picture of the reference view using the determined disparity vector 102. R and further adding the motion vector to a list of motion vector predictor candidates, which is determined by the block 40 R , i.e. motion vector 42 R and the associated motion vector.

[0140] As mentioned above, when deriving the list of motion vector predictor candidates, the dependent view reconstructor may use spatially and / or temporally adjacent blocks of the dependent view 22, i.e., blocks, spatially and / or temporally adjacent current block 50, C The dependent view reconstructor 26 may be configured to spatially and / or temporally predict one or more further motion vectors from the dependent view, and the one or more further motion vectors or versions derived therefrom are then added by the dependent view reconstructor 26 to a list of candidate motion vector predictors.

[0141] The dependent view reconstructor is implemented in block 50. C For this purpose, the current block 50 extracts from the multi-view data stream, and more specifically, from its dependent view portion 22, index information that identifies one of a list of motion vector predictor candidates. C is assumed to undergo motion compensated prediction, i.e., associated with a temporal prediction mode, the dependent view reconstructor 26 reconstructs block 50 using a motion vector that is equal to or at least dependent on the identified motion vector candidate, i.e., the one indexed or identified by the index information. C Block 50 performs motion compensation prediction C The expanded list of motion predictor candidates and the associated overhead is reduced by the motion vector candidates 42 determined from the reference view. RThis is relatively small compared to the gain in motion vector prediction quality resulting from the addition of

[0142] As described above, the motion information extracted by the dependent view reconstructor 26 for the current block 50c need not be limited to index information. C block 50, extracting a motion vector difference for the identified motion vector candidate, and further such that the motion vector used further depends on the motion vector difference and the sum of the identified motion vector candidate, i.e., one identified by the index information from the list of motion vector predictor candidates; C The device may be further configured to perform a reconstruction of the

[0143] In the above, motion and disparity compensated prediction are strictly distinguished. However, the difference between both can vanish, for example, if the same mode is used to signal both, for the difference between both can be derived simply from an index that indexes the picture compared to the one on which the motion / compensated prediction is performed. The aforementioned functionality of the decoder-dependent view reconstructor can then be rewritten by replacing "motion" with "motion / disparity" so that there is no difference. Naturally, measures must be taken so that the vector candidates actually identified refer to the same type of previously coded picture, i.e., before in time or before in the view direction, or even the addition to the list is conditionally restricted and performed accordingly.

[0144] According to the above-mentioned method 1, the dependent view reconstructor 26 performs the following steps in block 50: CFor this purpose, it is further configured to extract a reference picture index that identifies reference pictures of a list of reference pictures, including the current picture 32t1(T) of the reference view 20 and the already decoded picture 32t2(t<T) of the dependent view 22. Further, the dependent view reconstructor 26 may be configured to perform motion compensation prediction using, as a reference, one of the already decoded pictures of the dependent view as identified by the reference picture index for the reference picture that is one of the already decoded pictures of the dependent view 22. Further, when the reference picture is the current picture 32t1(T) of the reference view, the determined disparity vector 102 or the modified disparity vector derived from the determined disparity vector 102 is added to the list of disparity vector prediction candidates, and index information for identifying one of the lists of disparity vector prediction factor candidates from the multi-view data stream is extracted. Further, using a disparity vector that depends on the disparity vector candidate specified using the current picture 32t1(T) of the reference view 20 as a reference, block 50 C performs disparity compensation prediction of block 50 C to reconstruct it. Also, the difference between motion compensation and disparity compensation prediction can be resolved. The same prediction mode can be signaled for block 50 C The reference picture index is defined by a reference picture buffer or list of reference pictures that includes both those that were temporally previous, i.e., pictures decoded before the dependent view, and those that were previous to the view, i.e., pictures decoded before other views, and is indexed.

[0145] As will be apparent from method 2, the dependent view reconstructor 26 may be configured to perform the derivation of the list of motion vector predictor candidates via a list of motion / disparity vector predictor candidates, which is a list of motion / disparity parameter candidates, each of which includes a number of hypotheses and, for each hypothesis, a motion / disparity motion vector and a reference index that identifies a reference picture from such common list of reference pictures just outlined. The dependent view reconstructor then performs ... including a motion / disparity motion vector and a reference index that identifies a reference picture from such common list of reference pictures just outlined. R and a list of motion / disparity parameter candidates depending on the motion / disparity parameters associated with the motion / disparity parameter candidate specified by the index information, and further, a motion / disparity parameter depending on the motion / disparity parameter candidate specified by the index information is used to perform a block 50. C Block 50 performs motion / disparity compensated prediction C The motion parameters can be determined simultaneously as described above for multiple hypotheses, and for each hypothesis, a reference index and a motion / disparity vector difference. As described above, multiple hypotheses can be pre-determined, for example, via the type of picture.

[0146] Then, as described in Method 3, the dependent view reconstructor performs the motion / disparity prediction as far as block 50 is concerned. R From Block 50 C The method may be further configured to further employ a division for

[0147] The encoder of FIG. 2 is configured according to the latter aspect to act accordingly in encoding the multi-view signal into a multi-view data stream. In particular, the dependent view reconstructor 26 is implemented as shown in block 50. C , a list of motion vector predictor candidates is derived in the same way, that is, a block 50 representing the disparity between the current picture of the dependent view 22 and the current picture of the reference view 20 of the multiview signal with the current picture of the dependent view 22. CThe disparity vector for is determined via motion and disparity vectors associated with a previously coded portion of the multiview signal. Then, the disparity vector for block 50 in the current picture of the reference view is determined via motion and disparity vectors associated with a previously coded portion of the multiview signal. R is determined using the determined disparity vector, and a motion vector is added to a list of motion vector predictor candidates, which depends on the motion vector associated with the determined block of the picture of the reference view. The dependent view reconstructor is C , inserting index information into the multi-view data stream that identifies one of a list of motion vector predictor candidates, and further performing a motion vector prediction in block 50 using a motion vector that depends on the identified motion vector candidate. C Block 50 performs motion compensation prediction C Encode the following.

[0148] Although some aspects are described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, with blocks or apparatus corresponding to method steps or features of method steps. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of any of the most important method steps may be performed by such an apparatus.

[0149] Depending on specific implementation requirements, the embodiments of the present invention can be implemented in hardware or in software. The implementation can be carried out using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM or FLASH memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system so that the respective method is executed. The digital storage medium may therefore be computer readable.

[0150] Some embodiments according to the present invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0151] Generally, embodiments of the present invention can be implemented as a computer program product having program code which operates to perform one of the methods when the computer program product is run on a computer. The program code may for example be stored on a machine readable carrier.

[0152] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0153] In other words, a method embodiment of the present invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0154] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium or computer readable medium) comprising recorded thereon the computer program for performing one of the methods described herein. The data carrier, digital storage medium or recording medium is typically tangible and / or non-transitory.

[0155] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals may for example be arranged to be transmitted via a data communication connection, for example via the Internet.

[0156] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0157] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0158] Further embodiments according to the present invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, include a file server for transferring the computer program to the receiver.

[0159] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0160] The above-described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. It is therefore intended that the present invention be limited only by the scope of the appended claims and not by the specific details shown herein as descriptions and illustrations of the embodiments.

[0161] literature [1] ITU-T and ISO / IEC JTC 1, "Advanced video coding for generic audiovisual s ITU-T Recommendation H.264 and ISO / IEC 14496-10 (MPEG-4 AVC), 2010 [2] A. Vetro, T. Wiegand, G.J. Sullivan, “Overview of the Stereo and Multivi ew Video Coding Extension of the H.264 / MPEG-4 AVC Standard", Proceedings of IEEE , vol. 99, no. 4, pp. 626-642, Apr. 2011 [3] H. Schwarz, D. Marpe, T. Wiegand, "Overview of the Scalable Video Coding Extension of the H.264 / AVC Standard", IEEE Transactions on Circuits and Systems for Video Technology, vol. 17, no. 9, pp. 1103-1120, Sep. 2007

Claims

1. 1. An apparatus for reconstructing a multi-view signal encoded in a data stream, comprising: for a coding block of a current picture in a dependent view of the multiview signal to which a default coding mode is applied, estimating a first motion vector based on a second motion vector associated with a reference block in a current picture of a reference view of the multiview signal that corresponds to the coding block of the current picture in the dependent view; adding the first motion vector to a list of candidate motion vector predictors; deriving the list of candidate motion vector predictors based on reference picture indexes transmitted in the data stream, including extracting from the data stream an index indicating a selected motion vector predictor from the list of candidate motion vector predictors; reconstructing the coding block in the current picture of the dependent view by performing motion compensated prediction based on the selected motion vector predictor, the reference picture index and a motion vector difference; An apparatus comprising: a dependent-view reconstructor configured to:

2. 2. The apparatus of claim 1, wherein, according to the encoding mode, multiple motion hypotheses are combined with a reference picture index, a motion vector predictor index, and a motion vector difference associated with each motion hypothesis, and the apparatus is configured to derive a list of candidate motion vector predictors for a given motion hypothesis based on the reference picture index transmitted in the data stream for each motion hypothesis, and to reconstruct the coding block by performing motion compensated prediction based on the multiple motion hypotheses.

3. 3. The apparatus of claim 1, wherein the dependent view reconstructor is configured to obtain a disparity vector representing a disparity between the current picture of the dependent view and the current picture of the reference view with respect to the coding block of the current picture in the dependent view, and to identify the reference block in the current picture of the reference view based on the disparity vector.

4. obtaining motion data associated with the current picture of the reference view; applying the motion data associated with the current picture of the reference view to a depth map estimate of a previous picture of the reference view to generate a first estimated depth map for the current picture of the reference view; deriving a second estimated depth map for the current picture of the dependent view based on the first estimated depth map for the current picture of the reference view; determining the disparity vector based on the second estimated depth map for the current picture of the dependent view; The apparatus of claim 3 , further comprising a depth estimator configured to:

5. 5. The apparatus of claim 4, wherein the disparity vector for the coding block in the dependent view is determined further based on at least one disparity vector associated with one or more previously decoded coding blocks of the multiview signal.

6. The dependent view reconstructor extracting said motion vector differences from said data stream; Obtaining a refined motion vector based on the selected motion vector predictor and the motion vector difference.

6. The apparatus of claim 2, further configured to:

7. The apparatus of claim 1 , configured to decode the index based on context-adaptive binary arithmetic coding.

8. The apparatus of claim 7 , configured to decode the index based on context-adaptive binary arithmetic coding.

9. An apparatus for encoding a multi-view signal into a data stream, comprising: for a coding block of a current picture in a dependent view of the multiview signal to which a default coding mode is applied, estimating a first motion vector based on a second motion vector associated with a reference block in a current picture of a reference view of the multiview signal that corresponds to the coding block of the current picture in the dependent view; adding the first motion vector to a list of candidate motion vector predictors based on a reference picture index; determining a selected motion vector predictor from the list of candidate motion vector predictors; inserting into the data stream an index indicating the selected motion vector predictor and the reference picture index of the list of candidate motion vector predictors for reconstruction of the coding block by performing a motion compensated prediction based on the selected motion vector predictor, the reference picture index and a motion vector difference. An apparatus comprising: a dependent-view encoder configured to:

10. 10. The apparatus of claim 9, wherein, according to the encoding mode, multiple motion hypotheses are combined with a reference picture index, a motion vector predictor index, and a motion vector difference associated with each motion hypothesis, and the apparatus is configured to derive a list of the candidate motion vector predictors for a given motion hypothesis based on the reference picture index, and to transmit the reference picture index to the data stream for each motion hypothesis for reconstruction of the coding block by motion compensated prediction based on the multiple motion hypotheses.

11. 11. The apparatus of claim 9 or 10, wherein the dependent view encoder is configured to obtain, with respect to the coding block of the current picture in the dependent view, a disparity vector representing a disparity between the current picture of the dependent view and the current picture of the reference view, and to identify the reference block in the current picture of the reference view based on the disparity vector.

12. obtaining motion data associated with the current picture of the reference view; applying the motion data associated with the current picture of the reference view to a depth map estimation of a previous picture of the reference view to generate a first estimated depth map for the current picture of the reference view; deriving a second estimated depth map for the current picture of the dependent view based on the first estimated depth map for the current picture of the reference view; and determining the disparity vector based on the second estimated depth map for the current picture of the dependent view; The apparatus of claim 11 , further comprising a depth estimator configured to estimate the disparity vector by:

13. 13. The apparatus of claim 12, wherein the disparity vector for the coding block in the dependent view is determined further based on at least one disparity vector associated with one or more previously decoded coding blocks of the multiview signal.

14. The dependent view encoder comprises: determining the motion vector difference based on a difference between an original motion vector of the coding block and the selected motion vector predictor; Inserting the motion vector differences into the data stream.

14. The apparatus of claim 10 to 13, further configured to:

15. 15. An apparatus according to any of claims 9 to 14, configured to code the index based on context-adaptive binary arithmetic coding.

16. The apparatus of claim 15 , configured to encode the index based on context-adaptive binary arithmetic coding.

17. 1. A method for reconstructing a multi-view signal encoded into a data stream, comprising the steps of: deriving a list of candidate motion vector predictors based on reference picture indexes transmitted in the data stream for a coding block of a current picture in a dependent view of the multiview signal to which a default coding mode is applied, the deriving step comprising: estimating a first motion vector based on a second motion vector associated with a reference block in a current picture of a reference view of the multiview signal that corresponds to the coding block of the current picture in the dependent view; adding the first motion vector to the list of candidate motion vector predictors; and extracting from the data stream an index indicating a selected motion vector predictor from the list of candidate motion vector predictors; reconstructing the coding block in the current picture of the dependent view by performing motion compensated prediction based on the selected motion vector predictor, the reference picture index and a motion vector difference; A method comprising:

18. 1. A method for encoding a multi-view signal into a data stream, comprising: deriving a list of candidate motion vector predictors based on reference picture indexes for a coding block of a current picture in a dependent view of the multiview signal to which a default coding mode is applied, the deriving step comprising: estimating a first motion vector based on a second motion vector associated with a reference block in a current picture of a reference view of the multiview signal that corresponds to the coding block of the current picture in the dependent view; adding the first motion vector to the list of candidate motion vector predictors; and determining a selected motion vector predictor from the list of candidate motion vector predictors; inserting an index into the data stream indicating the selected motion vector predictor and the reference picture index from the list of candidate motion vector predictors to reconstruct the coding block by performing motion compensated prediction based on the selected motion vector predictor, the reference picture index and a motion vector difference; A method comprising:

Images