Depth-color alignment using auxiliary metadata for transcoding of volumetric video
By determining and encoding subset metadata of depth views in multi-view depth images, the problem of complexity of depth map distortion operation in misaligned image processing is solved, and more efficient transcoding and memory management is achieved, improving image quality .
Patent Information
- Application Number
- CN202380032610.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-05
- Filing Date
- 2023-03-23
- Publication Date
- 2024-11-22
AI Technical Summary
The prior art lacks effective auxiliary metadata encoding methods when processing unaligned multi-view depth images, resulting in increased operational complexity, computing burden and memory usage of depth maps, making it difficult to reduce transcoding complexity and improve image quality.
By determining a subset of the depth view and associating its metadata with an unaligned color view, it is encoded in the data stream to limit the number of depth map distortion operations, reducing transcoding complexity and computing requirements.
It effectively reduces the number of depth map distortion operations, reduces calculation and memory requirements, and improves the quality of transcoding depth maps, especially in the encoding and decoding of multi-view depth images.
Smart Images

Figure CN119013964A_ABST
Abstract
Description
Technical Field
[0001] The present principles generally relate to the field of multi-view plus depth (MVD) content, and more particularly, the general principles relate to MVD content acquired by unaligned color and depth sensors.This document should also be understood in the context of encoding and formatting metadata associated with the encoded MVD content. Background Art
[0002] This section is intended to introduce the reader to various aspects of the technology that may be related to various aspects of the present principles described and / or claimed below. This discussion is considered to be helpful in providing the reader with background information to better understand the various aspects of the present principles. Therefore, it should be understood that these statements should be read in this light and not as admissions of prior art.
[0003] Multi-view plus depth (MVD) images are acquired by a set of color and depth sensors located at different points of a 3D scene and facing the 3D scene. The result of this acquisition is a set of color images and depth maps. The color and depth sensors may be aligned (i.e., the depth sensor and the color camera have a common focus), or they may be misaligned. The number of depth sensors may be different from the number of color cameras. Since a set of color images and depth maps represents a large amount of data, methods that reduce the size of the data are often used to encode and compress MVD images. For example, encoding MVD images as a pair of face atlases is a technique that allows redundant information between (color and / or depth) views to be removed and the atlas to be compressed using image or video compression methods. Unaligned MVD images are easier to capture and easier to encode than aligned MVD images that require high-tech camera rigs and generate more depth maps.
[0004] For example, MVD images are used to synthesize virtual views of a scene (views that do not correspond to one of the camera's views). View synthesis requires depth map warping and color blending operations. Encoded unaligned MVD images can also be converted into aligned MVD images. This process includes synthesizing a depth map for each color image, i.e., processing a warp for each unaligned depth map for each color image. However, each depth map does not contribute to view synthesis at the same level depending on the position of the sensor view and the position of the view to be synthesized. There is a lack of a technology to encode auxiliary metadata associated with the encoded MVD images to limit the number of depth map warping operations in order to reduce the complexity, computational burden and memory usage of transcoding. Summary of the invention
[0005] The following is a brief summary of the present principles to provide a basic understanding of some aspects of the present principles. This summary is not a comprehensive overview of the present principles. It is not intended to identify the key or critical elements of the present principles. The following summary only presents some aspects of the present principles in a simplified form as a prelude to a more detailed description below.
[0006] This principle relates to a method comprising:
[0007] - obtaining a multi-view plus depth image including a color view and a depth view. At least one color view is not aligned with the depth view, i.e., there is no depth view having the same focus as the color view;
[0008] - for an unaligned color view, determining a subset of depth views. The depth views of the subset contribute to generating a new depth view aligned with the unaligned color view; and
[0009] - encoding the multi-view plus depth image and metadata representing a subset of the first depth view in a data stream, wherein the metadata is associated with the unaligned color views.
[0010] The present principles also relate to a processing device comprising a memory associated with a processor configured to implement the above-described method.
[0011] The present principles also relate to a method comprising:
[0012] - decoding a multi-view plus depth image from a data stream, the multi-view plus depth image comprising a color view and a depth view. At least one color view is an unaligned color view, i.e., a color view that does not share its focus with a depth view. The data stream also comprises metadata representing subsets of the depth views, wherein each subset is associated with a color view;
[0013] - For an unaligned color view, generating a depth view aligned with the unaligned color view by warping a subset of the depth views to the unaligned color view.
[0014] The present principles also relate to a processing device comprising a memory associated with a processor configured to implement the above-mentioned method.
[0015] The present principles also relate to a data stream that carries data representing:
[0016] - a multi-view-plus-depth image comprising a color view and a depth view, wherein at least one of the unaligned color views is unaligned with the depth view; and
[0017] - Metadata representing subsets of depth views, where each subset is associated with an unaligned color view. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The present disclosure will be better understood, and other specific features and advantages will become apparent, upon reading the following description, which refers to the accompanying drawings, in which:
[0019] - Figure 1 shows an example camera array layout including four depth sensors placed around a three-by-three color camera arrangement;
[0020] - Figure 2A illustrate Figure 1 The contribution of the depth map and color image captured by the sensor to the synthesized view;
[0021] - Figure 2B illustrate Figure 1 The potential contribution of the depth map of to the depth maps associated with the nine color views;
[0022] - Figure 3 shows an example architecture of a processing engine that may be configured to implement a method according to the present principles;
[0023] - Figure 4 An example of an embodiment of a data stream syntax for encoding an unaligned MVD image or a sequence of unaligned MVD images according to the present principles is shown. DETAILED DESCRIPTION
[0024] The present principles will be described more fully below with reference to the accompanying drawings, in which examples of the present principles are shown. However, the present principles can be implemented in many alternative forms and should not be construed as being limited to the examples described herein. Therefore, although the present principles are susceptible to various modifications and alternative forms, specific examples thereof are shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that the present invention is not intended to limit the present principles to the specific forms disclosed, but rather, the present disclosure is intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the present principles as defined by the claims.
[0025] The terms used herein are only used to describe specific examples and are not intended to limit the present principle. The singular forms "a", "an" and "the" used herein are also intended to include plural forms, unless the context clearly states otherwise. It should be further understood that when the terms "comprises", "comprising", "includes" and / or "including" are used in this specification, the existence of the features, integers, steps, operations, elements and / or components is specified, but the existence or addition of one or more other features, integers, steps, operations, elements, components and / or their combinations is not excluded. In addition, when it is mentioned that an element "responds to" or "is connected to" another element, it can directly respond to or be connected to another element, or there may be an intermediate element. On the contrary, when it is mentioned that an element "directly responds to" or "is directly connected to" another element, there is no intermediate element. As used herein, the term "and / or" includes any and all combinations of one or more related listed items, and can be abbreviated as " / ".
[0026] It should be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first element may be referred to as the second element, and similarly, the second element may be referred to as the first element without departing from the teaching of the present principle.
[0027] Although some diagrams include arrows on communication paths to show a primary direction of communication, it is understood that communication can occur in the opposite direction of the arrows shown.
[0028] Some examples are described with respect to block diagrams and operational flow charts, where each block represents a circuit element, module, or portion of code that includes one or more executable instructions for implementing a specified logical function. It should also be noted that in other embodiments, the functions shown in the blocks may not occur in the order shown. For example, two blocks shown in succession may actually be executed substantially simultaneously, or may sometimes be executed in reverse order, depending on the functions involved.
[0029] References herein to "according to an example" or "in an example" mean that a particular feature, structure, or characteristic described in connection with the example may be included in at least one embodiment of the present principles. The phrases "according to an example" or "in an example" appearing in multiple places in the specification do not necessarily all refer to the same example, nor are separate or alternative examples mutually exclusive of other examples.
[0030] Reference signs appearing in the claims are for illustration purposes only and shall not have a limiting effect on the scope of the claims.The examples and variations may be used in any combination or sub-combination even if not explicitly described.
[0031] Figure 1 An example camera array layout is shown, which includes four depth sensors placed around a three-by-three color camera arrangement. The principles of the present invention are applicable to any other arbitrary number and relative positions of color and depth sensors. Figure 1 In the example of FIG. 1 , color cameras Vc1 to Vc9 are not aligned with depth sensors Vd1 to Vd4 . The number of depth sensors is smaller than the number of color cameras. Figure 1 The positions and orientations of a set of color cameras, each capturing a color image, and the positions and orientations of a set of depth sensors, each capturing a depth map, are diagrammatically shown.
[0032] Figure 2A illustrate Figure 1 The contributions of the depth map and color image captured by the sensors in the synthesized view SV. To synthesize the view SV, the renderer backprojects each (u,v) pixel of the decoded depth map to its (x,y,z) 3D position and reprojects the corresponding color attributes of the color image to the corresponding pixel coordinates (u',v') in the virtual viewport reference frame (i.e., the pixel array of the view to be synthesized). This backprojection plus reprojection process is not simple when the depth map (according to the associated depth sensor) is not aligned with the color image (according to the associated color camera). In fact, the color value of the point (x,y,z) (obtained by backprojecting the pixels of at least one depth map) is not directly available. Existing view rendering algorithms adopt a two-stage visibility and coloring scheme. First, a viewport depth map (also called a visibility map) is generated by warping all depth maps to the viewport, selecting the best depth candidate for each pixel, and filtering the generated depth map. Then, according to the visibility map, the color of the viewport is generated by mixing the colors of the source views. This two-stage approach to view rendering does not require the depth and color source views to be spatially aligned. However, the process of blending colors on a misaligned visibility map is more complicated than the process of blending colors on an aligned visibility map.
[0033] In some use cases, the MVD source must be regenerated on the decoder side and transcoded into an MVD signal with aligned color and depth views. For example, such transcoding may be required when inputting an autostereoscopic multi-view display. It may also be necessary to generate aligned MVD images when transcoding, for example, transcoding an MIV (MPEG Immersive Video) signal to other 3D compression formats that assume that the color and depth components are spatially aligned, such as 3D-HEVC (High Efficiency Video Coding). The generation of a depth map aligned with the decoded color view involves depth warping, selection and filtering operations similar to the visibility stage of the rendering algorithm. However, since it is not a single viewport depth map that is to be calculated (but a depth map corresponding to each source color view), the computational requirements increase proportionally with the number of color cameras.
[0034] Figure 2B The potential contribution of the depth map from the top left corner depth sensor Vd1 to the depth maps associated with the nine color views is illustrated. The top left corner depth sensor Vd1 may contribute primarily to the 4 nearby color views (Vc1, Vc2, Vc4, Vc5) (rather than to the other 5 color views), and the number of depth warping operations can be reduced to the closest color views. The operation may be more complex depending on the 3D geometry of the captured scene and the position and orientation of the depth sensor.
[0035] According to the present principles, the unaligned MVD images are encoded with auxiliary metadata that provides information to the decoder to limit the number of depth map warping operations, thereby reducing the complexity, computational burden, and memory footprint of transcoding. The greater the number of acquisition cameras (e.g., approximately 30 color cameras around a basketball court when recording a game), the more useful and relevant such auxiliary metadata is to the transcoding process.
[0036] The present principles relate to the transmission of a static or dynamic 3D scene ingested by a volumetric video encoder (e.g., a MIV encoder) in a multi-view plus depth (MVD) representation using spatially unaligned color and depth views originating from non-co-located color and depth sensors.
[0037] According to the present principles, for each source color view not associated with a corresponding aligned depth map, the auxiliary metadata indicates a subset of source depth views that must be considered to generate a depth map aligned with each given color view.
[0038] These auxiliary metadata allow minimizing the computation and caching requirements of transcoding on the decoder side by reducing the number of depth map warping operations (shifting memory and computation requirements to the encoder side), and improving the quality of the transcoded depth maps by reducing the risk of errors when choosing among different depth candidate values originating from different depth sensors, since a smaller subset of more reliable candidates is provided.
[0039] According to the present principles, for each color view, a subset of source depth views that may contribute to generating an aligned depth map is determined. The following method is proposed:
[0040]
[0041] For each source color view Vc i , i∈[1,9], the output of this analysis stage is the depth view Vd j The number N i and index {Idx k , k∈{1,N i}}, whose depth map may bei In the example layout, N i In [0,4], Idx k Within [1,4].
[0042] In a first embodiment, auxiliary metadata is provided using a modified 3D transmission format.The considered 3D transmission format is MIV, but the present principles are applicable to any other 3D format capable of ingesting MVD input with misaligned color and depth views.
[0043] According to the present principles, data structures such as miv_view_params_list() of MIV contain auxiliary metadata, which carries global information about camera parameters. Similar to what is done for camera extrinsic parameters, camera intrinsic parameters and depth quantization parameters, a dynamic temporal update mechanism can also be implemented for depth warp parameters.
[0044]
[0045] casme_unaligned_depth_flag equal to 1 specifies that, in the view parameter list representing the volumetric frame, some views contain geometry but no texture, while some other views with different camera parameters contain texture but no geometry. casme_unaligned_depth_equal_flag equal to 0 specifies that, when video geometry data is present, views that contain texture also contain geometry.
[0046]
[0047] came_update_depth_warping_flag is equal to 1, indicating that the miv_view_params_update_depth_warping() syntax structure is present in this syntax structure. came_update_depth_warping_flag is equal to 0, indicating that the miv_view_params_update_depth_warping() syntax structure is not present in this syntax structure. When not present, the value of came_update_depth_warping_flag is inferred to be equal to 0.
[0048]
[0049] mvp_num_texture_only_views_minus1 plus 1 specifies the number of views that contain textures but no geometry. The value of mvp_num_texture_only_views_minus1 should be in the range of 0 to mvp_num_views_minus1, inclusive.
[0050] mvp_texture_only_view_idx[v] specifies the index of the view with index v in the sublist of views that contain textures but not geometry, among all view lists representing the volume frame. The value of mvp_texture_only_view_idx[v] shall be in the range of 0 to mvp_num_views_minus1, inclusive.
[0051] dw_num_depth_views_minus1[i] plus 1 specifies the number of views that contain geometry that may contribute to the geometry of the view of index i. The value of dw_num_depth_views_minus1[v] should be in the range of 0 to mvp_num_views_minus1, inclusive.
[0052] dw_depth_view_idx[i][k] specifies the index of the view with index k in the view sublist that contains geometry and may contribute to the geometry of the view with index i, among all view lists representing the volumetric frame. The value of dw_depth_view_idx[i][k] shall be in the range of 0 to mvp_num_views_minus1, inclusive.
[0053]
[0054] mvpudw_num_view_updates_minus1 plus 1 specifies the number of depth_warping(i) syntax structures present in this syntax structure. The value of mvpudw_num_view_updates_minus1 should be in the range of 0 to mvp_num_texture_only_views_minus1, inclusive.
[0055] mvpudw_view_idx[i] specifies the view index for which updated depth warp parameters are sent. The value of mvpudw_view_idx[i] shall be in the range of 0 to mvp_num_views_minus1, inclusive, and shall be equal to one of the values of mvp_texture_only_view_idx[v], where v is in the range of 0 to mvp_num_texture_only_views_minus1. Bitstream conformance of this version of this document requires that for all j != k, the value of mvpudw_view_idx[j] shall not be equal to mvpudw_view_idx[k].
[0056] In the second embodiment, the signaling of the depth warp assistance metadata is carried by an optional Supplemental Enhancement Information (SEI) message, whose syntax and semantics are similar to those in the first embodiment. The SEI message can be inserted at any synthesis time in the bitstream, and as long as the depth warp information is not refreshed by a new SEI message, it is valid for the rest of the sequence.
[0057]
[0058]
[0059] At the decoder side, auxiliary metadata is used to generate depth views aligned with color views. The output depth map spatially aligned with the color views is computed using the visibility step of the view rendering algorithm. According to the present principles, the method takes as input only a subset of the decoded depth views indicated by metadata signaling.
[0060]
[0061] Figure 3 FIG. 3 shows an example architecture of a processing engine 30 that can be configured to implement the methods described herein. Figure 3 The device of the architecture is linked to other devices via its bus 31 and / or via the I / O interface 36 .
[0062] The device 30 comprises the following elements linked together by a data and address bus 31:
[0063] - a microprocessor 32 (or CPU), such as a DSP (or digital signal processor);
[0064] - ROM (or read-only memory) 33;
[0065] - RAM (or random access memory) 34;
[0066] - storage interface 35;
[0067] - an I / O interface 36 for receiving data to be transmitted from an application; and
[0068] -power supply( Figure 3 not shown), such as batteries.
[0069] According to an example, the power supply is located outside the device. In each memory mentioned, the word "register" used in the specification can correspond to a small capacity (a few bits) area or a very large area (such as an entire program or a large amount of received or decoded data). ROM 33 includes at least programs and parameters. ROM 33 can store algorithms and instructions for executing the technology according to the present principles. When turned on, CPU 32 loads the program into RAM and executes the corresponding instructions.
[0070] RAM 34 includes in registers the program executed by CPU 32 and loaded after device 30 is turned on, input data in registers, intermediate data at different states of the method in registers, and other variables in registers used to execute the method.
[0071] The embodiments described herein can be implemented, for example, in methods or processes, devices, computer program products, data streams or signals. Even if only discussed in the context of a single form embodiment (e.g., discussed only as a method or device), the implementation of the discussed features can also be implemented in other forms (e.g., programs). Devices can be implemented, for example, with appropriate hardware, software, and firmware. Methods can be implemented, for example, with devices (such as processors, which generally refer to processing equipment, such as computers, microprocessors, integrated circuits, or programmable logic devices). Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs") and other devices that facilitate information communication between end users.
[0072] The device 30 is linked to a set of sensors 37 and a set of rendering devices 38, for example, via a bus 31. The sensor 37 may be, for example, a camera, a microphone, a temperature sensor, an inertial measurement unit, a GPS, a humidity sensor, an infrared or ultraviolet light sensor, or a wind sensor. The rendering device 38 may be, for example, a display, a speaker, a vibrator, a heater, a fan, etc.
[0073] According to an example, the device 30 is configured to implement the method according to the present principles described in FIG. 5 and belongs to the following set:
[0074] -mobile device;
[0075] -communication equipment;
[0076] -Gaming equipment;
[0077] - Tablet computer (or tablet computer);
[0078] - Laptop;
[0079] -Still picture camera;
[0080] -Camcorder.
[0081] Figure 4 An example of an embodiment of a data stream syntax for encoding an unaligned MVD image or an unaligned MVD image sequence according to the present principles is shown. The structure includes a container that organizes the stream into independent syntax elements. The structure may include a header portion 41, which is a set of data common to each syntax element of the stream. For example, the header portion includes some metadata about the syntax elements, describing the properties and functions of each syntax element. The structure also includes a payload, which includes syntax element 42 and syntax element 43. Syntax element 42 includes data representing an unaligned MVD image, namely a color view and a depth map (also called a depth view). The image may have been compressed according to a compression method. Syntax element 43 is part of the data stream payload and includes data for encoding auxiliary metadata described in accordance with the present principles. An auxiliary metadata item refers to an unaligned color view of an MVD image and includes a subset of the depth view of the MVD image.
[0082] The implementation described herein can be implemented, for example, in a method or process, a device, a computer program product, a data stream or a signal. Even if only discussed in the context of a single form embodiment (e.g., discussed only as a method or device), the implementation of the discussed features can also be implemented in other forms (e.g., programs). The device can be implemented, for example, with appropriate hardware, software, and firmware. The method can be implemented, for example, with a device (such as a processor, which generally refers to a processing device, such as a computer, a microprocessor, an integrated circuit, or a programmable logic device). The processor also includes a communication device, such as a smart phone, a tablet computer, a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.
[0083] The implementation of the various processes and features described herein may be embodied in a variety of different devices or applications, particularly, for example, devices or applications related to data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and / or depth information. Examples of such devices include encoders, decoders, post-processors for processing outputs from decoders, pre-processors for providing inputs to encoders, video encoders, video decoders, video codecs, network servers, set-top boxes, laptops, personal computers, mobile phones, PDAs, and other communication devices. It should be clear that the device can be mobile and can even be installed in a moving vehicle.
[0084] Furthermore, the methods may be implemented by instructions executed by a processor, and the instructions (and / or data values resulting from the implementation) may be stored on a processor-readable medium, such as an integrated circuit, a software carrier, or other storage device, such as a hard disk, a compact disk ("CD"), an optical disk (such as a DVD, commonly referred to as a digital versatile disk or digital video disk), a random access memory ("RAM"), or a read-only memory ("ROM"). The instructions may form an application tangibly embodied on a processor-readable medium. The instructions may, for example, be hardware, firmware, software, or a combination thereof. The instructions may, for example, be found in an operating system, a separate application, or a combination of both. Thus, a processor may, for example, be characterized as a device configured to perform a process and a device including a processor-readable medium (e.g., a storage device) having instructions for performing the process. Furthermore, the processor-readable medium may store data values resulting from the implementation in addition to or in lieu of the instructions.
[0085] It will be apparent to one skilled in the art that an implementation may produce signals that are formatted in a variety of ways to carry information that can be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the syntax of the embodiment, or to carry as data the actual syntax values written by the embodiment. Such a signal may, for example, be formatted as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting may, for example, include encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may, for example, be analog or digital information. As is well known, signals may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.
[0086] Many embodiments have been described. However, it should be understood that a variety of modifications may be made. For example, elements of different embodiments may be combined, supplemented, modified, or removed to produce other embodiments. In addition, it will be understood by those of ordinary skill in the art that other structures and processes may be substituted for the disclosed structures and processes, and that the resulting embodiments will perform at least substantially the same functions in at least substantially the same manner to achieve at least substantially the same results as the disclosed embodiments. Therefore, the present application includes these and other embodiments.
Claims
1. A method comprising: --obtaining a multi-view plus depth image comprising a color view and a first depth view, wherein at least one color view is not aligned with the first depth view; - for an unaligned color view, determining a subset of the first depth view, wherein the subset of the first depth view contributes to generating a new depth view aligned with the unaligned color view; as well as - encoding the multiview-plus-depth image and metadata representative of the subset of the first depth view in a data stream.
2. The method of claim 1 , wherein the subset of the first depth view is determined by: - for each first depth view, generating a second depth view warped to said unaligned color view; - for each pixel of the unaligned color view, counting the contribution of each second depth view to the new depth view; as well as - selecting the subset of the first depth views according to the counted contributions of the respective second depth views. The method according to claim 2 , wherein if the contribution of the second view is greater than a given contribution value, the corresponding first depth view is selected. 4 . The method according to claim 1 , wherein a subset of the first depth view is determined for each non-aligned color view of the multi-view plus depth image.
5. A device comprising a memory associated with a processor, the processor being configured to: - acquiring a multi-view plus depth image comprising a color view and a first depth view, wherein at least one color view is not aligned with the first depth view; - for an unaligned color view, determining a subset of the first depth view, wherein the subset of the first depth view contributes to generating a new depth view aligned with the unaligned color view; and - encoding the multiview-plus-depth image and metadata representative of the subset of the first depth view in a data stream.
6. The apparatus of claim 5, wherein the subset of the first depth view is determined by: - for each first depth view, generating a second depth view warped to said unaligned color view; - for each pixel of the unaligned color view, counting the contribution of each second depth view to the new depth view; and - selecting the subset of the first depth views according to the counted contributions of the respective second depth views. The apparatus according to claim 6 , wherein if the contribution of the second view is greater than a given contribution value, the corresponding first depth view is selected.
8. The apparatus of any one of claims 5 to 7, wherein a subset of the first depth view is determined for each non-aligned color view of the multi-view plus depth image.
9. A method comprising: - decoding a multi-view-plus-depth image from a data stream, the multi-view-plus-depth image comprising color views and depth views, wherein at least one unaligned color view is unaligned with a depth view, and metadata representing subsets of the depth views, wherein each subset is associated with an unaligned color view; as well as - for the at least one unaligned color view, generating a depth view aligned with the at least one unaligned color view by warping the depth views of the subset to the unaligned color view.
10. An apparatus comprising a memory associated with a processor, the processor configured to: - decoding a multi-view-plus-depth image from a data stream, the multi-view-plus-depth image comprising a color view and a depth view, wherein at least one unaligned color view is unaligned with a depth view, and metadata representing subsets of the depth views, wherein each subset is associated with an unaligned color view; and - for the at least one unaligned color view, generating a depth view aligned with the at least one unaligned color view by warping the depth views of the subset to the unaligned color view.
11. A data stream carrying data representing: - a multi-view plus depth image comprising a color view and a depth view, wherein at least one of the unaligned color views is unaligned with the depth view; and - Metadata representing subsets of depth views, where each subset is associated with an unaligned color view.
