Image processing apparatus and method

Abstract

The present disclosure relates to an image processing device and method that make it possible to suppress a reduction in the quality of an image obtained by rendering 3D data including a plurality of layer images. The present invention performs: generating inter-layer relative information on a relative relationship between layers of 3D data including a layer image of a color component and a layer image of an alpha component; and encoding the 3D data to generate a bitstream, and adding the inter-layer relative information to the bitstream. The present invention also performs: decoding the bitstream to generate 3D data; analyzing the interlayer relative information extracted from the bitstream; and rendering the 3D data based on the inter-layer relative information and generating a viewpoint image as viewed from any viewpoint. The present disclosure can be applied to, for example, an image processing apparatus, an electronic device, an image processing method, and a program.

Description

Technical Field

[0001] This disclosure relates to image processing apparatus and methods, and more specifically, to image processing apparatus and methods capable of suppressing the degradation of image quality obtained by rendering 3D data comprising multiple layers of images. Background Technology

[0002] In related technologies, NeRF (Representing Scenes as Neural Radiance Fields for View Synthesis) has been developed as a method for representing the three-dimensional shape of an object. In NeRF, a radiance field corresponding to the space including the object is generated, and the radiance field is approximated by a neural network (see, for example, Non-Patent Documents 1 to 3). NeRF uses an MLP (Multilayer Perceptron) to represent the spatial information of the 3D object.

[0003] In addition, there is already a method called MPI (Multi-Plane Image) that divides three-dimensional space into multiple layers (2D images) and represents them. In order to accelerate the rendering process of NeRF, a method called NeX has been considered to convert MLP into MPI and process it (see, for example, Non-Patent Document 4). MPI is a group of images in which multiple 2D images are arranged in a layer shape, and 3D data can be processed as 2D images.

[0004] In addition, a method has been proposed to extend the specification of MPI by using SEI (Supplemental Enhancement Information) to the bitstream obtained by encoding MPI (see, for example, Non-Patent Literature 5). Citation List

[0005] Non-patent literature

[0006] Non-patent literature 1: Ben Mildenhall, Pratul P.Srinivasan, Matthew Tancik, Jonathan T.Barron, Ravi Ramamoorthi, Ren Ng, "NeRF: Representing scenes asneural radiance fields for view synthesis", ECCV 2020, 2020 / 3 / 19.

[0007] Non-Patent Literature 2: Thomas Muller, Alex Evans, Christoph Schied, Alexander Keller, “Instant neural graphics primitives with a multiresolution hashencoding”, arXiv preprint arXiv:2201.05989, 2022 / 1 / 16.

[0008] Non-patent document 3: Ricardo Martin-Brualla, Noha Radwan, Mehdi SMSajjadi, Jonathan T.Barron, Alexey Dosovitskiy, Daniel Duckworth, "NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections", https: / / arxiv.org / abs / 2008.02268, 2020 / 8 / 5 .

[0009] Non-Patent Literature 4: Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, “NeX: Real-time View Synthesis with Neural Basis Expansion”, arXiv: 2103.05606v2 [cs.CV] April 12, 2021.

[0010] Non-Patent Literature 5: Taoran Lu, Peng Yin, Guan-Ming Su, Dae Yeol Lee, Tsung-wei Huang, Sean McCarthy, Walt Husak, Gary J. Sullivan: AHG9: Multiplane Image Information SEI, JVET-AE0066-v5, 5th edition, 2023-07-15. Summary of the Invention

[0011] Technical issues

[0012] However, even with the extensions described in Non-Patent Document 5, there is still a problem that the image quality obtained by rendering 3D data is reduced due to the constraints on the characteristics of the MPI layer.

[0013] This disclosure is made in view of such circumstances, and its purpose is to enable the suppression of image quality degradation when rendering 3D data that includes multiple layers of images.

[0014] Solution to the problem

[0015] According to one aspect of the present technology, an image processing apparatus includes: an interlayer relative information generation unit that generates interlayer relative information about the relative relationships between layers of 3D data, the 3D data including layer images of color components and layer images of alpha components; and an encoding unit that generates a bitstream by encoding the 3D data and adds the interlayer relative information to the bitstream.

[0016] According to another aspect of the present technology, an image processing method includes: generating interlayer relative information about the relative relationships between layers of 3D data, said 3D data including layer images of color components and layer images of alpha components; and generating a bitstream by encoding the 3D data and adding the interlayer relative information to the bitstream.

[0017] According to another aspect of the present technology, an image processing apparatus includes: a decoding unit that decodes a bitstream and generates 3D data including layer images of color components and layer images of alpha components; an interlayer relative information analysis unit that analyzes interlayer relative information about the relative relationships between layers of the 3D data extracted from the bitstream; and a viewpoint image generation unit that renders the 3D data based on the interlayer relative information and generates a viewpoint image viewed from any viewpoint.

[0018] According to another aspect of the present invention, an image processing method includes: decoding a bitstream and generating 3D data comprising layer images including color components and alpha components; analyzing interlayer relative information about the relative relationships between layers of the 3D data extracted from the bitstream; and rendering the 3D data based on the interlayer relative information, and generating a viewpoint image viewed from any viewpoint.

[0019] In an image processing apparatus and method according to one aspect of the present technology, the following operations are performed: generating interlayer relative information about the relative relationships between layers of 3D data, the 3D data including layer images of color components and layer images of alpha components; generating a bitstream by encoding the 3D data; and adding the interlayer relative information to the bitstream.

[0020] In other aspects of the image processing apparatus and method according to the present technology, the following operations are performed: decoding a bitstream and generating 3D data including layer images of color components and layer images of alpha components; analyzing interlayer relative information about the relative relationships between layers of the 3D data extracted from the bitstream; and rendering the 3D data based on the interlayer relative information and generating a viewpoint image viewed from any viewpoint. Attached Figure Description

[0021] [ Figure 1 ] Figure 1 It is a graph used to describe 3D data using MPI.

[0022] [ Figure 2 ] Figure 2 It is a graph used to describe 3D data using MPI.

[0023] [ Figure 3 ] Figure 3 This is a diagram used to describe NeX.

[0024] [ Figure 4 ] Figure 4 This is a diagram used to describe NeX.

[0025] [ Figure 5 ] Figure 5 This is a diagram used to describe NeX.

[0026] [ Figure 6 ] Figure 6 This is a diagram used to describe NeX.

[0027] [ Figure 7 ] Figure 7 This is a diagram illustrating an example of a method for extending the relative relationships between layers.

[0028] [ Figure 8 ] Figure 8 This is a diagram illustrating an example of the grammar.

[0029] [ Figure 9 ] Figure 9 This is a diagram used to describe the packing method for layer images.

[0030] [ Figure 10 ] Figure 10 This is a diagram used to describe the packing method for layer images.

[0031] [ Figure 11 ] Figure 11 It is an extended graph used to describe the ratio of the number of layers between the color component layers and the alpha component layers.

[0032] [ Figure 12 ] Figure 12It is an extended graph used to describe the resolution ratio between layers representing the color components and layers representing the alpha components.

[0033] [ Figure 13 ] Figure 13 It is an extended graph used to describe the ratio of the number of layers to the resolution ratio between the layers representing the color components and the layers representing the alpha components.

[0034] [ Figure 14 ] Figure 14 It is an extended graph used to describe the ratio of the number of layers to the resolution ratio between the layers representing the color components and the layers representing the alpha components.

[0035] [ Figure 15 ] Figure 15 This is a diagram illustrating an example of a method for extending the layer layout.

[0036] [ Figure 16 ] Figure 16 This is a diagram showing an example of an additional layer.

[0037] [ Figure 17 ] Figure 17 This is a diagram illustrating an example of a non-equally spaced arrangement function.

[0038] [ Figure 18 ] Figure 18 This is a diagram illustrating an example of control information regarding the layer layout.

[0039] [ Figure 19 ] Figure 19 This is a diagram illustrating an example of a method for expanding the shape of a layer.

[0040] [ Figure 20 ] Figure 20 This is a diagram showing an example of layer shape.

[0041] [ Figure 21 ] Figure 21 This is a diagram illustrating an example of control information regarding the layer shape.

[0042] [ Figure 22 ] Figure 22 This is another example of a diagram showing control information regarding the layer shape.

[0043] [ Figure 23 ] Figure 23 This is a block diagram illustrating an example of the main configuration of the encoding device.

[0044] [ Figure 24 ] Figure 24 This is a flowchart illustrating an example of the encoding process.

[0045] [ Figure 25 ] Figure 25This is a block diagram illustrating an example of the main configuration of the decoding device.

[0046] [ Figure 26 ] Figure 26 This is a flowchart illustrating an example of the decoding process.

[0047] [ Figure 27 ] Figure 27 This is a block diagram illustrating an example of the main configuration of a computer. Detailed Implementation

[0048] In the following text, a mode for implementing the present disclosure (hereinafter referred to as "implementation mode") will be described. Note that the description will be given in the following order.

[0049] 1. Supporting documents containing technical content and terminology.

[0050] 2. Utilizing MPI's 3D data

[0051] 3. Regarding the expansion of layers

[0052] 4. First Embodiment (Encoding Device)

[0053] 5. Second Implementation Method (Decoding Device)

[0054] 6. Supplementary Explanation

[0055] <1. Supporting documents containing technical content, technical terminology, etc.>

[0056] The scope of this technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents known at the time of submission, and the content of other documents cited in the following non-patent documents.

[0057] Non-patent literature 1: (as described above)

[0058] Non-patent literature 2: (as described above)

[0059] Non-patent literature 3: (as described above)

[0060] Non-patent literature 4: (as described above)

[0061] Non-patent literature 5: (as described above)

[0062] That is, the content described in the aforementioned non-patent documents, as well as the content of other documents cited in the aforementioned non-patent documents, are also used as the basis for determining whether the support requirements are met.

[0063] <2. Utilizing MPI 3D Data>

[0064] <nerf>

[0065] In related technologies, NeRF (representing a scene as a neural radiation field for view synthesis) has been used as a method for representing the three-dimensional shape of an object. In NeRF, a radiation field corresponding to the space including the object is generated, and the radiation field is approximated by a neural network, as described in Non-Patent Documents 1 to 3.

[0066] A radiation field represents a scene in space by including the color in each direction at each location within that space, as well as the opacity at each location within that space. Here, opacity is an indicator of the presence of an object and can also be considered as density. That is, the shape of a 3D space can be represented by obtaining a radiation field with high density at the coordinates of the object. NeRF uses neural networks to approximate such a radiation field. In other words, NeRF is a technique that generates a radiation field corresponding to a 3D object, approximates the radiation field with a neural network, and performs rendering using a neural network.

[0067] In NeRF, objects in 3D space are picked up from different locations, and the camera device pose of the acquired images is obtained. The neural network is then trained using the picked images and the camera device pose to generate a neural network (MLP (Multilayer Perceptron)) that approximates the 3D shape of the object. During inference, the pose information of the desired new viewpoint is input into the neural network to obtain a rendered image of that new viewpoint. In this specification, such pose information for a specific viewpoint (any viewpoint) is also referred to as viewpoint information. Furthermore, the rendered image of a specific viewpoint (any viewpoint) is also referred to as a viewpoint image.

[0068] <nex>

[0069] Furthermore, a method called MPI (Multi-Plane Image) already exists that represents three-dimensional space by dividing it into multiple layer images (2D images). In Non-Patent Document 4, a method called NeX was proposed to accelerate NeRF rendering by converting MLP into MPI and processing it. MPI is an image group in which multiple 2D images are arranged in a layer shape, and 3D data can be processed as 2D images.

[0070] In MPI, for example, Figure 1 Layers 11 to 14 in the MPI form multiple layers (2D planes) in three-dimensional space, and each layer represents an object. Each layer is 2D data representing a 2D planar distribution of predetermined information, and therefore can also be considered an image. In this specification, the layer is also referred to as a layer image. The MPI includes multiple RGBA images. An RGBA image is a combination of images of color components (RGB) and alpha components (Alpha). The alpha component is the component that indicates the presence of an object. The alpha component can represent the presence of an object by values ​​"0" and "1", or it can represent the probability of the object's presence (or the probability of the object's non-existence). In other words, the alpha component can also be considered as transmittance. RGBA images are generated by learning from multi-view images using, for example, a DNN (Deep Neural Network). That is, the MPI is 3D data including layer images of color components and layer images of alpha components.

[0071] The viewpoint image (the rendered image for a specified viewpoint) shows the composite result (overlay result) of objects from each layer as seen from the viewpoint. For example, as Figure 1 As shown, assume that object 21 exists in layer 11, object 22 exists in layer 12, object 23 exists in layer 13, and object 24 exists in layer 14. In the rendered image, these objects are superimposed according to their positional relationship when viewed from the viewpoint. Figure 2 The rendered image 41 in the middle shows Figure 1 An example of a rendered image from viewpoint 31, and rendered image 42 shows... Figure 1 The rendered image of viewpoint 32. The three-dimensional effect (the three-dimensional shape of the objects) is expressed by changing the overlapping method of each object in objects 21 to 24 according to the viewpoint position, as described above.

[0072] In NeX, the NeRF MLP is transformed into an MPI (also known as NeX MPI) for processing. That is, the three-dimensional space is represented by multiple layers. Figure 3 An example of a layer configuration for NeX is shown. In the case of NeX MPI, multiple main layers 51 are formed, such as main layer 51-1, main layer 51-2, ..., and multiple sub-layers 52, such as sub-layers 52-1 to sub-layers 52-11, sub-layers 52-21, ..., are further formed between the main layers 51. That is, in a layer group representing three-dimensional space, some layers at predetermined intervals form main layers 51, and the remaining layers between these main layers form sub-layers 52.

[0073] The number of layers in the layer group, the number of main layers, and the number of sub-layers between the main layers can all be freely set. For example, a three-dimensional space (one scene) at a certain moment can be represented by 192 layers, and one layer out of every 12 layers (i.e., a total of 16 layers) can be set as the main layer, and the remaining 176 layers can be set as sub-layers. In this case, one layer out of 12 consecutive layers in the layer group is the main layer, and the remaining 11 layers are sub-layers. Furthermore, for example, the number of main layers can be set to 32. In this case, the number of layers in the layer group representing a scene is 384. Of course, this embodiment is not limited to these examples.

[0074] In the case of NeX, such as Figure 4 As shown, the data includes alpha image 71, RGB_base image 72, RGB_view image 73, and base image 74. Alpha image 71 is the distribution information of the alpha component indicating the presence of an object in a single layer of the layer group representing three-dimensional space. That is, alpha image 71 indicates the location where the object exists in that layer. Note that alpha image 71 can indicate the probability of the object's presence. Furthermore, alpha image 71 can indicate the location where the object does not exist in that layer, and can indicate the probability that the object does not exist. In other words, alpha image 71 can indicate the transmittance at each location in the layer. Alpha image 71 is generated for each sub-layer. That is, for all layers of the layer group representing three-dimensional space (… Figure 3 Each layer in the main layer 51 and sub-layer 52 generates an alpha image 71.

[0075] RGB_base image 72 and RGB_view image 73 are information indicating the distribution of RGB components, which indicate the color information of an object in multiple consecutive layers within a layer group. Note that the RGB_base image and RGB_view image are also collectively referred to as RGB images.

[0076] RGB_base image 72 contains color distribution information that is independent of the viewpoint angle. RGB_base image 72 comprises a single image. In other words, RGB_base image 72 contains color distribution information with strong color components when viewed from the front of the image.

[0077] RGB_view image 73 contains color distribution information that depends on the viewpoint angle. RGB_view image 73 comprises multiple images. These multiple images contain color distribution information with strong color components when viewed from different viewpoints. For example, RGB_view image 73 includes four images: image k0 with strong color components when viewed from above, image k0 with strong color components when viewed from below, image k0 with strong color components when viewed from the left, and image k0 with strong color components when viewed from the right.

[0078] For each main layer (i.e., for each of multiple consecutive layers in a layer group), generate an RGB_base image 72 and an RGB_view image 73. For example, in Figure 3 In this case, RGB_base image 72 and RGB_view image 73 are generated one by one for the main layer 51-1 and sub-layers 52-1 to 52-11. That is, RGB_base image 72 indicates the color distribution information for 12 consecutive layers and is generated one by one for each of the 12 layers. Similarly, RGB_view image 73 indicates the color distribution information for 12 consecutive layers and four images are generated for each of the 12 layers.

[0079] The base image 74 comprises the same number of images as the RGB_view images 73. For example, in the case of forming four RGB_view images 73 as described above, four base images 74 are also formed. Each base image 74 corresponds to a different RGB_view image 73, and the base image 74 indicates the blending ratio of the corresponding RGB_view image 73 for each viewpoint direction. That is, the pixel position of the base image 74 is determined according to the viewpoint direction, and it is blended with the RGB_view image 73 corresponding to the base image 74 according to the blending ratio indicated by the pixel value. For example, in the case of a frontal viewpoint, as... Figure 5 As shown, each RGB_view image in RGB_view images 73 (k1 to k4) is multiplied by the blending ratio indicated by the center pixel value of the corresponding base image 74 (H1 to H4), and the result of the multiplication is added to the RGB_base image 72 (k0). Additionally, in the case of an upper left viewpoint, as... Figure 6 As shown, each RGB_view image in RGB_view images 73 (k1 to k4) is multiplied by a blending ratio indicated by the top left pixel value of the corresponding base image 74 (H1 to H4), and the result of the multiplication is added to RGB_base image 72 (k0).

[0080] In NeX MPI, as described above, expressive power can be enhanced by modifying the RGB components based on the viewpoint orientation. Note that by setting an alpha image for each sub-layer and an RGB image for each master layer, expressive power can be improved while suppressing the increase in data volume and enhancing learning efficiency.

[0081] <Extensions to MPI and Constraints on Layer Characteristics>

[0082] In addition, non-patent document 5 discloses a method for extending the specification of MPI by using SEI (Supplemental Enhancement Information) to the bitstream obtained by encoding MPI.

[0083] However, even with the extension using the method described in Non-Patent Document 5, constraints remain regarding the characteristics of the MPI layers. For example, there are constraints on the layer arrangement, number, resolution, shape, etc. Due to these constraints, there is a problem of reduced image quality obtained by rendering 3D data. For example, there is a problem of reduced image quality and resolution in the rendered image.

[0084] <3. About Layer Expansion>

[0085] Method 1

[0086] For example, in 3D data including layer images of color components and alpha components as described above, the relative relationships between layers are fixed. That is, the decoder needs to determine the relative relationships between layers, and there are constraints that make it difficult to optimize these relationships based on factors such as content. Therefore, there is a problem of reduced quality in the rendered image.

[0087] Therefore, in order to suppress the degradation of rendered image quality caused by such constraints, such as Figure 7 As shown in the top part of the table (Method 1), relative information about the color layer and the alpha layer is transmitted. In this specification, the layer containing the color components is also referred to as the color layer. Furthermore, the layer containing the alpha components is also referred to as the alpha layer.

[0088] For example, a first image processing apparatus is configured to include: an interlayer relative information generation unit that generates interlayer relative information about the relative relationships between layers of 3D data, the 3D data including layer images of color components and layer images of alpha components; and an encoding unit that generates a bitstream by encoding the 3D data and adds the interlayer relative information to the bitstream. For example, a first image processing method is configured to include: generating interlayer relative information about the relative relationships between layers of 3D data by a first image processing apparatus, the 3D data including layer images of color components and layer images of alpha components; and generating a bitstream by encoding the 3D data and adding the interlayer relative information to the bitstream. For example, a first program is configured to cause the first image processing apparatus to perform the following operations: generating interlayer relative information about the relative relationships between layers of 3D data, the 3D data including layer images of color components and layer images of alpha components; and generating a bitstream by encoding the 3D data and adding the interlayer relative information to the bitstream.

[0089] For example, a second image processing apparatus is configured to include: a decoding unit that decodes a bitstream and generates 3D data comprising layer images including color components and alpha components; an interlayer relative information analysis unit that analyzes interlayer relative information extracted from the bitstream regarding the relative relationships between layers of the 3D data; and a viewpoint image generation unit that renders the 3D data based on the interlayer relative information and generates a viewpoint image viewed from any viewpoint. For example, a second image processing method is configured to include: decoding a bitstream by a second image processing apparatus and generating 3D data comprising layer images including color components and alpha components; analyzing interlayer relative information extracted from the bitstream regarding the relative relationships between layers of the 3D data; and rendering the 3D data based on the interlayer relative information and generating a viewpoint image viewed from any viewpoint. For example, the second program is configured to cause the second image processing device to perform the following operations: decode the bitstream and generate 3D data including layer images of color components and layer images of alpha components; analyze interlayer relative information about the relative relationships between layers of the 3D data extracted from the bitstream; and render the 3D data based on the interlayer relative information and generate a viewpoint image viewed from any viewpoint.

[0090] This interlayer relative information can be added to the bitstream using any method. For example, the interlayer relative information can be stored in the SEI of the bitstream added to the 3D data. Of course, other methods can be applied.

[0091] By transmitting interlayer relative information from the encoding side to the decoding side as described above, the decoder can easily identify the relative relationships based on the interlayer relative information, even if the relative relationships between layers are optimized according to content, etc. Therefore, the degradation of rendered image quality can be suppressed.

[0092] <Method 1-1>

[0093] For example, in the method described in Non-Patent Document 5, by having as Figure 8 The SEI in the syntax shown is added to the bitstream to extend the MPI specification. As shown in this syntax, information used to set the relationship between the number of color layers and the number of alpha layers is not transmitted in the SEI. That is, the relationship between the number of color layers and the number of alpha layers is fixed (e.g., one-to-one).

[0094] In the method described in Non-Patent Document 5, the layers constituting the MPI are packaged and encoded using a predetermined method. At this time, as... Figure 9 As shown in A, a color layer group 111 in which layer groups constituting the color components of MPI are arranged, and an alpha layer group 112 in which layer groups constituting the alpha components are arranged are formed. In the case of the method described in non-Patent Document 5, since the ratio of the number of color layers to the number of alpha layers is 1:1, the size and shape of the color layer group 111 are the same as the size and shape of the alpha layer group 112.

[0095] The color layer group 111 and the alpha layer group 112 are arranged and packaged. For example, as... Figure 9 As shown in B, color layer group 111 and alpha layer group 112 can be arranged vertically and packaged (stacked). Figure 10 As shown in Figure A, color layer group 111 and alpha layer group 112 can be arranged horizontally and packaged (stacked). Figure 10 As shown in B, color layer group 111 and alpha layer group 112 can be arranged and packed (stacked) along the time (T) direction. That is, color layer group 111 and alpha layer group 112 can be stacked as consecutive frames. Color layer group 111 and alpha layer group 112 are stacked according to the method specified in the pre-prepared predetermined options.

[0096] However, since the ratio of the number of color layers to the number of alpha layers, as described above, is fixed, there is a constraint that makes it difficult to optimize the layer ratio based on factors such as content. Therefore, this results in a reduction in the quality of the rendered image.

[0097] For example, if the number of alpha layers is insufficient, the sampling density of the alpha components (shape information) becomes inadequate depending on the viewpoint position. This leads to a subjective quality degradation in the rendered image known as the stacked card effect—where the outer contours of objects become coarse. Increasing the number of layers can suppress the stacked card effect, but it increases processing time and data volume for encoding, decoding, rendering, etc. Since the ratio of color layers to alpha layers is fixed, the number of color layers increases as the number of alpha layers increases. Therefore, even with a sufficient number of color layers, there is a problem of unnecessarily increasing processing time and data volume. Furthermore, the processing time and data volume for encoding, decoding, rendering, etc., are typically finite in a system, making it difficult to increase the number of layers indefinitely. Therefore, when processing time and data volume reach the system's upper limit, it becomes difficult to suppress the subjective quality degradation of the rendered image due to insufficient alpha layers.

[0098] Therefore, in the case of applying method 1, such as from Figure 7 As shown in the second row from the top of the table, information regarding the relative number of alpha layers relative to color layers can be transmitted (Method 1-1). For example, in the first image processing apparatus, the first image processing method, and the first procedure, the inter-layer relative information added to the bitstream may include information regarding the relative number of alpha component layers relative to color component layers. Similarly, in the second image processing apparatus, the second image processing method, and the second procedure, the inter-layer relative information added to the bitstream may include information regarding the relative number of alpha component layers relative to color component layers.

[0099] Information about the relative number of alpha layers relative to color layers can be added to the bitstream using any method. For example, this information can be stored in the SEI of the bitstream added to the 3D data. Of course, other methods can be applied.

[0100] For example, as by Figure 11 The parameter "mpii_opacity_layer_num_minus1" indicated by box 113 in section A can be stored in the SEI of the bitstream added to the 3D data. The value of the parameter "mpii_opacity_layer_num_minus1" indicates the relative number of alpha layers relative to the color layers. In other words, the value of the parameter "mpii_opacity_layer_num_minus1" indicates the layer ratio between the color layers and the alpha layers. That is, in the 3D data, the color layers and alpha layers will be configured according to the ratio indicated by the parameter "mpii_opacity_layer_num_minus1".

[0101] For example, when the parameter "mpii_opacity_layer_num_minus1" is set to "5", the ratio of color layers to alpha layers in 3D data is 1:4. That is, during packaging, for each color layer group, four times the number of alpha layers are stacked. For example, as... Figure 11 B to Figure 11 As shown in D, it has a color layer group 111 ( Figure 9 The four alpha layer groups 112 (A) have the same size and shape as the other two alpha layers. Figure 9 A) is stacked relative to color layer group 111. Figure 11 In this context, the four alpha groups 112 are also referred to as alpha group 112-1, alpha group 112-2, alpha group 112-3, and alpha group 112-4.

[0102] In this case, for example, such as Figure 11 As shown in B, color layer group 111, alpha layer group 112-1, alpha layer group 112-2, alpha layer group 112-3, and alpha layer group 112-4 can be arranged vertically and packaged (stacked). Furthermore, as... Figure 11 As shown in C, color layer group 111, alpha layer group 112-1, alpha layer group 112-2, alpha layer group 112-3, and alpha layer group 112-4 can be arranged horizontally and packaged (stacked). Furthermore, as... Figure 11 As shown in D, color layer group 111, alpha layer group 112-1, alpha layer group 112-2, alpha layer group 112-3, and alpha layer group 112-4 can be arranged and packed (stacked) along the time (T) direction. That is, color layer group 111, alpha layer group 112-1, alpha layer group 112-2, alpha layer group 112-3, and alpha layer group 112-4 can be stacked as consecutive frames. Color layer group 111, alpha layer group 112-1, alpha layer group 112-2, alpha layer group 112-3, and alpha layer group 112-4 can be stacked according to the method specified from these pre-prepared predetermined options.

[0103] As described above, by transmitting information about the relative number of alpha layers to color layers from the encoding side to the decoding side, the decoder can easily identify the layer ratio between the color and alpha layers based on this information. In other words, the layer ratio between the color and alpha layers can be made variable. For example, the layer ratio between the color and alpha layers can be optimized based on content, etc.

[0104] As described above, by making the ratio of color layers to alpha layers variable, the number of alpha layers can be increased without increasing the number of color layers. For example, when the number of color layers is sufficient, alpha layer shortages and the stacking card effect can be suppressed without unnecessarily increasing processing time and data volume. Furthermore, even when processing time or data volume reaches the system's limits, alpha layer shortages and the stacking card effect can still be suppressed.

[0105] That is, by transmitting information about the relative number of alpha layers to color layers from the encoding side to the decoding side, the subjective quality reduction of the rendered image due to insufficient number of alpha layers can be suppressed.

[0106] <Method 1-2>

[0107] For example, in the method described in Non-Patent Document 5, such as Figure 8 As shown in the syntax, SEI does not transmit information for setting the relationship between the resolution of the color layer and the resolution of the alpha layer. That is, the relationship between the resolution of the color layer and the resolution of the alpha layer (also considered as the relationship between the number of pixels or the layer size) is fixed.

[0108] In the method described in non-patent literature 5, such as Figure 9 and Figure 10 As shown, the resolution (number of pixels, e.g., size) of the color layer and the alpha layer is the same. That is, the resolution ratio between the color layer and the alpha layer is fixed (1:1).

[0109] However, since the resolution ratio between the color layer and the alpha layer is fixed, as described above, there is a constraint that makes it difficult to optimize the resolution ratio based on factors such as content. Therefore, there is a problem of reduced image quality.

[0110] Generally, higher resolution is more important in the color layer than in the alpha layer. Insufficient resolution in the color layer leads to texture blurring and reduced reproducibility during rendering (i.e., a decrease in the subjective quality of the rendered image). Increasing the resolution can suppress this texture blurring, but this increases processing time and data volume for encoding, decoding, and rendering. Because the resolution ratio between the color layer and the alpha layer is fixed, increasing the resolution of the color layer also increases the resolution of the alpha layer. Therefore, even when the alpha layer resolution is sufficient, there is a problem of unnecessarily increasing processing time and data volume. Furthermore, the processing time and data volume for encoding, decoding, and rendering are usually finite in a system, and it is difficult to increase the resolution indefinitely. Therefore, when the processing time and data volume reach the system's limits, it becomes difficult to suppress the decrease in subjective quality of the rendered image due to insufficient color layer resolution.

[0111] Therefore, in the case of applying method 1, such as from Figure 7 As shown in the third row from the top of the table, information regarding the relative resolution of the alpha layer relative to the color layer can be transmitted (methods 1-2). For example, in the first image processing apparatus, the first image processing method, and the first procedure, the inter-layer relative information added to the bitstream may include information regarding the relative resolution of the alpha component layer relative to the color component layer. Similarly, in the second image processing apparatus, the second image processing method, and the second procedure, the inter-layer relative information added to the bitstream may include information regarding the relative resolution of the alpha component layer relative to the color component layer.

[0112] Information about the relative resolution of the alpha layer relative to the color layer can be added to the bitstream using any method. For example, this information can be stored in the SEI of the bitstream added to the 3D data. Of course, other methods can be applied.

[0113] For example, such as Figure 12 The parameter "mpii_opacity_layer_div_scale_log2", indicated by box 114 in A, can be stored in the SEI of the bitstream added to the 3D data. The value of the parameter "mpii_opacity_layer_div_scale_log2" indicates the relative resolution of the alpha layer relative to the color layer. In other words, the value of the parameter "mpii_opacity_layer_div_scale_log2" indicates the resolution ratio between the color layer and the alpha layer. That is, the size ratio between the color layer and the alpha layer is indicated by the parameter "mpii_opacity_layer_div_scale_log2".

[0114] For example, when the resolution ratio between the color layer and the alpha layer is 4:1, the size of the alpha layer group is also 1 / 4 of the size of the color layer group, such as... Figure 11 B or Figure 11 As shown in C. At this point, layers can be grouped so that they can be stacked in a rectangular pattern. For example, in... Figure 11 When the color layer group and alpha layer group shown in B are stacked vertically, an alpha layer group 115 can be formed, such that the length of the alpha layer group 115 in the horizontal direction is the same as the length of the color layer group 111 in the horizontal direction. Furthermore, in cases such as... Figure 11 When the color layer group and alpha layer group shown in C are stacked in the horizontal direction, an alpha layer group 116 can be formed, such that the length of the alpha layer group 116 in the vertical direction is the same as the length of the color layer group 111 in the vertical direction.

[0115] The region targeted for encoding needs to be rectangular. Therefore, by forming layer groups as described above, and stacking them rectangularly according to the value of the parameter "mpii_opacity_layer_div_scale_log2" (i.e., the resolution ratio), the layer groups can be packed without waste. That is, the rectangular region targeted for encoding can be filled by the layer group without leaving any free space. Therefore, unnecessary increases in data volume can be suppressed. In other words, as long as free space is allowed in the rectangular region targeted for encoding after packing, there is no need to stack the color layer group and the alpha layer group rectangularly.

[0116] By transmitting information about the relative resolution of the alpha layer relative to the color layer from the encoding side to the decoding side, as described above, the decoder can easily identify the resolution ratio between the color layer and the alpha layer based on this information. In other words, the resolution ratio between the color layer and the alpha layer can be made variable. For example, the resolution ratio between the color layer and the alpha layer can be optimized based on content, etc.

[0117] As described above, by making the resolution ratio between the color layer and the alpha layer variable, the resolution of the color layer can be increased without increasing the resolution of the alpha layer. For example, when the resolution of the alpha layer is sufficient, insufficient resolution of the color layer can be suppressed and texture blurring can be prevented without unnecessarily increasing processing time and data volume. Furthermore, even when processing time and data volume reach the system's limits, insufficient resolution of the color layer and texture blurring can still be suppressed.

[0118] That is, by transmitting information about the relative resolution of the alpha layer relative to the color layer from the encoding side to the decoding side, the subjective quality reduction of the rendered image due to insufficient resolution of the color layer can be suppressed.

[0119] <Methods 1-3>

[0120] Furthermore, both the layer ratio and resolution ratio between the color layer and the alpha layer can be varied. That is, in the case of applying method 1, such as... Figure 7 As shown in the bottom row of the table, information regarding the relative number of alpha layers relative to color layers and information regarding the relative resolution of alpha layers relative to color layers can be transmitted (methods 1-3). For example, in the first image processing apparatus, the first image processing method, and the first procedure, the inter-layer relative information added to the bitstream may include information regarding the relative number of alpha component layers relative to color component layers and information regarding the relative resolution of alpha component layers relative to color component layers. Similarly, in the second image processing apparatus, the second image processing method, and the second procedure, the inter-layer relative information added to the bitstream may include information regarding the relative number of alpha component layers relative to color component layers and information regarding the relative resolution of alpha component layers relative to color component layers.

[0121] Similarly, in this case, information about the relative number of alpha layers relative to color layers, and information about the relative resolution of alpha layers relative to color layers, can be added to the bitstream by any method. For example, this type of information can be stored in the SEI of the bitstream added to the 3D data. Of course, other methods can be applied.

[0122] For example, such as Figure 13 As shown in box 117 of A, the parameters "mpii_opacity_layer_num_minus1" and "mpii_opacity_layer_div_scale_log2" can be stored in the SEI of the bitstream added to the 3D data. These parameters are as described above in <Method 1-1> and <Method 1-2>.

[0123] In this way, the effects obtained by applying method 1-1 and the effects obtained by applying method 1-2 described above can be obtained simultaneously.

[0124] Furthermore, in this case, since both the layer ratio and resolution ratio between the color layer and the alpha layer are variable, therefore, for example, as Figure 13 As shown in Figure B, a layer group comprising alpha layers with sizes different from the color layers, such as in alpha layer group 118, can be formed to have the same size as color layer group 111. In this way, for example, not only can it be as in... Figure 13 As shown in example B, layers are stacked vertically or as in Figure 14 Layers can be stacked horizontally as shown in example A, and can also be stacked horizontally as shown in... Figure 14 Layers are stacked along the time direction as shown in the example shown in B. That is, color layer group 111 and alpha layer group 118 can be stacked as consecutive frames. As described above, packing can be performed more efficiently due to the increased diversity of stacking.

[0125] Method 2

[0126] In the method described in Non-Patent Document 5, the following constraint exists as described above: for 3D data including layer images of color components and layer images of alpha components, the layer arrangement pattern is fixed. For example, when the positions of the nearest and farthest layers are specified, the remaining layers are arranged at equal intervals between them.

[0127] However, depth and parallax are generally inversely related. Therefore, to ensure that the quality of an object remains constant regardless of its position in the depth direction, it is necessary to have a higher layer density at closer depths. Consequently, in the case of layers arranged at equal intervals as described above, for example in scenes including distant views such as outdoors, there is a problem of reduced quality of objects in the foreground.

[0128] Typically, objects are not uniformly distributed along the depth direction. Furthermore, the quality of an object is not necessarily uniform along the depth direction. Therefore, arranging layers at equal intervals along the depth direction will degrade the quality of the rendered image.

[0129] For example, when aiming to represent a background such as the sky at infinity, the furthest layer needs to be set to infinity. In this case, even if other objects exist only at a short distance from the viewpoint, the following problems arise: the layers are arranged at equal intervals over a wide area extending to infinity, resulting in insufficient layers for each object's location, making it difficult to represent the object's shape and texture with sufficient precision. Furthermore, even for a single object, the quality at near depth, clearly visible from the viewpoint, has a greater impact on image quality than the quality at far depth. However, arranging layers at equal intervals along the depth direction makes it difficult to represent the object's shape and texture at near depth with sufficient precision. There is also a problem of degraded rendered image quality due to reduced 3D data quality (reduced reproducibility).

[0130] Increasing the number of layers can suppress the degradation of 3D data quality, but it leads to an increase in processing time and data volume for encoding, decoding, and rendering. In other words, simply increasing the total number of layers increases layer density even at unnecessary depths. This presents the problem of needing to add unnecessary layers and unnecessarily increasing processing time and data volume. Furthermore, the processing time and data volume for encoding, decoding, and rendering are typically finite in a system, and it's difficult to increase the number of layers indefinitely. Therefore, when processing time and data volume reach the system's limits, it's difficult to suppress the subjective degradation of rendered image quality by arranging layers at equal intervals.

[0131] Note that in the method described in Non-Patent Document 5, the position of each layer can be specified. As a result, the layer arrangement intervals can be non-equidistant. However, in this case, there is a problem that the position of all layers needs to be specified, and the amount of data unnecessarily increases.

[0132] Therefore, in order to suppress the degradation of rendered image quality caused by such constraints, such as Figure 15 As shown in the top part of the table, information about the layer arrangement is transmitted (Method 2).

[0133] For example, a first image processing apparatus includes: a layer arrangement information generation unit that generates layer arrangement information about the arrangement of layers in 3D data, the 3D data including layer images of color components and layer images of alpha components; and an encoding unit that generates a bitstream by encoding the 3D data and adds the layer arrangement information to the bitstream. For example, a first image processing method includes: generating layer arrangement information about the arrangement of layers in 3D data by a first image processing apparatus, the 3D data including layer images of color components and layer images of alpha components; and generating a bitstream by encoding the 3D data by the first image processing apparatus and adding the layer arrangement information to the bitstream. For example, a first program causes the first image processing apparatus to perform the following operations: generating layer arrangement information about the arrangement of layers in 3D data including layer images of color components and layer images of alpha components; and generating a bitstream by encoding the 3D data and adding the layer arrangement information to the bitstream.

[0134] For example, a second image processing apparatus includes: a decoding unit that decodes a bitstream and generates 3D data including layer images of color components and alpha components; a layer layout information analysis unit that analyzes layer layout information extracted from the bitstream regarding the layer arrangement of the 3D data; and a viewpoint image generation unit that renders the 3D data based on the layer layout information and generates a viewpoint image viewed from any viewpoint. For example, a second image processing method includes: decoding a bitstream by a second image processing apparatus and generating 3D data including layer images of color components and alpha components; analyzing layer layout information extracted from the bitstream regarding the layer arrangement of the 3D data; and rendering the 3D data based on the layer layout information and generating a viewpoint image viewed from any viewpoint. For example, the second program causes the second image processing apparatus to perform the following operations: decode the bitstream and generate 3D data including layer images of color components and alpha components; analyze layer arrangement information about the arrangement of layers in the 3D data extracted from the bitstream; and render the 3D data based on the layer arrangement information and generate a viewpoint image viewed from any viewpoint.

[0135] The layer layout information can be added to the bitstream using any method. For example, the layer layout information can be stored in the SEI of the bitstream added to the 3D data. Of course, other methods can also be applied.

[0136] By transmitting layer layout information from the encoding side to the decoding side as described above, the decoder can easily arrange the layers at any position based on the layer layout information. Therefore, for example, layers can be arranged with non-equidistant intervals while suppressing unnecessary increases in data volume. As a result, the layer layout can be optimized, for example, based on content, etc. Thus, it is possible to suppress unnecessary increases in data volume while suppressing a decrease in the quality of the rendered image.

[0137] <Method 2-1>

[0138] For example, information about the arrangement of some layers in 3D data can be transmitted as layer arrangement information. Some layers can be, for example, additional layers added to a basic layer group to arrange layers in a predetermined arrangement pattern. That is, in the case of applying method 2, such as from... Figure 15 As shown in the second row from the top of the table, information about the arrangement of additional layers can be transmitted (Method 2-1).

[0139] For example, in the first image processing apparatus, the first image processing method, and the first program, the layer arrangement information added to the bitstream may include information about the arrangement of some layers of the 3D data. Similarly, in the second image processing apparatus, the second image processing method, and the second program, the layer arrangement information added to the bitstream may include information about the arrangement of some layers of the 3D data.

[0140] For example, such as Figure 16 As shown in A, assume that the 3D data layers are arranged at a distance Z from the viewpoint. Near To Z Far The basic layer group includes layers 211 to 215, as well as additional layers 221 and 222. Additional layers 221 and 222 are layers outside the basic layer group (i.e., layers that do not belong to the basic layer group). For example, when the layers of the basic layer group (layers 211 to 215) are arranged according to a predetermined arrangement pattern, additional layers 221 and 222 are arranged at locations that do not follow the predetermined arrangement pattern.

[0141] By providing additional layers in addition to the basic layer group as described above, the layer density at each location in the depth direction can be easily and freely controlled. For example, by using additional layers, the layer density at locations where objects exist in the depth direction can be easily increased. Furthermore, it is also easy to arrange the reference layer group within the range where objects exist in the depth direction, and to arrange the additional layers at infinity. In this way, the number of layers at desired locations can be increased without increasing the number of layers at unnecessary locations, and therefore, the degradation of the rendered image quality can be suppressed while suppressing the unnecessary increase in data volume.

[0142] Note that in Figure 16 The example in A shows a case with two additional layers (additional layer 221 and additional layer 222), but the number of additional layers can be any. Furthermore, the arrangement pattern of the base layers (predetermined arrangement pattern) can be predetermined according to standard specifications, or it can be specified by the user, etc. Moreover, the predetermined arrangement pattern can be any pattern. For example, layers can be arranged at equal intervals, or they can be arranged at non-equal intervals. For example, a function-based arrangement pattern can be used. For example, an arrangement pattern in which the distance between layers increases with increasing distance from the viewpoint can be used (e.g., an inversely proportional pattern or a logarithmic pattern).

[0143] As described above, the information regarding the arrangement of additional layers can be added to the bitstream as layer arrangement information.

[0144] For example, in the first image processing apparatus, the first image processing method, and the first program, information regarding the arrangement of layers added to the bitstream may include information indicating the positions of these layers in the depth direction. Similarly, in the second image processing apparatus, the second image processing method, and the second program, information regarding the arrangement of layers added to the bitstream may include information indicating the positions of these layers in the depth direction.

[0145] For example, in the first image processing apparatus, the first image processing method, and the first program, information regarding the arrangement of layers added to the bitstream may include information indicating the number of these layers. Similarly, in the second image processing apparatus, the second image processing method, and the second program, information regarding the arrangement of layers added to the bitstream may include information indicating the number of these layers.

[0146] For example, it can be adopted as follows Figure 16 The syntax indicated by block 231 in B stores the parameters in the SEI of the bitstream added to the 3D data. The value of the parameter "mpii_num_layers_explicit_depth" indicates the number of layers (e.g., additional layers). The parameter "depth_rep_info_element" indicates the position of some layers (e.g., additional layers) in the depth direction. That is, in the layer layout information of this example, the position is specified for each additional layer.

[0147] As described above, by transmitting information about the arrangement of some layers (e.g., additional layers) from the encoding side to the decoding side, the decoder can easily place these layers (additional layers) at locations other than those conforming to a predetermined arrangement pattern of the basic layer group. Therefore, the density of layers at each location in the depth direction can be freely controlled, and the degradation of the rendered image quality can be suppressed while preventing unnecessary increases in data volume.

[0148] <Method 2-2>

[0149] For example, the layers that make up a layer group of 3D data can be arranged with non-equidistant intervals. In other words, the arrangement pattern of the layer group that makes up the 3D data can be a non-equidistant arrangement pattern in which layers are arranged with non-equidistant intervals. In other words, a non-equidistant arrangement pattern can be specified as the arrangement pattern of the layer group that makes up the 3D data, and the information used to specify the non-equidistant arrangement pattern can be transmitted as layer arrangement information.

[0150] This non-uniform spacing arrangement pattern can be specified by any method. For example, it can be specified by using a predefined function. In the following text, the function used to specify the non-uniform spacing arrangement pattern is also referred to as the non-uniform spacing arrangement function. That is, the non-uniform spacing arrangement function is a function that arranges layers with non-uniform spacing. Specifically, in the case of applying method 2, such as from... Figure 15 As shown in the third row from the top of the table, information about the arrangement of layers based on the non-equal spacing arrangement function can be transmitted (as layer arrangement information) (Method 2-2).

[0151] For example, in the first image processing apparatus, the first image processing method, and the first program, the layer arrangement information added to the bitstream may include information about the arrangement of layers of 3D data based on a non-uniform spacing function. Similarly, in the second image processing apparatus, the second image processing method, and the second program, the layer arrangement information added to the bitstream may include information about the arrangement of layers of 3D data based on a non-uniform spacing function.

[0152] The non-uniform spacing function can be any function. For example, the function can make layers denser at closer depths (i.e., the layer density increases). In other words, the function can make layers sparser at farther depths (i.e., the layer density decreases). For example, suppose the indices are arranged in ascending order from near depth to far depth. The function described above can be a function that arranges layers according to a pattern where the difference in indices between the farthest layer and the current layer is inversely proportional to the spacing between the current layer and its adjacent layers (hereinafter also referred to as an inverse-proportional pattern) (hereinafter also referred to as an inverse-proportional layout function). Alternatively, the function described above can be a function that arranges layers according to a pattern where the spacing between layers increases exponentially based on the index value (hereinafter also referred to as a logarithmic pattern) (hereinafter also referred to as a logarithmic layout function).

[0153] For example, the relationship between layer index and layer depth (position in the depth direction) is... Figure 17 The curve is shown in the graph. Figure 17 In the graph, line 241 indicates the relationship between the index (i) and the depth (Z) of each layer when the equal-interval arrangement function is applied, which specifies the equal-interval arrangement pattern of arranging layers at equal intervals. Note that the index (i) of each layer is normalized between "0" and "1". Curve 242 indicates the relationship between the index (i) and the depth (Z) of each layer when the inverse-proportional arrangement function is applied. Curve 243 indicates the relationship between the index (i) and the depth (Z) of each layer when the logarithmic arrangement function is applied.

[0154] For example, the depth (Z(i)) of each layer (i) can be represented by the following expression (1).

[0155]

[0156] Under the condition of equally spaced arrangement function, it can satisfy and Under the condition of inverse proportional arrangement function, it can satisfy... and In the case of a logarithmic arrangement function, it can satisfy... , By specifying the layout pattern using such a function, the position of each layer can be specified without increasing the amount of data. Furthermore, each layer can be easily arranged according to a predetermined layout pattern. Additionally, applying an inverse or logarithmic layout function makes it easy to make layers denser at closer depths (i.e., the layer density becomes higher). In other words, it's easy to make layers sparser at farther depths (i.e., the layer density becomes lower). Therefore, the quality degradation of objects in the foreground due to the inverse relationship between depth and parallax can be suppressed.

[0157] <Method 2-2-1>

[0158] The layer layout pattern described above can be predetermined and can be configured by users, etc. For example, in the case of applying method 2-2, such as from Figure 15 As shown in the fourth row from the top of the table, information can be transmitted indicating whether the arrangement of the layers is based on a non-equidistant arrangement function (as layer arrangement information) (Method 2-2-1).

[0159] For example, in the first image processing apparatus, the first image processing method, and the first program, the "information about arrangement based on a non-equal spacing arrangement function" added to the bitstream may include information indicating whether the layers are arranged based on a non-equal spacing arrangement function. Similarly, in the second image processing apparatus, the second image processing method, and the second program, the "information about arrangement based on a non-equal spacing arrangement function" added to the bitstream may include information indicating whether the layers are arranged based on a non-equal spacing arrangement function.

[0160] "Information indicating whether the layers are arranged based on a non-equal spacing arrangement function" can be information used to specify the mode of the method for arranging layers.

[0161] For example, in the first image processing apparatus, the first image processing method, and the first program, the "information about the arrangement based on the non-equal spacing arrangement function" added to the bitstream may include information about the mode for specifying the method of arranging the layers. Similarly, in the second image processing apparatus, the second image processing method, and the second program, the "information about the arrangement based on the non-equal spacing arrangement function" added to the bitstream may include information about the mode for specifying the method of arranging the layers.

[0162] For example, it can transmit such as Figure 18 The parameter "mpii_layer_depth_type" is shown in box 251 in A. The parameter "mpii_layer_depth_type" is information used to specify the mode of the method for arranging the layers. For example, the parameter "mpii_layer_depth_type" can specify the mode to be applied through its value, such as... Figure 18 As shown in Table 252 of A. For example, when the value of the parameter "mpii_layer_depth_type" is "0", this indicates that a pattern for arranging layers based on an equal-interval arrangement function is specified. Furthermore, when the value of the parameter "mpii_layer_depth_type" is "1", this indicates that a pattern for specifying the positions of all layers is specified. Furthermore, when the value of the parameter "mpii_layer_depth_type" is "2", this indicates that a pattern for arranging layers based on an inverse arrangement function is specified. Furthermore, when the value of the parameter "mpii_layer_depth_type" is "3", this indicates that a pattern for arranging layers based on a logarithm arrangement function is specified. That is, the parameter "mpii_layer_depth_type" can also be considered as information indicating whether layers are arranged based on a non-equal-interval arrangement function.

[0163] By transmitting the parameter "mpii_layer_depth_type" as "information about the arrangement based on the non-uniform spacing function" from the encoding side to the decoding side, the decoder can arrange the layers in the mode specified by the parameter "mpii_layer_depth_type". Therefore, more arrangement modes can be implemented, and layer arrangement can be optimized in more situations. That is, in more cases, the degradation of rendered image quality can be suppressed while suppressing unnecessary increases in data volume.

[0164] <Method 2-2-2>

[0165] Furthermore, in the case of applying method 2-2, such as from Figure 15 As shown in the fifth row from the top of the table, information can be transmitted regarding whether the indication interval is not explicitly indicated, whether the indication arrangement is a non-linear mode arrangement, and whether the indication arrangement is an inverse mode arrangement (as layer arrangement information) (Method 2-2-2).

[0166] For example, in the first image processing apparatus, the first image processing method, and the first program, the "information about the arrangement based on the non-equal spacing function" added to the bitstream may include information indicating whether the layers are arranged in an inverse proportional mode. Similarly, in the second image processing apparatus, the second image processing method, and the second program, the "information about the arrangement based on the non-equal spacing function" added to the bitstream may include information indicating whether the layers are arranged in an inverse proportional mode.

[0167] Furthermore, in the first image processing apparatus, the first image processing method, and the first program, the "information regarding the arrangement based on the non-equal spacing function" added to the bitstream may include information indicating whether the layers are arranged in a non-linear mode. Similarly, in the second image processing apparatus, the second image processing method, and the second program, the "information regarding the arrangement based on the non-equal spacing function" added to the bitstream may include information indicating whether the layers are arranged in a non-linear mode.

[0168] Furthermore, in the first image processing apparatus, the first image processing method, and the first program, the "information about the arrangement based on the non-equidistant arrangement function" added to the bitstream may include information indicating whether the interval between layers is not explicitly indicated. Similarly, in the second image processing apparatus, the second image processing method, and the second program, the "information about the arrangement based on the non-equidistant arrangement function" added to the bitstream may include information indicating whether the interval between layers is not explicitly indicated.

[0169] For example, the parameters "mpii_layer_depth_equal_distance_flag", "mpii_layer_depth_nonlinear_flag", and "mpii_layer_depth_inverse_flag" can be transmitted as "information about the arrangement based on the non-equal spacing function".

[0170] The parameter "mpii_layer_depth_equal_distance_flag" is a flag indicating whether the spacing between layers is not explicitly indicated. When the flag is true (e.g., "1"), this indicates that the spacing between layers is not explicitly indicated. When the flag is false (e.g., "0"), this indicates that the spacing between layers is explicitly indicated. For example, a false flag could indicate that a pattern used to specify the positions of all layers is specified.

[0171] The parameter "mpii_layer_depth_nonlinear_flag" is a flag indicating whether layers are arranged in a nonlinear mode. A nonlinear mode is a layout mode that arranges layers at nonlinear intervals. When the flag is true (e.g., "1"), this indicates that layers are arranged in a nonlinear mode. When the flag is false (e.g., "0"), this indicates that layers are not arranged in a nonlinear mode. For example, when the flag is false, this could indicate that a mode based on an equal-interval layout function is specified.

[0172] The parameter "mpii_layer_depth_inverse_flag" is a flag indicating whether layers are arranged in an inverse mode. When the flag is true (e.g., "1"), this indicates that layers are arranged in an inverse mode. When the flag is false (e.g., "0"), this indicates that layers are not arranged in an inverse mode. That is, when the flag is true, this indicates that a mode based on the inverse arrangement function (Inverse) is specified. Furthermore, when the flag is false, this indicates that a mode based on the logarithm arrangement function (Logarithm) is specified.

[0173] Such as Figure 18 The syntax shown in block 253 of section B stores these parameters in the SEI of the bitstream added to the 3D data. Specifically, when the parameter "mpii_layer_depth_equal_distance_flag" is true, the parameter "mpii_layer_depth_nonlinear_flag" can be set, and when the value of the parameter "mpii_layer_depth_nonlinear_flag" is true, the parameter "mpii_layer_depth_inverse_flag" can be set. When the parameter "mpii_layer_depth_equal_distance_flag" is false, a mode for specifying the positions of all layers can be specified. When the parameter "mpii_layer_depth_nonlinear_flag" is false, a mode for arranging layers based on an equal-interval arrangement function can be specified. When the parameter "mpii_layer_depth_inverse_flag" is true, a mode for arranging layers based on an inverse arrangement function can be specified. When the parameter "mpii_layer_depth_inverse_flag" is false, you can specify the mode for arranging layers based on the logarithmic arrangement function (Logarithm).

[0174] By transmitting each of these parameters as "information about the arrangement based on the non-equidistant arrangement function" from the encoding side to the decoding side, the decoder can arrange the layers in the pattern indicated by these parameters. Therefore, more arrangement patterns can be implemented, and layer arrangement can be optimized in more situations. That is, in more cases, the degradation of rendered image quality can be suppressed while suppressing unnecessary increases in data volume.

[0175] <Method 2-2-3>

[0176] Note that the parameter "mpii_layer_depth_explicit_distance_flag" can be used instead of "mpii_layer_depth_equal_distance_flag". The parameter "mpii_layer_depth_explicit_distance_flag" is a flag indicating whether the spacing between layers is explicitly indicated. When the flag is true (e.g., "1"), this indicates that the spacing between layers is explicitly indicated. When the flag is false (e.g., "0"), this indicates that the spacing between layers is not explicitly indicated. For example, when the flag is true, this can indicate that the pattern used to specify the positions of all layers is specified. That is, in the case of applying method 2-2, as... Figure 15 As shown in the bottom row of the table, information can be transmitted regarding whether the indicator interval is explicitly indicated, whether the indicator arrangement is a non-linear mode arrangement, and whether the indicator arrangement is an inverse mode arrangement (as layer arrangement information) (Method 2-2-3).

[0177] For example, in the first image processing apparatus, the first image processing method, and the first program, the "information about the arrangement based on the non-equidistant arrangement function" added to the bitstream may include information indicating whether the interval between layers is explicitly indicated. Similarly, in the second image processing apparatus, the second image processing method, and the second program, the "information about the arrangement based on the non-equidistant arrangement function" added to the bitstream may include information indicating whether the interval between layers is explicitly indicated.

[0178] Such as Figure 18 The syntax shown in block 254 of C stores these parameters in the SEI of the bitstream added to the 3D data. Specifically, when the parameter "mpii_layer_depth_explicit_distance_flag" is false, the parameter "mpii_layer_depth_nonlinear_flag" can be set, and when the value of the parameter "mpii_layer_depth_nonlinear_flag" is true, the parameter "mpii_layer_depth_inverse_flag" can be set. When the parameter "mpii_layer_depth_explicit_distance_flag" is true, a mode for specifying the positions of all layers can be specified. When the parameter "mpii_layer_depth_nonlinear_flag" is false, a mode for arranging layers based on an equal-interval arrangement function can be specified. When the parameter "mpii_layer_depth_inverse_flag" is true, a mode for arranging layers based on an inverse arrangement function can be specified. When the parameter "mpii_layer_depth_inverse_flag" is false, you can specify the mode for arranging layers based on the logarithmic arrangement function (Logarithm).

[0179] By transmitting each of these parameters as "information about the arrangement based on the non-equidistant arrangement function" from the encoding side to the decoding side, the decoder can arrange the layers in the pattern indicated by these parameters. Therefore, more arrangement patterns can be implemented, and layer arrangement can be optimized in more situations. That is, in more cases, the degradation of rendered image quality can be suppressed while suppressing unnecessary increases in data volume.

[0180] Method 3

[0181] In the method described in Non-Patent Document 5, as described above, there is a constraint: for 3D data including layer images of color components and alpha components, the shape of the layers is fixed. In this method, MPI is applied as 3D data, and the shape of the layers is planar. Therefore, in scenes with a wide field of view, there is a problem of reduced resolution near the center of the distant scene and reduced quality of the rendered image including this portion.

[0182] Therefore, in order to suppress the degradation of rendered image quality caused by such constraints, such as Figure 19 As shown in the top part of the table, information regarding the selection of layer shape is transmitted (Method 3). That is, the shape of the layer can be optional.

[0183] For example, a first image processing apparatus includes: a layer shape selection information generation unit that generates layer shape selection information regarding the selection of the shape of layers in 3D data, the 3D data including layer images of color components and layer images of alpha components; and an encoding unit that generates a bitstream by encoding the 3D data and adds the layer shape selection information to the bitstream. For example, a first image processing method includes: generating layer shape selection information regarding the selection of the shape of layers in 3D data by a first image processing apparatus, the 3D data including layer images of color components and layer images of alpha components; and generating a bitstream by encoding the 3D data by the first image processing apparatus and adding the layer shape selection information to the bitstream. For example, a first program causes the first image processing apparatus to perform the following operations: generating layer shape selection information regarding the selection of the shape of layers in 3D data, the 3D data including layer images of color components and layer images of alpha components; and generating a bitstream by encoding the 3D data and adding the layer shape selection information to the bitstream.

[0184] For example, a second image processing apparatus includes: a decoding unit that decodes a bitstream and generates 3D data including layer images of color components and alpha components; a layer shape selection information analysis unit that analyzes layer shape selection information extracted from the bitstream regarding the selection of the shapes of the layers in the 3D data; and a viewpoint image generation unit that renders the 3D data based on the layer shape selection information and generates a viewpoint image viewed from any viewpoint. For example, a second image processing method includes: decoding a bitstream by a second image processing apparatus and generating 3D data including layer images of color components and alpha components; analyzing layer shape selection information extracted from the bitstream regarding the selection of the shapes of the layers in the 3D data; and rendering the 3D data based on the layer shape selection information and generating a viewpoint image viewed from any viewpoint. For example, the second program causes the second image processing device to perform the following operations: decode the bitstream and generate 3D data including layer images of color components and layer images of alpha components; analyze layer shape selection information extracted from the bitstream regarding the selection of the shape of the layers in the 3D data; and render the 3D data based on the layer shape selection information and generate a viewpoint image viewed from any viewpoint.

[0185] like Figure 20 As shown in A, MPI comprises a group of planar layers. When the resolution of each layer is kept constant, there is a problem of reduced resolution near the center of distant scenes in scenes with wide field of view. Figure 20 In example A, when viewing the vicinity of the scene's center from viewpoint 311, the square 312 in the far-depth layer is larger than the square in the near-depth layer. Square 312 schematically indicates the pixels of each layer. Therefore, when the area covered by each layer is constant, the size of the square 312 in each layer is the same. However, since the layers are planar, the square 312 becomes larger as the layer moves further away from viewpoint 311. That is, the area covered by a single pixel becomes wider. In other words, the number of pixels included in the viewpoint image of viewpoint 311 decreases as the layer moves further away from viewpoint 311. That is, the further the layer is from viewpoint 311, the lower the resolution. Therefore, there is a problem of reduced quality in the rendered image including this portion.

[0186] Therefore, as Figure 20 As shown in B, the layer can have a curved surface shape. Figure 20 In the example of B, the layers are curved with a predetermined curvature, and the size difference of square 312 between the layers near the depth and the layers far from the depth is less than [missing information]. Figure 20 The case in example A. That is, the difference in the area covered by a pixel between layers at near depth and layers at far depth is less than... Figure 20 The case in example A. This suppresses the reduction in resolution of layers farther from viewpoint 311 in the viewpoint image included at viewpoint 311. Therefore, it is possible to suppress the reduction in the quality of the rendered image including this part.

[0187] Note that examples of 3D data including layers with curved surface shapes include: multi-cylinder images (MCI) including layers with cylindrical shapes and multi-sphere images (MSI) including layers with spherical shapes. In MCI, layers are considered to be curved in one direction. In MSI, layers are considered to be curved in two directions.

[0188] As described above, compared to MPI, MCI or MSI can suppress the resolution degradation near the center of distant scenes. However, in MPI, layer setup is easier than in MCI or MSI, and the increase in processing load can be suppressed.

[0189] As described above, by transmitting information about the selection of layer shapes and enabling the selection of layer shapes—for example, choosing and applying the optimal method from MPI, MCI, and MSI for the content—the quality of the rendered image can be suppressed while preventing unnecessary increases in the processing load such as encoding, decoding, and reproduction.

[0190] <Method 3-1>

[0191] For example, in the case of applying method 3, such as from Figure 19 As shown in the second row from the top of the table, information on curvature specification in a single direction and information indicating whether the shape is cylindrical can be transmitted (Method 3-1).

[0192] For example, in the first image processing apparatus, the first image processing method, and the first program, the layer shape selection information added to the bitstream may include curvature specification information for specifying the curvature of the layer in a single direction and information indicating whether the layer has a cylindrical shape. Similarly, in the second image processing apparatus, the second image processing method, and the second program, the layer shape selection information added to the bitstream may include curvature specification information for specifying the curvature of the layer in a single direction and information indicating whether the layer has a cylindrical shape.

[0193] For example, such as Figure 21 As shown in box 321 of A, the parameters "mpii_layer_curvature" and "mpii_cylindrical_layer_flag" can be added to the bitstream as layer shape selection information. The parameter "mpii_layer_curvature" is curvature specification information used to specify the curvature of the layer in a single direction. The parameter "mpii_cylindrical_layer_flag" is a flag indicating whether the layer has a cylindrical shape. When the flag is true (e.g., "1"), this indicates that the layer has a cylindrical shape (i.e., this is MCI). When the flag is false (e.g., "0"), this indicates that the layer does not have a cylindrical shape (i.e., this is not MCI).

[0194] According to such Figure 21 The syntax shown in box 321 of section A stores these parameters in the SEI of the bitstream added to the 3D data. Specifically, the parameter "mpii_layer_curvature" can be set. When this parameter is "0", it indicates that the layer is planar (i.e., this is MPI). On the other hand, when the parameter is greater than "0", it indicates that the layer has a curved surface shape. In this case, the parameter "mpii_cylindrical_layer_flag" can be set. When the flag is true, it indicates that the layer has a cylindrical shape (i.e., this is MCI). In this case, the curvature indicated by the parameter "mpii_layer_curvature" can be applied to the bending direction of the layer. This bending direction can be any direction, as long as it is a single direction. For example, the direction can be horizontal, vertical, or other directions. Furthermore, the bending direction can be predetermined or specified by the user, etc. Additionally, when the parameter "mpii_cylindrical_layer_flag" is false, it indicates that the layer does not have a cylindrical shape (i.e., this is not MCI). For example, if the flag information is false, this could indicate that the layer has a spherical shape (i.e., it is MSI). In this case, the curvature indicated by the parameter "mpii_layer_curvature" can be applied to the bending direction of the layer. This bending direction can be any direction, as long as it is multiple directions (e.g., two directions perpendicular to each other). For example, the direction could be horizontal and vertical, or other directions. Furthermore, the bending direction can be predetermined or specified by the user, etc.

[0195] <Method 3-2>

[0196] For example, in the case of applying method 3, such as from Figure 19 As shown in the third row from the top of the table, information can be transmitted indicating whether the layer is non-planar, curvature specification information in a single direction, and whether the layer has a cylindrical shape (Method 3-2).

[0197] For example, in the first image processing apparatus, the first image processing method, and the first procedure, the layer shape selection information added to the bitstream may include information indicating whether the layer is non-planar, curvature specification information for specifying the curvature of the layer in a single direction, and information indicating whether the layer has a cylindrical shape. Similarly, in the second image processing apparatus, the second image processing method, and the second procedure, the layer shape selection information added to the bitstream may include information indicating whether the layer is non-planar, curvature specification information for specifying the curvature of the layer in a single direction, and information indicating whether the layer has a cylindrical shape.

[0198] For example, such as Figure 21 As shown in box 322 of section B, the parameters "mpii_non_plane_flag", "mpii_layer_curvature", and "mpii_cylindrical_layer_flag" can be added to the bitstream as layer shape selection information. The parameter "mpii_non_plane_flag" is a flag indicating whether the layer is non-planar. When the flag is true (e.g., "1"), this indicates that the layer is non-planar. When the flag is false (e.g., "0"), this indicates that the layer is not non-planar (i.e., this is MPI). The parameters "mpii_layer_curvature" and "mpii_cylindrical_layer_flag" are the information described above.

[0199] According to Figure 21 The syntax shown in box 322 of section B stores these parameters in the SEI of the bitstream added to the 3D data. Specifically, the parameter "mpii_non_plane_flag" can be set. When the flag is false, this indicates that the layer is planar (i.e., it is MPI). On the other hand, when the flag is true, this indicates that the layer is non-planar. In this case, the parameters "mpii_layer_curvature" and "mpii_cylindrical_layer_flag" can be set. When the parameter "mpii_cylindrical_layer_flag" is true, this indicates that the layer has a cylindrical shape. Therefore, the curvature indicated by the parameter "mpii_layer_curvature" can be applied to a single bending direction of the layer. This bending direction can be any direction, as long as it is a single direction. For example, the direction can be horizontal, vertical, or other directions. Furthermore, the bending direction can be predetermined or specified by the user, etc. Additionally, when the parameter "mpii_cylindrical_layer_flag" is false, this indicates that the layer has a spherical shape. Therefore, the curvature indicated by the parameter "mpii_layer_curvature" can be applied to multiple curvature directions of the layer. This curvature direction can be any direction, as long as it is multiple directions (e.g., two directions perpendicular to each other). For example, the direction can be horizontal and vertical, or other directions. Furthermore, the curvature direction can be predetermined or specified by the user, etc.

[0200] <Method 3-3>

[0201] For example, in the case of applying method 3, such as from Figure 19 As shown in the fourth row from the top of the table, curvature specification information in multiple directions can be transmitted (Method 3-3).

[0202] For example, in the first image processing apparatus, the first image processing method, and the first program, the layer shape selection information added to the bitstream may include curvature specification information for specifying the curvature of the layer in multiple directions. Similarly, in the second image processing apparatus, the second image processing method, and the second program, the layer shape selection information added to the bitstream may include curvature specification information for specifying the curvature of the layer in multiple directions.

[0203] For example, such as Figure 22 As shown in box 331 of A, the parameters "mpii_layer_curvature_h" and "mpii_layer_curvature_v" can be added to the bitstream as layer shape selection information. The parameter "mpii_layer_curvature_h" is curvature specification information used to specify the curvature of the layer in the horizontal direction. The parameter "mpii_layer_curvature_v" is curvature specification information used to specify the curvature of the layer in the vertical direction. That is, the parameters "mpii_layer_curvature_h" and "mpii_layer_curvature_v" can be set, and the curvature indicated by these parameters can be applied to the horizontal and vertical directions of the layer. When the values ​​of these parameters are both "0", this indicates that the layer is planar (i.e., this is MPI). When one parameter is "0" and the other is not "0", this indicates that the layer has a cylindrical shape (i.e., this is MCI). When the values ​​of these parameters are not all "0", this indicates that the layer has a spherical shape (i.e., this is MSI).

[0204] <Method 3-4>

[0205] For example, in the case of applying method 3, such as from Figure 19 As shown in the fifth row from the top of the table, information indicating whether the layer is non-planar and curvature specification information in multiple directions can be transmitted (methods 3-4).

[0206] For example, in the first image processing apparatus, the first image processing method, and the first program, the layer shape selection information added to the bitstream may include information indicating whether the layer is non-planar and curvature specification information for specifying the curvature of the layer in multiple directions. Similarly, in the second image processing apparatus, the second image processing method, and the second program, the layer shape selection information added to the bitstream may include information indicating whether the layer is non-planar and curvature specification information for specifying the curvature of the layer in multiple directions.

[0207] For example, such as Figure 22 As shown in box 332 of B, the parameters "mpii_non_plane_flag", "mpii_layer_curvature_h", and "mpii_layer_curvature_v" can be added to the bitstream as layer shape selection information. These parameters are the information described above.

[0208] According to Figure 22 The syntax shown in block 332 of section B stores these parameters in the SEI of the bitstream added to the 3D data. Specifically, the parameter "mpii_non_plane_flag" can be set. When the flag is false, this indicates that the layer is planar (i.e., this is MPI). Conversely, when the flag is true, this indicates that the layer has a non-planar shape. In this case, the parameters "mpii_layer_curvature_h" and "mpii_layer_curvature_v" can be set, and the curvature indicated by these parameters can be applied to the horizontal and vertical directions of the layer. Among these parameters, if one parameter is "0" and the other is not, this indicates that the layer has a cylindrical shape (i.e., this is MCI). If neither parameter is "0", this indicates that the layer has a spherical shape (i.e., this is MSI).

[0209] <Method 3-5>

[0210] For example, in the case of applying method 3, such as Figure 19 As shown in the bottom row of the table, information can be transmitted indicating whether the layer is non-planar, whether the layer has a cylindrical shape, and curvature specification information in a single or multiple directions (methods 3-5).

[0211] For example, in the first image processing apparatus, the first image processing method, and the first program, the layer shape selection information added to the bitstream may include information indicating whether the layer is non-planar, information indicating whether the layer has a cylindrical shape, and curvature specification information for specifying the curvature of the layer in multiple directions. Similarly, in the second image processing apparatus, the second image processing method, and the second program, the layer shape selection information added to the bitstream may include information indicating whether the layer is non-planar, information indicating whether the layer has a cylindrical shape, and curvature specification information for specifying the curvature of the layer in multiple directions.

[0212] For example, such as Figure 22 As shown in box 333 in C, the parameters "mpii_non_plane_flag", "mpii_cylindrical_layer_flag", "mpii_layer_curvature", or "mpii_layer_curvature_h" and "mpii_layer_curvature_v" can be added to the bitstream as layer shape selection information. These parameters are the information described above.

[0213] According to such Figure 23 The syntax shown in block 333 of C stores these parameters in the SEI of the bitstream added to the 3D data. That is, the parameter "mpii_non_plane_flag" can be set. When the flag is false, this indicates that the layer is planar (i.e., it is MPI). On the other hand, when the flag is true, this indicates that the layer has a non-planar shape. In this case, the parameter "mpii_cylindrical_layer_flag" can be set. When the parameter "mpii_cylindrical_layer_flag" is true, this indicates that the layer has a cylindrical shape. In this case, the parameter "mpii_layer_curvature" can be set, and the curvature indicated by the parameter "mpii_layer_curvature" can be applied to a single curvature direction of the layer. This curvature direction can be any direction, as long as it is a single direction. For example, the direction can be horizontal, vertical, or other directions. Furthermore, the curvature direction can be predetermined or specified by the user, etc. Additionally, when the parameter "mpii_cylindrical_layer_flag" is false, this indicates that the layer has a spherical shape. In this case, the parameters "mpii_layer_curvature_h" and "mpii_layer_curvature_v" can be set, and the curvature indicated by these parameters can be applied to the horizontal and vertical directions of the layer.

[0214] By applying any of the methods 3-1 to 3-5 described above, for example, the optimal method can be selected and applied from MPI, MCI, and MSI for the content. As a result, the degradation in the quality of the rendered image can be suppressed while inhibiting the unnecessary increase in the processing load such as encoding, decoding, and reproduction.

[0215] <Regarding the application of each method>

[0216] Provided there are no contradictions, each of the methods described above can be combined with other methods. For example, they can be appropriately combined and applied. Figure 7 The table shows several methods. Additionally, they can be appropriately combined and applied. Figure 15 The table shows several methods. Furthermore, they can be appropriately combined and applied. Figure 19 The table shows several methods. Furthermore, they can be appropriately combined and applied. Figure 7 Method 1 shown in the table Figure 15 Method 2 or shown in the table Figure 19 The table shows various methods in method 3. Additionally, the methods shown in the tables of a specific figure can be appropriately combined with the methods shown in tables outside that specific figure. Of course, these methods can also be combined with other methods (not shown).

[0217] Note that in this specification, the description of a higher-level method may include the description of a lower-level method. For example, a description such as "applying method 1" may also include applying any one or more of methods 1-1, 1-2, and 1-3. Similarly, a description such as "applying method 2" may include applying any one or more of methods 2-1, 2-2, 2-2-1, 2-2-2, and 2-2-3. Similarly, a description such as "applying method 2-2" may include applying any one or more of methods 2-2-1, 2-2-2, and 2-2-3. Similarly, a description such as "applying method 3" may include applying any one or more of methods 3-1, 3-2, 3-3, 3-4, and 3-5.

[0218] Therefore, for example, it can be Figure 7 Method 1 and the table shown in the table Figure 15 The method 2 shown in the table is applied in combination.

[0219] For example, a first image processing apparatus may include: an interlayer relative information generation unit that generates interlayer relative information about the relative relationships between layers of 3D data, the 3D data including layer images of color components and layer images of alpha components; a layer arrangement information generation unit that generates layer arrangement information about the arrangement of layers in the 3D data; and an encoding unit that generates a bitstream by encoding the 3D data and adds the interlayer relative information and layer arrangement information to the bitstream. For example, a first image processing method may include: generating interlayer relative information about the relative relationships between layers of 3D data by a first image processing apparatus, the 3D data including layer images of color components and layer images of alpha components; generating layer arrangement information about the arrangement of layers in the 3D data by the first image processing apparatus; and generating a bitstream by encoding the 3D data by the first image processing apparatus and adding the interlayer relative information and layer arrangement information to the bitstream. For example, the first program may cause the first image processing apparatus to perform the following operations: generating interlayer relative information about the relative relationships between layers of 3D data, the 3D data including layer images of color components and layer images of alpha components; generating layer arrangement information about the arrangement of layers of the 3D data; and generating a bitstream by encoding the 3D data, and adding the interlayer relative information and the layer arrangement information to the bitstream.

[0220] For example, a second image processing apparatus may include: a decoding unit that decodes a bitstream and generates 3D data including layer images of color components and alpha components; an interlayer relative information analysis unit that analyzes interlayer relative information about the relative relationships between layers in the 3D data extracted from the bitstream; a layer layout information analysis unit that analyzes layer layout information about the arrangement of layers in the 3D data extracted from the bitstream; and a viewpoint image generation unit that renders the 3D data based on the interlayer relative information and the layer layout information, and generates a viewpoint image viewed from any viewpoint. For example, a second image processing method may include: decoding a bitstream by a second image processing apparatus and generating 3D data including layer images of color components and alpha components; analyzing interlayer relative information about the relative relationships between layers in the 3D data extracted from the bitstream by the second image processing apparatus; analyzing layer layout information about the arrangement of layers in the 3D data extracted from the bitstream by the second image processing apparatus; and rendering the 3D data based on the interlayer relative information and the layer layout information, and generating a viewpoint image viewed from any viewpoint. For example, the second program may enable the second image processing apparatus to perform the following operations: decode the bitstream and generate 3D data including layer images of color components and layer images of alpha components; analyze interlayer relative information about the relative relationships between layers of the 3D data extracted from the bitstream; analyze layer arrangement information about the arrangement of layers of the 3D data extracted from the bitstream; and render the 3D data based on the interlayer relative information and the layer arrangement information, and generate a viewpoint image viewed from any viewpoint.

[0221] In addition, it can Figure 7 Method 1 and the table shown in the table Figure 19 The table shows the combined application of method 3.

[0222] For example, a first image processing apparatus may include: an interlayer relative information generation unit that generates interlayer relative information about the relative relationships between layers of 3D data, the 3D data including layer images of color components and layer images of alpha components; a layer shape selection information generation unit that generates layer shape selection information about the selection of shapes for layers of the 3D data; and an encoding unit that generates a bitstream by encoding the 3D data and adds the interlayer relative information and layer shape selection information to the bitstream. For example, a first image processing method may include: generating interlayer relative information about the relative relationships between layers of 3D data by a first image processing apparatus, the 3D data including layer images of color components and layer images of alpha components; generating layer shape selection information about the selection of shapes for layers of the 3D data by the first image processing apparatus; and generating a bitstream by encoding the 3D data by the first image processing apparatus and adding the interlayer relative information and layer shape selection information to the bitstream. For example, the first program may cause the first image processing apparatus to perform the following operations: generating interlayer relative information about the relative relationships between layers of 3D data, the 3D data including layer images of color components and layer images of alpha components; generating layer shape selection information about the selection of shapes for the layers of the 3D data; and generating a bitstream by encoding the 3D data, and adding the interlayer relative information and the layer shape selection information to the bitstream.

[0223] For example, the second image processing apparatus may include: a decoding unit that decodes a bitstream and generates 3D data including layer images of color components and layer images of alpha components; an interlayer relative information analysis unit that analyzes interlayer relative information about the relative relationships between layers of the 3D data extracted from the bitstream; a layer shape selection information analysis unit that analyzes layer shape selection information about the selection of the shapes of the layers of the 3D data extracted from the bitstream; and a viewpoint image generation unit that renders the 3D data based on the interlayer relative information and the layer shape selection information, and generates a viewpoint image viewed from any viewpoint. For example, a second image processing method may include: decoding a bitstream by a second image processing device and generating 3D data including layer images of color components and alpha components; analyzing interlayer relative information extracted from the bitstream regarding the relative relationships between layers of the 3D data; analyzing layer shape selection information extracted from the bitstream regarding the selection of shapes for layers of the 3D data; and rendering the 3D data based on the interlayer relative information and layer shape selection information, and generating a viewpoint image viewed from any viewpoint. For example, a second procedure may cause the second image processing device to perform the following operations: decoding a bitstream and generating 3D data including layer images of color components and alpha components; analyzing interlayer relative information extracted from the bitstream regarding the relative relationships between layers of the 3D data; analyzing layer shape selection information extracted from the bitstream regarding the selection of shapes for layers of the 3D data; and rendering the 3D data based on the interlayer relative information and layer shape selection information, and generating a viewpoint image viewed from any viewpoint.

[0224] In addition, they can be appropriately combined and applied. Figure 7 Method 1 shown in the table Figure 15 Method 2 and shown in the table Figure 19 The table shows several methods in way 3.

[0225] For example, the first image processing apparatus may include: an interlayer relative information generation unit that generates interlayer relative information about the relative relationships between layers of 3D data, the 3D data including layer images of color components and layer images of alpha components; a layer arrangement information generation unit that generates layer arrangement information about the arrangement of layers in the 3D data; a layer shape selection information generation unit that generates layer shape selection information about the selection of shapes for layers in the 3D data; and an encoding unit that generates a bitstream by encoding the 3D data and adds the interlayer relative information, layer arrangement information, and layer shape selection information to the bitstream. For example, a first image processing method may include: generating interlayer relative information about the relative relationships between layers of 3D data, the 3D data including layer images of color components and layer images of alpha components, by a first image processing apparatus; generating layer arrangement information about the arrangement of layers in the 3D data by the first image processing apparatus; generating layer shape selection information about the selection of shapes for the layers in the 3D data by the first image processing apparatus; and generating a bitstream by the first image processing apparatus by encoding the 3D data, and adding the interlayer relative information, layer arrangement information, and layer shape selection information to the bitstream. For example, a first program may cause the first image processing apparatus to perform the following operations: generating interlayer relative information about the relative relationships between layers of 3D data, the 3D data including layer images of color components and layer images of alpha components; generating layer arrangement information about the arrangement of layers in the 3D data; generating layer shape selection information about the selection of shapes for the layers in the 3D data; and generating a bitstream by encoding the 3D data, and adding the interlayer relative information, layer arrangement information, and layer shape selection information to the bitstream.

[0226] For example, the second image processing apparatus may include: a decoding unit that decodes a bitstream and generates 3D data including layer images of color components and layer images of alpha components; an interlayer relative information analysis unit that analyzes interlayer relative information about the relative relationships between layers of the 3D data extracted from the bitstream; a layer arrangement information analysis unit that analyzes layer arrangement information about the arrangement of layers of the 3D data extracted from the bitstream; a layer shape selection information analysis unit that analyzes layer shape selection information about the selection of the shape of layers of the 3D data extracted from the bitstream; and a viewpoint image generation unit that renders the 3D data based on the interlayer relative information, layer arrangement information, and layer shape selection information, and generates a viewpoint image viewed from any viewpoint. For example, the second image processing method may include: decoding a bitstream by a second image processing device and generating 3D data including layer images of color components and alpha components; analyzing interlayer relative information about the relative relationships between layers of the 3D data extracted from the bitstream by the second image processing device; analyzing layer arrangement information about the arrangement of layers of the 3D data extracted from the bitstream by the second image processing device; analyzing layer shape selection information about the selection of the shapes of layers of the 3D data extracted from the bitstream by the second image processing device; and rendering the 3D data by the second image processing device based on the interlayer relative information, layer arrangement information, and layer shape selection information, and generating a viewpoint image viewed from any viewpoint. For example, the second program may enable the second image processing apparatus to perform the following operations: decode the bitstream and generate 3D data including layer images of color components and layer images of alpha components; analyze interlayer relative information about the relative relationships between layers of the 3D data extracted from the bitstream; analyze layer arrangement information about the arrangement of layers of the 3D data extracted from the bitstream; analyze layer shape selection information about the selection of the shapes of layers of the 3D data extracted from the bitstream; and render the 3D data based on the interlayer relative information, layer arrangement information, and layer shape selection information, and generate a viewpoint image viewed from any viewpoint.

[0227] <4. First Implementation Method>

[0228] <Encoding device>

[0229] This technology can be applied to any device. For example, it can be applied to an encoding device that encodes 3D data of layer images including color components and alpha components, and generates a bitstream. Figure 23 This is a block diagram illustrating an example configuration of an encoding device as one aspect of an image processing apparatus to which this technology is applied. Figure 23 The encoding apparatus 500 (first image processing apparatus) shown is an apparatus for generating 3D data, including layer images of color components and alpha components, using multi-view images, encoding the 3D data, and generating its bitstream. Therefore, the encoding apparatus 500 can also be considered as a 3D data generation apparatus for generating 3D data or a bitstream generation apparatus for generating bitstreams.

[0230] Figure 23 It shows the main processing units, data flow, etc., and Figure 23 Those shown are not necessarily the entire configuration. That is, within the encoding device 500, there may be... Figure 23 The processing unit not shown in the diagram is a block, or it may exist. Figure 23 The processing or data flow is not indicated by arrows or other means.

[0231] like Figure 23 As shown, the encoding device 500 (first image processing device) includes a layer image generation unit 511, a layer image packing unit 512, an SEI generation unit 513, an encoding unit 514, a storage unit 515, and a transmission unit 516.

[0232] The layer image generation unit 511 performs processing related to the generation of layer images constituting 3D data. For example, the layer image generation unit 511 can acquire multi-view images provided from outside the encoding device 500. Furthermore, the layer image generation unit 511 can acquire setting information regarding layer generation, such as information input by a user or through an application.

[0233] The settings information can include any type of information, as long as it is related to layer generation. For example, settings related to the resolution and size of the layer image can be included in the settings information. Furthermore, settings related to the number of layers to be generated can also be included in the settings information.

[0234] Furthermore, when applying method 1 described above, settings related to the relative relationships between layers can be included in the setting information. For example, the setting information may include settings related to the relative number of alpha layers relative to color layers, settings related to the relative resolution of alpha layers relative to color layers, etc.

[0235] Furthermore, when applying method 2 described above, settings related to the layer arrangement can be included in the setting information. For example, the setting information may include settings related to the arrangement of additional layers, settings related to the arrangement of layers based on a non-equidistant arrangement function, etc.

[0236] Furthermore, when applying method 3 described above, settings related to the selection of the layer's shape can be included in the setting information. For example, the setting information may include settings related to specifying curvature, settings related to whether the layer has a cylindrical shape, settings related to whether the layer is non-planar, etc.

[0237] The layer image generation unit 511 can generate layer images constituting 3D data based on setting information and using multi-view images. In addition, the layer image generation unit 511 can generate metadata of the layer images (which is also considered metadata of the 3D data).

[0238] Metadata can contain anything. For example, information indicating the resolution and size of the layer images that constitute the 3D data generated by the layer image generation unit 511 can be included in the metadata. Furthermore, information indicating the number of layers that constitute the 3D data generated by the layer image generation unit 511 can be included in the metadata.

[0239] Furthermore, when applying method 1 described above, information regarding the relative relationships between layers in the 3D data generated in the layer image generation unit 511 can be included in the metadata. For example, the metadata may include information indicating the relative number of alpha layers relative to color layers, information indicating the relative resolution of alpha layers relative to color layers, etc., in the 3D data generated by the layer image generation unit 511.

[0240] Furthermore, when applying method 2 described above, information regarding the arrangement of layers in the 3D data generated in the layer image generation unit 511 can be included in the metadata. For example, the metadata may include information about the arrangement of additional layers in the 3D data generated by the layer image generation unit 511, information about the arrangement of layers based on a non-equidistant arrangement function, etc.

[0241] Furthermore, when applying method 3 described above, information regarding the selection of the layer's shape can be included in the metadata. For example, the metadata may include information in the 3D data generated by the layer image generation unit 511 indicating the curvature of the layer, information indicating whether the layer has a cylindrical shape, information indicating whether the layer is non-planar, etc.

[0242] The method for generating layer images (3D data) and their metadata can be any method. For example, the layer image generation unit 511 can have a learning model (neural network) that receives input from setup information and multi-view images, and outputs 3D data of layer images including color components and alpha components, along with their metadata. The layer image generation unit 511 inputs the provided setup information and multi-view images into the learning model, and causes the learning model to output layer images and their metadata. The layer image generation unit 511 can provide the layer image packaging unit 512 with the layer image group (i.e., 3D data of layer images including color components and alpha components) and its metadata generated in this way.

[0243] The layer image packaging unit 512 performs processing related to packaging layer images. For example, the layer image packaging unit 512 can acquire a group of layer images (i.e., 3D data of layer images including color components and alpha components) and their metadata provided by the layer image generation unit 511. In addition, the layer image packaging unit 512 can acquire control information regarding the packaging of layer images, such control information being input by a user or through an application.

[0244] Control information can include any type of information, as long as it relates to packing layer images. For example, control information can include information for specifying packing methods (e.g., whether to stack layers along a spatial direction (vertical or horizontal), whether to stack layers along a temporal direction, etc.).

[0245] The layer image packing unit 512 can pack the layer image group based on the provided control information and generate the packed layer image group, i.e., the packed image. The layer image packing unit 512 can provide the generated packed image to the encoding unit 514. In addition, the layer image packing unit 512 can provide the provided metadata to the SEI generation unit 513.

[0246] SEI generation unit 513 performs processing related to SEI generation. For example, SEI generation unit 513 can obtain metadata provided by layer image packing unit 512. SEI generation unit 513 can generate SEI based on metadata. That is, SEI generation unit 513 can generate SEI that includes metadata. SEI can include any type of information.

[0247] SEI generation unit 513 may include, for example, interlayer relative information generation unit 521, layer layout information generation unit 522, and layer shape selection information generation unit 523.

[0248] The interlayer relative information generation unit 521 performs processing related to the generation of interlayer relative information. For example, the interlayer relative information generation unit 521 may generate interlayer relative information for 3D data based on information about the relative relationships between layers in the 3D data generated by the layer image generation unit 511, included in the metadata provided to the SEI generation unit 513. For example, the interlayer relative information generation unit 521 may generate information about the relative number of alpha layers relative to color layers, information about the relative resolution of alpha layers relative to color layers, etc., as interlayer relative information for 3D data.

[0249] The layer layout information generation unit 522 performs processing related to the generation of layer layout information. For example, the layer layout information generation unit 522 can generate layer layout information for 3D data based on information about the layer layout of the 3D data generated by the layer image generation unit 511, which is included in the metadata provided to the SEI generation unit 513. For example, the layer layout information generation unit 522 can generate information about the layout of additional layers, information about the layout of layers based on a non-equidistant layout function, etc., as layer layout information for 3D data.

[0250] The layer shape selection information generation unit 523 performs processing related to the generation of layer shape selection information. For example, the layer shape selection information generation unit 523 may generate layer shape selection information for 3D data based on information about the selection of the shape of layers in the 3D data generated by the layer image generation unit 511, which is included in the metadata provided to the SEI generation unit 513. For example, the layer shape selection information generation unit 523 may generate information such as curvature specification information of the layer in a single or multiple directions, information indicating whether the layer has a cylindrical shape, and information indicating whether the layer is non-planar as layer shape selection information for the 3D data.

[0251] As described above, the SEI generation unit 513 can generate an SEI that includes inter-layer relative information, layer arrangement information, and layer shape selection information. Of course, the SEI can include information other than these types. For example, the SEI can include information indicating the resolution and size of the layer images constituting the 3D data generated by the layer image generation unit 511, information indicating the number of layers constituting the 3D data, etc.

[0252] SEI generation unit 513 can provide the generated SEI to encoding unit 514.

[0253] Note that in SEI generation unit 513, if method 1 described above is not applied, the interlayer relative information generation unit 521 can be omitted. Furthermore, if method 2 described above is not applied, the layer layout information generation unit 522 can be omitted. Furthermore, if method 3 described above is not applied, the layer shape selection information generation unit 523 can be omitted.

[0254] Encoding unit 514 performs processing related to the encoding of 3D data. For example, encoding unit 514 can acquire a packaged image provided by layer image packaging unit 512. Encoding unit 514 can acquire an SEI provided by SEI generation unit 513. Encoding unit 514 can encode the acquired packaged image and generate a bitstream as encoded data. The encoding method can be any method. For example, encoding unit 514 can encode the packaged image using a predetermined 2D codec. Furthermore, encoding unit 514 can add the acquired SEI to the bitstream. Encoding unit 514 can provide the bitstream generated in this way and with the SEI added to it to storage unit 515. Furthermore, encoding unit 514 can provide the bitstream to transmission unit 516.

[0255] Storage unit 515 includes a storage medium such as a hard disk or semiconductor memory, and utilizes the storage medium to perform processes related to the storage and retrieval of information. For example, storage unit 515 can acquire a bit stream provided from encoding unit 514 and store the bit stream in the storage medium. Storage unit 515 can read the bit stream stored in the storage medium at a predetermined time or based on a request from an external source (e.g., transmission unit 516), and provide the bit stream to transmission unit 516. Furthermore, storage unit 515 may include a removable medium and its driver, and store the acquired bit stream in the removable medium attached to the driver. In this case, the removable medium can be removed from the driver. That is, the bit stream can be output to the outside of encoding device 500 while stored in the removable medium.

[0256] The transmission unit 516 includes a communication device for communicating with other devices, and performs processing related to information transmission using the communication device. For example, the transmission unit 516 can acquire a bit stream provided by the encoding unit 514. Furthermore, the transmission unit 516 can acquire a bit stream read from the storage unit 515. The transmission unit 516 can use the aforementioned communication device to transmit the acquired bit stream to other devices.

[0257] In such an encoding apparatus 500, the technique described above in <3. Regarding the extension of layers> can be applied. For example, method 1 can be applied to the encoding apparatus 500. In this case, the interlayer relative information generation unit 521 can generate interlayer relative information about the relative relationships between layers of 3D data, which includes layer images of color components and layer images of alpha components. Furthermore, the encoding unit 514 can generate a bitstream by encoding the 3D data and add the interlayer relative information to the bitstream.

[0258] In this way, for example, even if the relative relationships between layers are optimized according to content, the decoder can easily identify these relationships based on the inter-layer relative information. Therefore, the encoding device 500 can suppress the degradation of the quality of the rendered image.

[0259] Alternatively, method 2 can be applied to the encoding device 500. In this case, the layer arrangement information generation unit 522 can generate layer arrangement information about the arrangement of layers in 3D data, which includes layer images of color components and layer images of alpha components. Furthermore, the encoding unit 514 can generate a bitstream by encoding the 3D data and add the layer arrangement information to the bitstream.

[0260] In this way, the decoder can easily arrange the layers at any location based on the layer arrangement information. Therefore, for example, the encoding device 500 can arrange the layers at non-uniform intervals while suppressing unnecessary increases in data volume. As a result, for example, the encoding device 500 can optimize the layer arrangement according to content, etc. Therefore, the encoding device 500 can suppress the degradation of rendered image quality while suppressing unnecessary increases in data volume.

[0261] Alternatively, method 3 can be applied to the encoding device 500. In this case, the layer shape selection information generation unit 523 can generate layer shape selection information regarding the selection of the shape of layers in 3D data, which includes layer images of color components and layer images of alpha components. Furthermore, the encoding unit 514 can generate a bitstream by encoding the 3D data and add the layer shape selection information to the bitstream.

[0262] In this way, the encoding device 500 can select the shape of the layer. For example, the encoding device 500 can select and apply the best method from MPI, MCI, and MSI for the content. As a result, the encoding device 500 can suppress the degradation of the quality of the rendered image while suppressing the unnecessary increase in the load of processing such as encoding, decoding, and reproduction.

[0263] <Encoding Process>

[0264] Reference Figure 24 The flowchart is used to illustrate an example of the encoding process performed by the encoding device 500.

[0265] When the encoding process begins, in step S501, the layer image generation unit 511 generates a layer image and its metadata using multi-view images based on the input settings related to layer generation.

[0266] In step S502, the layer image packaging unit 512 packages the layer image generated in step S501 according to the input control information and generates a packaged image.

[0267] In step S503, the interlayer relative information generation unit 521 uses the metadata generated in step S501 to generate interlayer relative information as SEI.

[0268] In step S504, the layer layout information generation unit 522 uses the metadata generated in step S501 to generate layer layout information as SEI.

[0269] In step S505, the layer shape selection information generation unit 523 uses the metadata generated in step S501 to generate layer shape selection information as SEI.

[0270] In step S506, encoding unit 514 encodes the packaged image generated in step S502 and generates its bitstream. Furthermore, encoding unit 514 adds the SEI generated in each of steps S503 to S505 to the bitstream.

[0271] In step S507, storage unit 515 stores the bit stream generated in step S506 and to which SEI has been added.

[0272] In step S508, storage unit 515 reads the bit stream stored in step S507.

[0273] In step S509, the transmission unit 516 transmits the bit stream generated in step S506 and with SEI added, or the bit stream read from the storage unit 515 in step S508.

[0274] The encoding process ends when the processing in step S509 is completed.

[0275] By performing the processing operation in this manner, the encoding device 500 can suppress the degradation of the quality of the rendered image.

[0276] Note that if method 1 described above is not applied, step S503 can be skipped (omitted). If method 2 described above is not applied, step S504 can be skipped (omitted). If method 3 described above is not applied, step S505 can be skipped (omitted).

[0277] <5. Second Implementation Method>

[0278] <Decoding device>

[0279] This technology can be applied to a decoding device that decodes a bitstream obtained by encoding 3D data, which consists of layer images of color components and layer images of alpha components. Figure 25 This is a block diagram illustrating an example configuration of a decoding device as one aspect of an image processing apparatus to which this technology is applied. Figure 25 The decoding device 600 (second image processing device) shown is, for example, a device composed of encoding device 500 ( Figure 23 A device for decoding the generated bitstream, generating (reconstructing) 3D data including layer images of color components and alpha components, generating a viewpoint image by rendering 3D data at a desired viewpoint, and outputting the viewpoint image.

[0280] Figure 25 It shows the main processing units, data flow, etc., and Figure 25 Those shown are not necessarily the entire configuration. That is, within the decoding device 600, there may be... Figure 25 The processing unit not shown in the diagram is a block, or it may exist. Figure 25 The processing or data flow is not indicated by arrows or other means.

[0281] like Figure 25 As shown, the decoding device 600 (second image processing device) includes a receiving unit 611, a storage unit 612, a decoding unit 613, an SEI analysis unit 614, a layer image unpacking unit 615, a rendering unit 616, and an output unit 617.

[0282] The receiving unit 611 has a communication function for communicating with other devices, and uses this communication function to perform processing related to information reception. For example, the receiving unit 611 can acquire a bit stream transmitted from the encoding device 500 (transmission unit 516). The receiving unit 611 can provide the received bit stream to the storage unit 612. Furthermore, the receiving unit 611 can provide the received bit stream to the decoding unit 613.

[0283] Storage unit 612 includes a storage medium such as a hard disk or semiconductor memory, and utilizes the storage medium to perform processes related to the storage and retrieval of information. For example, storage unit 612 can acquire a bit stream provided from receiving unit 611 and store the bit stream in the storage medium. Storage unit 612 can read the bit stream stored in the storage medium at a predetermined time or based on a request from an external source (e.g., decoding unit 613), and provide the bit stream to decoding unit 613. Furthermore, storage unit 612 may include a removable medium and its driver, read a bit stream stored in the removable medium attached to the driver at a predetermined time or based on a request from an external source (e.g., decoding unit 613), and provide the bit stream to decoding unit 613. That is, the bit stream can be provided from encoding device 500, etc., to decoding device 600 while stored in the removable medium.

[0284] Decoding unit 613 performs processing related to the decoding of the bitstream. For example, decoding unit 613 can acquire the bitstream provided from receiving unit 611 or storage unit 612. Decoding unit 613 can extract the SEI added to the acquired bitstream and provide the SEI to SEI analysis unit 614. Furthermore, decoding unit 613 can decode the acquired bitstream and generate a layered image of (reconstructed) 3D data, which includes layer images of color components and layer images of alpha components. Note that the decoding method can be any method. For example, decoding unit 613 can decode the bitstream using a predetermined 2D codec. Decoding unit 613 can provide the layered image unpacking unit 615 with the layered image unpacking image generated as described above.

[0285] SEI analysis unit 614 performs processing related to SEI analysis. For example, SEI analysis unit 614 can acquire the SEI provided by decoding unit 613. SEI can include any type of information. For example, SEI can include inter-layer relative information, layer arrangement information, and layer shape selection information. Of course, the SEI can include information other than these types of information. For example, SEI can include information indicating the resolution and size of the layer image constituting the 3D data, or information indicating the number of layers constituting the 3D data.

[0286] SEI analysis unit 614 can analyze SEI and generate metadata for 3D data based on the analysis results. That is, SEI analysis unit 614 can generate metadata that reflects the content of the acquired SEI.

[0287] SEI analysis unit 614 may include, for example, interlayer relative information analysis unit 621, layer layout information analysis unit 622, and layer shape selection information analysis unit 623.

[0288] The interlayer relative information analysis unit 621 performs processing related to the analysis of the interlayer relative information included in the SEI. For example, the interlayer relative information analysis unit 621 can analyze the interlayer relative information included in the SEI provided by the decoding unit 613, and generate information about the relative relationships between layers of the 3D data as metadata of the 3D data based on the analysis results. For example, in this way, the interlayer relative information analysis unit 621 can generate information in the 3D data indicating the relative number of alpha layers relative to color layers, information indicating the relative resolution of alpha layers relative to color layers, etc., as metadata of the 3D data.

[0289] The layer layout information analysis unit 622 performs processing related to the analysis of layer layout information included in the SEI. For example, the layer layout information analysis unit 622 can analyze the layer layout information included in the SEI provided by the decoding unit 613, and generate information about the layer layout of the 3D data as metadata of the 3D data based on the analysis results. For example, in this way, the layer layout information analysis unit 622 can generate information about the layout of additional layers in the 3D data, information about the layer layout based on a non-equidistant layout function, etc., as metadata of the 3D data.

[0290] The layer shape selection information analysis unit 623 performs processing related to the analysis of layer shape selection information included in the SEI. For example, the layer shape selection information analysis unit 623 can analyze the layer shape selection information included in the SEI provided by the decoding unit 613, and generate information about the selection of layer shapes in the 3D data as metadata of the 3D data based on the analysis results. For example, in this way, the layer shape selection information analysis unit 623 can generate information in the 3D data indicating the curvature of the layer, information indicating whether the layer has a cylindrical shape, information indicating whether the layer is non-planar, etc., as metadata of the 3D data.

[0291] SEI analysis unit 614 can provide the metadata generated as described above to layer image unpacking unit 615.

[0292] The layer image unpacking unit 615 performs processing related to unpacking the packaged image. For example, the layer image unpacking unit 615 can acquire the packaged image provided by the decoding unit 613. Furthermore, the layer image unpacking unit 615 can acquire metadata provided by the SEI analysis unit 614. The layer image unpacking unit 615 can unpack the acquired packaged image according to the control information included in the acquired metadata, and generate unpacked layer image groups (layer image groups for color components and layer image groups for alpha components). The layer image unpacking unit 615 can arrange the layer image groups based on the information included in the metadata, and generate 3D data including layer images for color components and layer images for alpha components. The layer image unpacking unit 615 can provide the generated 3D data (layer image groups) and its metadata to the rendering unit 616.

[0293] The rendering unit 616 performs processing related to the rendering of 3D data. For example, the rendering unit 616 can acquire 3D data (layer image group) and its metadata provided by the layer image unpacking unit 615. Furthermore, the rendering unit 616 can acquire viewpoint position information, such as that input by a user or through an application. Viewpoint position information indicates the position, pose, etc., of the viewpoint when rendering the 3D data. The rendering unit 616 can render the 3D data based on the viewpoint position information and metadata. That is, the rendering unit 616 can determine the content of the 3D data based on the metadata, perform rendering at the viewpoint indicated by the viewpoint position information, and generate a viewpoint image (rendered image). In other words, the rendering unit 616 can also be considered a viewpoint image generation unit that generates the viewpoint image. The rendering unit 616 can provide the generated viewpoint image to the output unit 617.

[0294] Output unit 617 performs processing related to the output of the viewpoint image. For example, output unit 617 may include a display device such as a monitor, and utilize the display device to display the viewpoint image provided from rendering unit 616. Output unit 617 may include a communication device that communicates with other devices, and may utilize the communication device to transmit the viewpoint image provided from rendering unit 616 to the outside of decoding device 600 (e.g., other devices). Output unit 617 may include a removable medium and its driver, and may store the viewpoint image provided from rendering unit 616 in the removable medium attached to the driver via the driver. The removable medium is configured to be detachable from the driver.

[0295] In such a decoding apparatus 600, the technique described above in <3. Extensions about layers> can be applied. For example, method 1 can be applied to the decoding apparatus 600. In this case, the decoding unit 613 can decode the bitstream and generate 3D data including layer images of color components and layer images of alpha components. The interlayer relative information analysis unit 621 can analyze the interlayer relative information about the relative relationships between layers of the 3D data extracted from the bitstream. The rendering unit 616 can render the 3D data based on the interlayer relative information and generate a viewpoint image viewed from any viewpoint.

[0296] In this way, for example, even if the encoder optimizes the relative relationships between layers based on content, the decoding device 600 can easily identify these relationships based on the inter-layer relative information. Therefore, the decoding device 600 can suppress the degradation of the rendered image quality.

[0297] Alternatively, method 2 can be applied to the decoding device 600. In this case, the decoding unit 613 can decode the bitstream and generate 3D data including layer images of color components and alpha components. The interlayer relative information analysis unit 621 can analyze layer arrangement information about the arrangement of layers in the 3D data extracted from the bitstream. The rendering unit 616 can render the 3D data based on the layer arrangement information and generate a viewpoint image viewed from any viewpoint.

[0298] In this way, the decoding device 600 can easily arrange the layers at any position based on the layer arrangement information. Therefore, the decoding device 600 can, for example, arrange the layers at non-uniform intervals while suppressing unnecessary increases in data volume. As a result, the decoding device 600 can optimize the layer arrangement, for example, based on the content. Therefore, the decoding device 600 can suppress the degradation of the rendered image quality while suppressing unnecessary increases in data volume.

[0299] Alternatively, method 3 can be applied to the decoding device 600. In this case, the decoding unit 613 can decode the bitstream and generate 3D data including layer images of color components and alpha components. The interlayer relative information analysis unit 621 can analyze layer shape selection information extracted from the bitstream regarding the selection of the shapes of the layers in the 3D data. The rendering unit 616 can render the 3D data based on the layer shape selection information and generate a viewpoint image viewed from any viewpoint.

[0300] In this way, the encoder can select the shape of the layer. For example, the encoder can select and apply the best method from MPI, MCI, and MSI for the content. The decoding device 600 can easily recognize the selection result. As a result, the decoding device 600 can suppress the degradation of the rendered image quality while suppressing the unnecessary increase in the load of processing such as encoding, decoding, and reproduction.

[0301] <Decoding Process>

[0302] Reference Figure 26 The flowchart is used to illustrate an example of the decoding process performed by the decoding device 600.

[0303] When the decoding process begins, in step S601, the receiving unit 611 receives the bit stream.

[0304] In step S602, storage unit 612 stores the bit stream.

[0305] In step S603, storage unit 612 reads the bit stream.

[0306] In step S604, the decoding unit 613 decodes the bitstream and generates a packetized image.

[0307] In step S605, the interlayer relative information analysis unit 621 in the SEI analysis unit 614 analyzes the interlayer relative information included in the SEI extracted from the bitstream, and generates information about the relative relationship between layers of the 3D data based on the analysis results, as metadata of the 3D data.

[0308] In step S606, the layer layout information analysis unit 622 in the SEI analysis unit 614 analyzes the layer layout information included in the SEI extracted from the bitstream, and generates information about the layer layout of the 3D data based on the analysis results, as metadata of the 3D data.

[0309] In step S607, the layer shape selection information analysis unit 623 in the SEI analysis unit 614 analyzes the layer shape selection information included in the SEI extracted from the bitstream, and generates information about the selection of the shape of the layers of the 3D data based on the analysis results, as metadata of the 3D data.

[0310] In step S608, the layer image unpacking unit 615 unpacks the packaged image generated in step S604 and generates layer image groups for color components and layer image groups for alpha components. The layer image unpacking unit 615 arranges the layer image groups based on the information included in the metadata generated in each of steps S605 to S607, and generates 3D data including layer images for color components and alpha components.

[0311] In step S609, the rendering unit 616 renders 3D data at the viewpoint indicated by the viewpoint location information based on the metadata generated in each of the processes in steps S605 to S607, and generates its viewpoint image.

[0312] In step S610, the output unit 617 outputs the viewpoint image.

[0313] The decoding process ends when step S610 is completed.

[0314] By performing the processing in this manner, the decoding device 600 can suppress the degradation of the quality of the rendered image.

[0315] Note that if method 1 described above is not applied, step S605 can be skipped (omitted). If method 2 described above is not applied, step S606 can be skipped (omitted). If method 3 described above is not applied, step S607 can be skipped (omitted).

[0316] <6. Supplementary Explanation>

[0317] <3D Data>

[0318] In the above description, 3D data is described using MPI, MCI, MSI, and NeX, which applies MPI, MCI, or MSI, as examples. However, this technique is not limited to these examples and can be applied to any type of 3D data, including layer images of color components and layer images of alpha components.

[0319] Computers

[0320] The aforementioned series of processing operations can be performed by hardware or software. When these processing operations are performed by software, the program constituting the software is installed on the computer (also known as an information processing device or image processing device). Here, the computer includes computers embedded in dedicated hardware, as well as general-purpose computers, for example, that can perform various functions by installing various programs.

[0321] Figure 27 This is a block diagram illustrating an example of the hardware configuration of a computer that performs the above series of processing operations through a program.

[0322] exist Figure 27 In the computer 900 shown, the central processing unit (CPU) 901, read-only memory (ROM) 902 and random access memory (RAM) 903 are connected to each other via bus 904.

[0323] The input / output interface 910 is also connected to the bus 904. The input unit 911, output unit 912, storage unit 913, communication unit 914 and driver 915 are connected to the input / output interface 910.

[0324] Input unit 911 includes, for example, a keyboard, mouse, microphone, touch panel, input terminals, etc. Output unit 912 includes, for example, a display, speaker, output terminals, etc. Storage unit 913 includes, for example, a hard disk, RAM disk, non-volatile memory, etc. Communication unit 914 includes, for example, a network interface. Driver 915 drives removable media 921, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0325] In a computer configured as described above, for example, CPU 901 loads a program stored in storage unit 913 into RAM 903 via input / output interface 910 and bus 904, and executes the program to perform a series of processing operations as described above. RAM 903 also appropriately stores data required by CPU 901 to perform various processing operations.

[0326] For example, a program executed by a computer can be applied by being recorded on a removable medium 921, which serves as the encapsulation medium. In this case, the program can be installed on the storage unit 913 via the input / output interface 910 by loading the removable medium 921 into the drive 915.

[0327] Alternatively, the program can be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received via the communication unit 914 and installed on the storage unit 913.

[0328] Alternatively, the program can be pre-installed on ROM 902 or storage unit 913.

[0329] <Targets to which this technology can be applied>

[0330] This technology can be applied to any encoding / decoding method.

[0331] Furthermore, this technology can be applied to any form. For example, it can be applied to various electronic devices.

[0332] In addition, this technology can also be implemented as part of the configuration of a device, for example, as a processor (e.g., a video processor) in a system-wide integrated circuit (LSI), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, a collection including units with added functions (e.g., a video collection), etc.

[0333] Furthermore, this technology can also be applied to network systems comprising multiple devices. For example, this technology can also be implemented as cloud computing, where processing is shared and collaboratively performed by multiple devices via a network. For example, this technology can be implemented in cloud services that provide image (moving image) related services to any terminal, such as computers, audiovisual (AV) devices, portable information processing terminals, and Internet of Things (IoT) devices.

[0334] Note that in this specification, "system" refers to a collection of multiple components (devices, modules (parts), etc.), and it is not important whether all components are housed in the same housing. Therefore, "system" refers to both multiple devices housed in separate housings and connected via a network, and multiple modules housed in a single housing.

[0335] <Fields and applications of this technology>

[0336] Systems, devices, and processing units utilizing this technology can be used in any field, such as transportation, medical care, crime prevention, agriculture, animal husbandry, mining, beauty and personal care, factories, home appliances, weather, and nature monitoring. Furthermore, their applications are entirely flexible.

[0337] For example, this technology can be applied to systems or devices provided for offering viewing content, etc. Furthermore, for example, this technology can be applied to systems and devices provided for transportation, such as traffic condition monitoring and automatic driving control. Furthermore, for example, this technology can be applied to systems or devices provided for safety, etc. Furthermore, for example, this technology can be applied to systems or devices provided for automatic machine control, etc. Furthermore, for example, this technology can be applied to systems and devices used in agriculture and animal husbandry. Furthermore, this technology can be applied to systems and devices for monitoring the natural state of things such as volcanoes, forests, and oceans, and wildlife, etc. Furthermore, for example, this technology can be applied to systems and devices provided for sports, etc.

[0338] <Other>

[0339] Note that in this specification, "flag" refers to information used to identify multiple states, including not only information identifying the states of true (1) and false (0), but also information capable of identifying three or more states. Therefore, such a "flag" can take values ​​such as 1 and 0, or three or more values. That is, the number of bits constituting the "flag" is freely chosen and can be one bit or more bits. Furthermore, since identification information (including tags) is assumed to be not only in the form of identification information included in the bitstream, but also in the form of difference information relative to information used as a specific reference included in the bitstream, in this specification, "tag" and "identification information" include not only the information itself, but also the difference information relative to the information used as a reference.

[0340] Furthermore, various types of information (metadata, etc.) about the captured image can be transmitted or recorded in any form, as long as such information is associated with the captured image. Here, the term "associated" means, for example, that one piece of data can be used (linked) while processing another piece of data. That is, related data can be collected as a single data set or processed as separate data sets. For example, information associated with encoded data (images) can be transmitted on a different transmission path than the transmission path of the encoded data (images). Additionally, for example, information associated with encoded data (images) can be recorded on a different recording medium than the recording medium used for the encoded data (images) (or in different recording areas of the same recording medium). Note that "associated" can be applied to a portion of the data, not the entire data. For example, an image and the information corresponding to that image can be associated with each other in any unit, such as multiple frames, a single frame, or a portion of a frame.

[0341] Note that in this specification, terms such as “composite,” “multiplex,” “add,” “integrate,” “include,” “store,” “place,” “insert,” and “embed” mean to put multiple things together into one thing, for example, to put encoded data and metadata together into one piece of data, and also mean a method of “association” mentioned above.

[0342] Furthermore, the implementation of this technology is not limited to the above-described implementation, but can be modified in various ways without departing from the spirit of this technology.

[0343] For example, a configuration described as a single device (or processing unit) can be divided and configured into multiple devices (or processing units). Conversely, a configuration described above as multiple devices (or processing units) can be integrated into a single device (or processing unit). Furthermore, needless to say, configurations other than those described above can be added to the configuration of each device (or each processing unit). Moreover, when the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) can be included in the configuration of another device (or another processing unit).

[0344] Furthermore, for example, the above procedure can be executed by any device. In the above case, the device includes the necessary functions (function blocks, etc.) and can obtain the necessary information.

[0345] Additionally, for example, each step of a flowchart can be executed by a single device or by multiple devices in a distributed manner. Furthermore, when a single step includes multiple processing operations, these multiple processing operations can be executed by a single device or by multiple devices in a distributed manner. In other words, multiple processing operations included in one step can be executed as processing operations for multiple steps. Conversely, processing operations described as multiple steps can be executed together as a single step.

[0346] Additionally, for example, a program executed by a computer can be configured such that the processing operations describing the steps of the program are executed sequentially in the order described in this specification, executed in parallel, or executed individually when necessary (e.g., when called). That is, each step can be processed in a different order than described above, provided there is no contradiction. Furthermore, the processing operations describing the steps of the program can be executed in parallel with the processing operations of another program, or can be combined with the processing operations of another program.

[0347] Furthermore, for example, provided there is no contradiction, multiple technologies related to this technology can each be implemented independently in a single component. Of course, any of the multiple technologies can also be combined. For example, part or all of the technology described in any embodiment can be combined with part or all of the technology described in another embodiment. Moreover, any part or all of the technology described above can be combined with another technology not described above.

[0348] Note that this technology can be configured as described below.

[0349] (1) An image processing apparatus, comprising: An interlayer relative information generation unit is configured to generate interlayer relative information about the relative relationships between layers of 3D data, said 3D data including layer images of color components and layer images of alpha components; and The encoding unit is configured to generate a bitstream by encoding 3D data and to add interlayer relative information to the bitstream.

[0350] (2) The image processing apparatus according to (1), wherein, Interlayer relative information includes information about the relative number of layers for the alpha component relative to the number of layers for the color component.

[0351] (3) The image processing apparatus according to (1) or (2), wherein, Interlayer relative information includes information about the relative resolution of layers with alpha components relative to layers with color components.

[0352] (4) An image processing method, comprising: Generate interlayer relative information about the relative relationships between layers of 3D data, including layer images of color components and layer images of alpha components; and A bitstream is generated by encoding 3D data and adding interlayer relative information to the bitstream.

[0353] (5) A program for causing a computer to perform a process, said process comprising: Generate interlayer relative information about the relative relationships between layers of 3D data, wherein the 3D data layers include layer images of color components and layer images of alpha components; and A bitstream is generated by encoding 3D data and adding interlayer relative information to the bitstream.

[0354] (11) An image processing apparatus, comprising: A layer layout information generation unit is configured to generate layer layout information about the arrangement of layers in 3D data, the 3D data including layer images of color components and layer images of alpha components; and The encoding unit is configured to generate a bitstream by encoding 3D data and to add layer layout information to the bitstream.

[0355] (12) The image processing apparatus according to (11), wherein, Layer layout information includes information about the layout of some layers in the 3D data.

[0356] (13) The image processing apparatus according to (12), wherein, Information about the arrangement of some layers in a layer includes information indicating the position of some layers in the depth direction.

[0357] (14) The image processing apparatus according to (13), wherein, Information about the arrangement of some layers in a layer includes information indicating the number of some layers in the layer.

[0358] (15) The image processing apparatus described in any one of (11) to (14), wherein, Layer layout information includes information about the layout of layers of 3D data based on a non-uniformly spaced layout function.

[0359] (16) The image processing apparatus according to (15), wherein, Information regarding the arrangement based on the non-equal spacing function includes information indicating whether the layers are arranged based on the non-equal spacing function.

[0360] (17) The image processing apparatus according to (16), wherein, Information about layouts based on non-equal spacing layout functions includes information about the patterns used for the layout methods of specified layers.

[0361] (18) The image processing apparatus according to (15), wherein, Information regarding the arrangement based on the non-equal spacing function includes information indicating whether the layers are arranged in an inverse proportional mode.

[0362] (19) The image processing apparatus according to (18), wherein, Information regarding the arrangement based on the non-equal spacing function includes information indicating whether the layers are arranged in a non-linear mode.

[0363] (20) The image processing apparatus according to (19), wherein, Information regarding the arrangement based on the non-equidistant arrangement function includes information on whether the spacing between indicator layers is not explicitly indicated.

[0364] (21) An image processing method, comprising: Generate layer layout information about the arrangement of layers in 3D data, including layer images of color components and layer images of alpha components; and A bitstream is generated by encoding 3D data, and layer layout information is added to the bitstream.

[0365] (22) A program for causing a computer to perform a process, said process comprising: Generate layer layout information about the arrangement of layers in 3D data, including layer images of color components and layer images of alpha components; and A bitstream is generated by encoding 3D data, and layer layout information is added to the bitstream.

[0366] (31) An image processing apparatus, comprising: A layer shape selection information generation unit is configured to generate layer shape selection information regarding the selection of the shape of layers in 3D data, said 3D data including layer images of color components and layer images of alpha components; and The encoding unit is configured to generate a bitstream by encoding 3D data and to add layer shape selection information to the bitstream.

[0367] (32) The image processing apparatus described in (31), wherein, Layer shape selection information includes curvature specification information for specifying the curvature of the layer in a single direction, and information indicating whether the layer has a cylindrical shape.

[0368] (33) The image processing apparatus described in (31), wherein, The layer shape selection information includes information indicating whether the layer is non-planar, curvature specification information for specifying the curvature of the layer in a single direction, and information indicating whether the layer has a cylindrical shape.

[0369] (34) The image processing apparatus described in (31), wherein, Layer shape selection information includes curvature specification information for specifying the curvature of the layer in multiple directions.

[0370] (35) The image processing apparatus described in (31), wherein, Layer shape selection information includes information indicating whether the layer is non-planar and curvature specification information for specifying the curvature of the layer in multiple directions.

[0371] (36) The image processing apparatus described in (31), wherein, Layer shape selection information includes information indicating whether the layer is non-planar, information indicating whether the layer has a cylindrical shape, and curvature specification information for specifying the curvature of the layer in multiple directions.

[0372] (37) An image processing method, comprising: Generate layer shape selection information regarding the selection of layer shapes for 3D data, including layer images of color components and layer images of alpha components; and A bitstream is generated by encoding 3D data, and layer shape selection information is added to the bitstream.

[0373] (38) A program for causing a computer to perform a process, said process comprising: Generate layer shape selection information regarding the selection of layer shapes for 3D data, including layer images of color components and layer images of alpha components; and A bitstream is generated by encoding 3D data, and layer shape selection information is added to the bitstream.

[0374] (41) An image processing apparatus, comprising: The decoding unit is configured to decode the bitstream and generate 3D data including layer images of color components and alpha components. Interlayer relative information analysis unit, configured to analyze interlayer relative information about the relative relationships between layers of 3D data extracted from the bitstream; and The viewpoint image generation unit is configured to render 3D data based on interlayer relative information and generate viewpoint images viewed from any viewpoint.

[0375] (42) The image processing apparatus described in (41), wherein, Interlayer relative information includes information about the relative number of alpha component layers relative to color component layers.

[0376] (43) The image processing apparatus according to (41) or (42), wherein, Interlayer relative information includes information about the relative resolution of the alpha component layer relative to the color component layer.

[0377] (44) An image processing method, comprising: The bitstream is decoded, and 3D data including layer images of color components and alpha components is generated. Analyze interlayer relative information about the relative relationships between layers in 3D data extracted from the bitstream; and 3D data is rendered based on interlayer relative information, and viewpoint images are generated from any viewpoint.

[0378] (45) A program for causing a computer to perform a process, said process comprising: The bitstream is decoded, and 3D data including layer images of color components and alpha components is generated. Analyze interlayer relative information about the relative relationships between layers in 3D data extracted from the bitstream; and 3D data is rendered based on interlayer relative information, and viewpoint images are generated from any viewpoint.

[0379] (51) An image processing apparatus, comprising: The decoding unit is configured to decode the bitstream and generate 3D data including layer images of color components and alpha components. A layer layout information analysis unit, configured to analyze layer layout information about the layer arrangement of 3D data extracted from a bitstream; and The viewpoint image generation unit is configured to render 3D data based on layer layout information and generate viewpoint images that can be viewed from any viewpoint.

[0380] (52) The image processing apparatus described in (51), wherein, Layer layout information includes information about the layout of some layers in the 3D data.

[0381] (53) The image processing apparatus described in (52), wherein, Information about the arrangement of some layers in a layer includes information indicating the position of some layers in the depth direction.

[0382] (54) The image processing apparatus described in (53), wherein, Information about the arrangement of some layers in a layer includes information indicating the number of some layers in the layer.

[0383] (55) The image processing apparatus described in any one of (51) to (54), wherein, Layer layout information includes information about the layout of layers of 3D data based on a non-uniformly spaced layout function.

[0384] (56) The image processing apparatus described in (55), wherein, Information regarding the arrangement based on the non-equal spacing function includes information indicating whether the layers are arranged based on the non-equal spacing function.

[0385] (57) The image processing apparatus described in (56), wherein, Information about layouts based on non-equal spacing layout functions includes information about the patterns used for the layout methods of specified layers.

[0386] (58) The image processing apparatus described in (55), wherein, Information regarding the arrangement based on the non-equal spacing function includes information indicating whether the layers are arranged in an inverse proportional mode.

[0387] (59) The image processing apparatus described in (58), wherein, Information regarding the arrangement based on the non-equal spacing function includes information indicating whether the layers are arranged in a non-linear mode.

[0388] (60) The image processing apparatus according to (59), wherein, Information regarding the arrangement based on the non-equidistant arrangement function includes information on whether the spacing between indicator layers is not explicitly indicated.

[0389] (61) An image processing method, comprising: The bitstream is decoded, and 3D data including layer images of color components and alpha components is generated. Analyze the layer layout information of the 3D data extracted from the bitstream; and Render 3D data based on layer layout information and generate viewpoint images viewed from any viewpoint.

[0390] (62) A program for causing a computer to perform a process, said process comprising: The bitstream is decoded, and 3D data including layer images of color components and alpha components is generated. Analyze the layer layout information of the 3D data extracted from the bitstream; and Render 3D data based on layer layout information and generate viewpoint images viewed from any viewpoint.

[0391] (71) An image processing apparatus, comprising: The decoding unit is configured to decode the bitstream and generate 3D data including layer images of color components and alpha components. A layer shape selection information analysis unit is configured to analyze layer shape selection information extracted from the bitstream regarding the selection of layer shapes for 3D data; and The viewpoint image generation unit is configured to render 3D data based on layer shape selection information and generate viewpoint images that can be viewed from any viewpoint.

[0392] (72) The image processing apparatus described in (71), wherein, Layer shape selection information includes curvature specification information for specifying the curvature of the layer in a single direction, and information indicating whether the layer has a cylindrical shape.

[0393] (73) The image processing apparatus described in (71), wherein, The layer shape selection information includes information indicating whether the layer is non-planar, curvature specification information for specifying the curvature of the layer in a single direction, and information indicating whether the layer has a cylindrical shape.

[0394] (74) The image processing apparatus described in (71), wherein, Layer shape selection information includes curvature specification information for specifying the curvature of the layer in multiple directions.

[0395] (75) The image processing apparatus described in (71), wherein, Layer shape selection information includes information indicating whether the layer is non-planar and curvature specification information for specifying the curvature of the layer in multiple directions.

[0396] (76) The image processing apparatus described in (71), wherein, Layer shape selection information includes information indicating whether the layer is non-planar, information indicating whether the layer has a cylindrical shape, and curvature specification information for specifying the curvature of the layer in multiple directions.

[0397] (77) An image processing method, comprising: The bitstream is decoded, and 3D data including layer images of color components and alpha components is generated. Analyze layer shape selection information extracted from the bitstream regarding the selection of layer shapes for 3D data; and 3D data is rendered based on layer shape selection information, and viewpoint images are generated from any viewpoint.

[0398] (78) A program for causing a computer to perform a process, said process comprising: The bitstream is decoded, and 3D data including layer images of color components and alpha components is generated. Analyze layer shape selection information extracted from the bitstream regarding the selection of layer shapes for 3D data; and 3D data is rendered based on layer shape selection information, and viewpoint images are generated from any viewpoint. List of reference numerals

[0399] 500 Encoding device, 511 Layer image generation unit, 512 Layer image packing unit, 513 SEI generation unit, 514 Encoding unit, 515 Storage unit, 516 Transmission unit, 521 Inter-layer relative information generation unit, 522 Layer layout information generation unit, 523 Layer shape selection information generation unit, 600 Decoding device, 611 Receiving unit, 612 Storage unit, 613 Decoding unit, 614 SEI analysis unit, 615 Layer image unpacking unit, 616 Rendering unit, 617 Output unit, 621 Inter-layer relative information analysis unit, 622 Layer layout information analysis unit, 623 Layer shape selection information analysis unit, 900 Computer< / nex> < / nerf>

Claims

1. An image processing apparatus, comprising: Interlayer relative information generation unit, configured to generate interlayer relative information about the relative relationships between layers of 3D data, the 3D data including layer images of color components and layer images of alpha components; as well as An encoding unit is configured to generate a bitstream by encoding the 3D data and to add the interlayer relative information to the bitstream.

2. The image processing apparatus according to claim 1, wherein, The interlayer relative information includes information about the relative number of layers for the alpha component relative to the layers for the color component.

3. The image processing apparatus according to claim 1, wherein, The interlayer relative information includes information about the relative resolution of the alpha component layer relative to the color component layer.

4. The image processing apparatus according to claim 1, further comprising: A layer layout information generation unit is configured to generate layer layout information about the arrangement of layers in the 3D data. The encoding unit is configured to add the layer layout information to the bitstream.

5. The image processing apparatus according to claim 4, wherein, The layer layout information includes information about the arrangement of some layers in the 3D data.

6. The image processing apparatus according to claim 5, wherein, Information regarding the arrangement of the layers includes information indicating the position of the layers in the depth direction.

7. The image processing apparatus according to claim 6, wherein, Information regarding the arrangement of the layers includes information indicating the number of the layers.

8. The image processing apparatus according to claim 4, wherein, The layer arrangement information includes information about the arrangement of the layers of the 3D data based on a non-equidistant arrangement function.

9. The image processing apparatus according to claim 8, wherein, Information regarding the arrangement based on the non-equal spacing function includes: information indicating whether the layer is arranged based on the non-equal spacing function.

10. The image processing apparatus according to claim 9, wherein, Information regarding the arrangement based on the non-equal spacing function includes: information about the pattern used to specify the arrangement method of the layer.

11. The image processing apparatus according to claim 8, wherein, Information regarding the arrangement based on the non-equal spacing function includes: information indicating whether the spacing between the layers is not explicitly indicated, information indicating whether the layers are arranged in a non-linear mode, and information indicating whether the layers are arranged in an inverse mode.

12. The image processing apparatus according to claim 1, further comprising: A layer shape selection information generation unit is configured to generate layer shape selection information regarding the selection of the shape of layers in the 3D data. The encoding unit is configured to add the layer shape selection information to the bitstream.

13. The image processing apparatus according to claim 12, wherein, The layer shape selection information includes curvature specification information for specifying the curvature of the layer in a single direction and information indicating whether the layer has a cylindrical shape.

14. The image processing apparatus according to claim 12, wherein, The layer shape selection information includes: information indicating whether the layer is non-planar, curvature specification information for specifying the curvature of the layer in a single direction, and information indicating whether the layer has a cylindrical shape.

15. The image processing apparatus according to claim 12, wherein, The layer shape selection information includes curvature specification information for specifying the curvature of the layer in multiple directions.

16. The image processing apparatus according to claim 12, wherein, The layer shape selection information includes information indicating whether the layer is non-planar and curvature specification information for specifying the curvature of the layer in multiple directions.

17. The image processing apparatus according to claim 12, wherein, The layer shape selection information includes: information indicating whether the layer is non-planar, information indicating whether the layer has a cylindrical shape, and curvature specification information for specifying the curvature of the layer in multiple directions.

18. An image processing method, comprising: Generate interlayer relative information about the relative relationships between layers of 3D data, which includes layer images of color components and layer images of alpha components; as well as A bitstream is generated by encoding the 3D data, and the interlayer relative information is added to the bitstream.

19. An image processing apparatus, comprising: The decoding unit is configured to decode the bitstream and generate 3D data including layer images of color components and layer images of alpha components. Interlayer relative information analysis unit, configured to: analyze interlayer relative information about the relative relationships between layers of the 3D data extracted from the bitstream; as well as A viewpoint image generation unit is configured to render the 3D data based on the interlayer relative information and generate a viewpoint image viewed from any viewpoint.

20. An image processing method, comprising: The bitstream is decoded, and 3D data including layer images of color components and alpha components is generated. Analyze the interlayer relative information extracted from the bitstream regarding the relative relationships between layers of the 3D data; as well as The 3D data is rendered based on the interlayer relative information, and a viewpoint image is generated from any viewpoint.

Images