Information processing device and method

Abstract

The present disclosure pertains to an information processing device and a method that make it possible to suppress an increase in transmission capacity. An atlas image of an alpha component is generated by atlas transformation of an alpha image of 3D data, and the atlas image of the alpha component is encoded. An atlas image of an RGB component is generated by atlas transformation of an RGB image of the 3D data, and the atlas image of the RGB component is encoded. Further, the encoded data is decoded to generate an atlas image of the alpha component, and the atlas image of the alpha component is subjected to inverse atlas transformation to generate an alpha image of the 3D data. The encoded data is decoded to generate an atlas image of the RGB component, and the atlas image of the RGB component is subjected to inverse atlas transformation to generate an RGB image of the 3D data. The present disclosure can be applied to, for example, an information processing device, an electronic apparatus, an information processing method, or a program.

Description

Information processing device and method

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that are capable of suppressing an increase in transmission capacity.

[0002] Conventionally, one method for representing the three-dimensional shape of an object is Representing Scenes as Neural Radiance Fields for View Synthesis (NeRF), which generates a radiance field corresponding to the space containing the object and approximates the radiance field using a neural network (see, for example, Non-Patent Documents 1 to 3). NeRF represents the spatial information of a 3D object using a Multi-Layer Perceptron (MLP).

[0003] There is also a method called MPI (Multi Plane Image) that divides a three-dimensional space into multiple layers (2D images) to represent it. To speed up the rendering process of NeRF, a method called NeX was devised that converts MLP to this MPI for handling (see, for example, Non-Patent Document 4). MPI is a group of images in which multiple 2D images are arranged in layers, and 3D data can be handled as a 2D image.

[0004] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, "NeRF: Representing scenes as neural radiance fields for view synthesis", ECCV 2020, 2020 / 3 / 19Thomas Muller, Alex Evans, Christoph Schied, Alexander Keller, "Instant neural graphics primitives with a multiresolution hash encoding", arXiv preprint arXiv:2201.05989, 2022 / 1 / 16Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, Daniel Duckworth, "NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections", https: / / arxiv.org / abs / 2008.02268, 2020 / 8 / 5Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, "NeX: Real-time View Synthesis with Neural Basis Expansion", arXiv:2103.05606v2 [cs.CV] 12 Apr 2021

[0005] However, 3D data that uses MPI, such as NeX, is large in data volume, and there is a risk that the transmission capacity will increase when transmitting it.

[0006] The present disclosure has been made in view of such circumstances, and makes it possible to suppress an increase in transmission capacity.

[0007] An information processing device according to one aspect of the present technology is an information processing device including: a first atlas conversion unit that performs atlas conversion on an alpha image of 3D data and generates an atlas image of an alpha component; a first encoding unit that encodes the atlas image of the alpha component; a second atlas conversion unit that performs atlas conversion on an RGB image of the 3D data and generates an atlas image of an RGB component; and a second encoding unit that encodes the atlas image of the RGB component.

[0008] An information processing method according to one aspect of the present technology is an information processing method that performs atlas transformation on an alpha image of 3D data, generates an atlas image of an alpha component, encodes the atlas image of the alpha component, performs atlas transformation on an RGB image of the 3D data, generates an atlas image of an RGB component, and encodes the atlas image of the RGB component.

[0009] According to another aspect of the present technology, an information processing device includes a first decoding unit that decodes encoded data and generates an atlas image of an alpha component, a first atlas inverse transformation unit that performs an atlas inverse transformation on the atlas image of the alpha component and generates an alpha image of 3D data, a second decoding unit that decodes encoded data and generates an atlas image of RGB components, and a second atlas inverse transformation unit that performs an atlas inverse transformation on the atlas image of the RGB components and generates an RGB image of the 3D data.

[0010] An information processing method according to another aspect of the present technology is an information processing method that decodes encoded data, generates an atlas image of an alpha component, performs an inverse atlas transform on the atlas image of the alpha component, generates an alpha image of 3D data, decodes the encoded data, generates an atlas image of RGB components, performs an inverse atlas transform on the atlas image of the RGB components, and generates an RGB image of the 3D data.

[0011] In an information processing device and method according to one aspect of the present technology, an alpha image of 3D data is atlas-transformed to generate an atlas image of an alpha component, and the atlas image of the alpha component is encoded; an RGB image of the 3D data is atlas-transformed to generate an atlas image of an RGB component, and the atlas image of the RGB component is encoded.

[0012] In an information processing device and method according to another aspect of the present technology, encoded data is decoded to generate an atlas image of an alpha component, the atlas image of the alpha component is subjected to an inverse atlas transform to generate an alpha image of 3D data, the encoded data is decoded to generate an atlas image of an RGB component, and the atlas image of the RGB component is subjected to an inverse atlas transform to generate an RGB image of the 3D data.

[0013] 1 is a diagram illustrating 3D data using MPI. FIG. 1 is a diagram illustrating 3D data using MPI. FIG. 1 is a diagram illustrating NeX. FIG. 1 is a diagram illustrating NeX. FIG. 1 is a diagram illustrating NeX. FIG. 1 is a diagram illustrating an example of a method for transmitting 3D data. FIG. 1 is a diagram illustrating use of atlas transformation. FIG. 1 is a diagram illustrating an example of atlas transformation of NeX data. FIG. 1 is a diagram illustrating an example of how a mask image is generated. FIG. 1 is a diagram illustrating an example of how atlas transformation of an RGB_base image using a mask image. FIG. 1 is a diagram illustrating an example of how atlas transformation of an RGB_view image using a mask image. FIG. 1 is a diagram illustrating an example of the configuration of a bit stream. FIG. 1 is a diagram illustrating an example of atlas transformation of NeX data. FIG. 1 is a diagram illustrating an example of the configuration of a bit stream. FIG. 1 is a diagram illustrating an example of how atlas transformation information is generated. FIG. 1 is a diagram illustrating an example of the configuration of a bit stream. A system diagram showing an example of the main configuration of an information processing system. A flowchart illustrating an example of the flow of a transmission process. A flowchart illustrating an example of the flow of a playback process. A block diagram illustrating an example of the main configuration of an encoding device. A flowchart illustrating an example of the flow of an encoding process. A block diagram illustrating an example of the main configuration of a decoding device. A flowchart illustrating an example of the flow of a decoding process. A block diagram illustrating an example of the main configuration of an encoding device. A flowchart illustrating an example of the flow of an encoding process. A block diagram illustrating an example of the main configuration of a decoding device. A flowchart illustrating an example of the flow of a decoding process. It is a block diagram showing an example of the main configuration of an encoding device. It is a block diagram showing an example of the main configuration of a decoding device. It is a flowchart explaining an example of the flow of decoding processing. It is a block diagram showing an example of the main configuration of a computer.

[0014] The following describes modes for carrying out the present disclosure (hereinafter referred to as embodiments). The description will be given in the following order: 1. Literature supporting technical content and technical terminology 2. Transmission capacity of 3D data using MPI 3. Encoding and decoding using atlas transformation 4. First embodiment (information processing system) 5. Supplementary notes

[0015] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following non-patent documents.

[0016] Non-patent document 1: (described above) Non-patent document 2: (described above) Non-patent document 3: (described above) Non-patent document 4: (described above)

[0017] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0018] <2. Transmission capacity of 3D data using MPI> <NeRF> Conventionally, as a method for representing the three-dimensional shape of an object, there has been NeRF (Representing Scenes as Neural Radiance Fields for View Synthesis), which generates a radiance field corresponding to a space including the object and approximates the radiance field using a neural network, as described in Non-Patent Documents 1 to 3.

[0019] A radiation field represents a scene in a space containing an object by using the color of each direction at each position in that space and the opacity of each position in that space. Here, opacity is an indicator of the presence of some kind of object, and can also be thought of as density. In other words, the shape of a three-dimensional space can be represented by obtaining a radiation field in which the density of coordinates where an object exists is high. NeRF approximates such a radiation field using a neural network. In other words, NeRF is a technology that generates a radiation field corresponding to a 3D object, approximates that radiation field using a neural network, and uses that neural network to perform rendering.

[0020] NeRF captures images of an object in 3D space from different positions, calculates the camera pose of the captured images, and trains a neural network using the captured images and camera pose to generate a neural network (MLP (Multi Layer Perceptron)) that approximates the radiation field representing the object's 3D shape. During inference, pose information (also called viewpoint information) from a desired new viewpoint is input to this neural network to obtain a rendered image from that new viewpoint.

[0021] <NeX> There is also a method called MPI (Multi Plane Image) that divides a three-dimensional space into multiple layers (2D images) to represent it. In Non-Patent Document 4, a method called NeX was proposed that converts MLP into this MPI and handles it in order to speed up the rendering process of NeRF. MPI is a group of images in which multiple 2D images are arranged in layers, and 3D data can be handled as a 2D image.

[0022] In MPI, multiple layers (2D planes) are formed in a three-dimensional space, such as layers 11 to 14 in Figure 1, and objects are represented on each layer. MPI is composed of multiple RGBA images. These RGBA images are generated by learning multi-viewpoint images using a deep neural network (DNN). A rendered image at a viewpoint position shows the composite (overlapping) results of objects in each layer as seen from that viewpoint. For example, as shown in Figure 1, object 21 exists in layer 11, object 22 exists in layer 12, object 23 exists in layer 13, and object 24 exists in layer 14. In the rendered image, these objects are overlaid according to their positional relationship as seen from the viewpoint position. Rendered image 41 in Figure 2 shows an example of a rendered image at viewpoint 31 in Figure 1, and rendered image 42 shows a rendered image at viewpoint 32 in Figure 1. The way objects 21 to 24 overlap changes depending on the viewpoint position, thereby expressing the three-dimensional effect of the three-dimensional space (the three-dimensional shape of the objects).

[0023] In NeX, NeRF's MLP is converted into this type of MPI (also referred to as NeX MPI) for handling. In other words, three-dimensional space is represented by multiple layers. An example of the layer structure of NeX is shown in Figure 3. In the case of NeX MPI, multiple main layers 51 are formed, such as main layer 51-1, main layer 51-2, ..., and multiple sublayers 52 are formed between the main layers 51, such as sublayers 52-1 to 52-11, sublayer 52-21, .... In other words, in the group of layers representing three-dimensional space, some layers at a predetermined interval are formed as main layers 51, and the remaining layers between them are formed as sublayers 52.

[0024] The number of layers in this layer group, the number of main layers, and the number of sub-layers between the main layers are all arbitrary. For example, a three-dimensional space (one scene) at a certain time may be represented by 192 layers, with one in 12 of those layers (i.e., a total of 16) being the main layer and the remaining 176 being sub-layers. In this case, one of the 12 consecutive layers in the layer group is the main layer and the remaining 11 are sub-layers. Also, for example, there may be 32 main layers. In this case, the number of layers in the layer group representing one scene is 384. Of course, this is not limited to these examples.

[0025] In the case of NeX, the data is composed of an alpha image 71, an RGB_base image 72, an RGB_view image 73, and a base image 74, as shown in FIG. 4. The alpha image 71 is distribution information of the alpha component indicating the presence of an object in a single layer among the layers representing a three-dimensional space. In other words, the alpha image 71 indicates where the object is located in that layer. This alpha image 71 is generated for each sublayer. That is, this alpha image 71 is generated for each of all layers (main layer 51 and sublayers 52 in FIG. 3) among the layers representing the three-dimensional space.

[0026] The RGB_base image 72 and the RGB_view image 73 are information indicating the distribution of RGB components that indicate color information of an object in multiple consecutive layers in a layer group. The RGB_base image and the RGB_view image are also collectively referred to as RGB images.

[0027] The RGB_base image 72 is color distribution information that does not depend on the angle of the viewpoint. The RGB_base image 72 is composed of a single image. In other words, the RGB_base image 72 is color distribution information with a strong color component when viewed from the front of the image.

[0028] The RGB_view image 73 is color distribution information that depends on the angle of the viewpoint. The RGB_view image 73 is composed of multiple images. The multiple images are color distribution information with strong color components when viewed from different viewpoints. For example, the RGB_view image 73 is composed of four images: an image k0 with strong color components when viewed from above, an image with strong color components when viewed from below, an image with strong color components when viewed from the left, and an image with strong color components when viewed from the right.

[0029] The RGB_base image 72 and the RGB_view image 73 are generated for each main layer, i.e., for each of a plurality of consecutive layers in the layer group. For example, in the case of Fig. 3, one RGB_base image 72 and one RGB_view image 73 are generated for each of the main layer 51-1 and the sublayers 52-1 to 52-11. In other words, the RGB_base image 72 represents color distribution information for 12 consecutive layers, and one image is generated for each of the 12 layers. Similarly, the RGB_view images 73 represent color distribution information for 12 consecutive layers, and four images are generated for each of the 12 layers.

[0030] The base images 74 are composed of the same number of images as the RGB_view images 73. For example, if four RGB_view images 73 are formed as described above, four base images 74 are also formed. Each base image 74 corresponds to a different RGB_view image 73, and indicates the blend ratio of the corresponding RGB_view image 73 for each viewpoint direction. In other words, the pixel position of the base image 74 is determined according to the viewpoint direction, and the RGB_view image 73 corresponding to that base image 74 is blended at the blend ratio indicated by the pixel value. For example, when the viewpoint is frontal, as shown in FIG. 5 , each RGB_view image 73 (k1 to k4) is multiplied by the blend ratio indicated by the central pixel value of the corresponding base image 74 (H1 to H4), and the multiplication result is added to the RGB_base image 72 (k0). Also, when the viewpoint is at the top left, as shown in FIG. 6, each RGB_view image 73 (k1 to k4) is multiplied by the blend ratio indicated as the pixel value at the top left of the corresponding base image 74 (H1 to H4), and the multiplication results are added to the RGB_base image 72 (k0).

[0031] NeX MPI can improve expressiveness by changing the RGB parts depending on the viewpoint in this way. By treating the alpha image as a sublayer and the RGB image as a main layer, it is possible to suppress the increase in data volume while improving expressiveness and achieve more efficient learning.

[0032] <MPI Transmission> One MPI transmission method is the MPEG Immersive Video (MIV) coding standard, which is being standardized by the Moving Picture Experts Group (MPEG). MPEG-I Part 12: Immersive Video (MIV) is a framework for encoding and transmitting multi-viewpoint video consisting of multiple textures and depths after preprocessing to reduce data, such as viewpoint reduction and pixel pruning, and it standardizes the metadata required for decoding and synthesis associated with preprocessing.

[0033] There is also MPEG-I Part 5-2nd edition (V3C (Visual Volumetric Video-based Coding)), which describes the details of MIV atlas processing and video processing. Immersive video has several degrees of freedom in terms of viewpoint, and there are several expression methods to achieve this.

[0034] As mentioned above, MPI is structured like a three-dimensional space sliced ​​into multiple layers (images). Therefore, each layer is sparse data. Therefore, the amount of data can be reduced by reducing the number of pixels (atlas) before transmission. MIV / V3C specifies information about the atlas (such as the extraction position).

[0035] In atlas transformation, areas where transparency (alpha component) is zero are deleted. The remaining areas (non-zero areas) are cut into rectangles and placed in the atlas image. This atlased data is transmitted as atlas transformation information indicating the details of the atlas transformation, and as an atlas image indicating the transformation result of the atlas transformation (i.e., the non-zero areas are placed).

[0036] In the case of MPI, the basic information of the atlas conversion information is stored in V3C_VPS. The atlas conversion information data is stored in V3C_AD. The atlas image (alpha component atlas image or RGB component atlas image) is stored in V3C_AVD.

[0037] V3C_VPS can transmit up to 63 pieces of atlas transformation information (basic information), including the identification (ID), width (horizontal) size of the atlas image, and height (vertical) size of the atlas image, etc.

[0038] The atlas image is linked to the alpha image or RGB image using identification information (vuh_atlas_id) in v3c_unit_header().

[0039] In addition, the method for transmitting atlases is standardized in MPEG-I Part 5. Non-zero regions can be rotated and placed in the atlas image.

[0040] <Data Volume of NeX MPI> The data volume of NeX MPI was large. As an example, let's assume 60 FPS 3D video data, alpha and RGB image resolution is 4K, the main layers that make up the scene are 32 layers with 12 layers each, and there are four basis images (Basis). Then, let's say we transmit the data in NeX MPI (4K32L) and play it back instantly (in real time) at the destination. In this case, the required transmission capacity is 524 GB / s. As another example, let's assume 60 FPS 3D video data, alpha and RGB image resolution is HD, the main layers that make up the scene are 16 layers with 12 layers each, and there are four basis images (Basis). Then, let's transmit the data in NeX MPI (HD16L) and play it back instantly (in real time) at the destination. In this case, the required transmission capacity is 83 GB / s. In this specification, transmission capacity refers to the amount of data per second that can be transmitted over a transmission path (which may include a communication medium, a data bus, an information processing unit, etc.). It is difficult to secure such a transmission capacity, and there is a risk that the playback process will fail. Therefore, there is a need to reduce the transmission capacity of NeX MPI.

[0041] This is not limited to NeX, but applies to any 3D data that uses MPI, such as NeX. 3D data that uses MPI, such as NeX, has a large data volume, and there has been a demand for reducing the transmission capacity.

[0042] <3. Encoding and decoding using atlas transformation> <Method 1> Therefore, as shown in the top row of the table in Fig. 7 , the alpha image of the 3D data and the RGB image that is smaller than that of the alpha image are each subjected to atlas transformation, encoded, and transmitted (Method 1).

[0043] For example, a first information processing device includes a first atlas conversion unit that performs atlas conversion on an alpha image of 3D data and generates an atlas image of an alpha component, a first encoding unit that encodes the atlas image of the alpha component, a second atlas conversion unit that performs atlas conversion on an RGB image of the 3D data and generates an atlas image of an RGB component, and a second encoding unit that encodes the atlas image of the RGB component.

[0044] In addition, in the first information processing method, an alpha image of the 3D data is atlas-transformed, an atlas image of the alpha component is generated, and the atlas image of the alpha component is encoded; an RGB image of the 3D data is atlas-transformed, an atlas image of the RGB component is generated, and the atlas image of the RGB component is encoded.

[0045] The first information processing device also includes a first decoding unit that decodes encoded data and generates an atlas image of an alpha component, a first atlas inverse transformation unit that performs an atlas inverse transformation on the atlas image of the alpha component and generates an alpha image of 3D data, a second decoding unit that decodes encoded data and generates an atlas image of RGB components, and a second atlas inverse transformation unit that performs an atlas inverse transformation on the atlas image of the RGB components and generates an RGB image of the 3D data.

[0046] In addition, in a second information processing method, coded data is decoded to generate an atlas image of an alpha component, the atlas image of the alpha component is subjected to an inverse atlas transform, and an alpha image of 3D data is generated; and coded data is decoded to generate an atlas image of an RGB component, the atlas image of the RGB component is subjected to an inverse atlas transform, and an RGB image of the 3D data is generated.

[0047] An alpha image is distribution information of an alpha component indicating the presence of an object in a single layer among a group of layers representing a three-dimensional space. An RGB image is information indicating the distribution of RGB components indicating the color information of the object in multiple consecutive layers among the group of layers. 3D data has an alpha image for each single layer of a three-dimensional space and an RGB image for each consecutive layer. Furthermore, atlas transformation is a process of arranging non-zero areas included in a single alpha image or non-zero areas included in an RGB image corresponding to non-zero areas included in one or more alpha images into an atlas image, and inverse atlas transformation is the inverse process of the atlas transformation.

[0048] For example, as shown in FIG. 8, non-zero regions are extracted from each layer of 3D data 111 (alpha and RGB images) consisting of multiple layers and placed in an atlas image 112. In this specification, this process is referred to as atlas transformation. The inverse process is referred to as inverse atlas transformation. Similar to the above-mentioned MPI, the alpha and RGB images of each layer in NeX MPI are generally sparse. Therefore, multiple non-zero regions can be placed in a single atlas image.

[0049] The first atlas conversion unit of the first information processing device performs such atlas conversion on the alpha image to generate an atlas image of the alpha component (atlas image (alpha)) as shown in FIG. 9 . Then, the first encoding unit of the first information processing device encodes the atlas image (alpha) using a 2D encoding method to generate encoded data. Similarly, the second atlas conversion unit of the first information processing device performs such atlas conversion on the RGB image to generate an atlas image of the RGB component (atlas image (RGB)) as shown in FIG. 9 . Then, the second encoding unit of the first information processing device encodes the atlas image (RGB) using a 2D encoding method to generate encoded data. These encoded data are transmitted to the second information processing device.

[0050] Then, a first decoding unit of the second information processing device decodes the coded data of the atlas image (alpha) to generate (restore) the atlas image (alpha). Then, a first atlas inverse transform unit of the second information processing device performs inverse atlas transform on the atlas image (alpha) to generate (restore) the alpha image. Similarly, a second decoding unit of the second information processing device decodes the coded data of the atlas image (RGB) to generate (restore) the atlas image (RGB). Then, a second atlas inverse transform unit of the second information processing device performs inverse atlas transform on the atlas image (RGB) to generate (restore) the RGB image.

[0051] By doing this, the transmission capacity required for data transmission of NeX MPI (4K32L) can be reduced from 524 GB / s to 7 GB / s (approximately 1 / 75th). Furthermore, the transmission capacity required for data transmission of NeX MPI (HD16L) can be reduced from 83 GB / s to 1.1 GB / s (approximately 1 / 75th). In other words, it is possible to suppress the increase in transmission capacity required for transmitting 3D data that uses MPI, such as NeX.

[0052] Although this specification uses NeX MPI as an example, this technology (Method 1) can be applied to any 3D data that uses MPI. In other words, the application of this technology is not limited to NeX MPI. Furthermore, the encoding method to which this technology (Method 1) is applied may be any encoding method that involves atlas transformation, such as V3C or another encoding method.

[0053] <Method 1-1> When this method 1 is applied, as shown in the second row from the top of the table in Figure 7, intermediate data (mask) may be generated by combining all sub-layers of the alpha image and the main layer (all layers of the "multiple consecutive layers" corresponding to one RGB image), and this mask may be used to perform atlas conversion of one RGB image corresponding to the "multiple consecutive layers" (Method 1-1).

[0054] For example, the above-mentioned first information processing device may further include a mask image generation unit that generates a mask image by combining non-zero areas of alpha images of multiple consecutive layers, and the second atlas conversion unit may perform atlas conversion on the RGB image using the mask image.

[0055] Since the alpha component is information indicating the presence of an object, the alpha image can be atlas-converted by extracting its non-zero regions. The RGB component portions corresponding to the object correspond to the non-zero regions of the alpha component. In other words, the second atlas conversion unit basically extracts the same regions as the alpha image from the RGB image and performs atlas conversion. However, as shown in FIG. 9 , an alpha image is formed for each sublayer (each single layer), and an RGB image is formed for each main layer (each consecutive layer). In other words, one RGB image contains color distribution information for multiple consecutive layers (e.g., 12 layers). In other words, one RGB image corresponds to multiple consecutive alpha images (e.g., 12 layers).

[0056] Therefore, simply extracting the same area from an RGB image as the non-zero area of ​​one alpha image and performing atlas conversion on it could result in a loss of color distribution information for the other layers in multiple consecutive layers (for example, if an RGB image is formed every 12 layers, information for the remaining 11 layers). In other words, in the case of 3D data that uses MPI, such as NeX MPI, and has fewer RGB images than alpha images, atlas conversion could result in a loss of RGB image information, potentially reducing the quality of the 3D data (the quality of the rendered image).

[0057] Therefore, the mask image generation unit of the first information processing device generates intermediate data (mask image) by synthesizing all layers of "successive layers" corresponding to one RGB image, as shown in Fig. 10. For example, if an RGB image is formed for every 12 layers, alpha images are synthesized (superimposed) for every 12 layers, and a mask image is generated in which the alpha images for the 12 layers are superimposed. This generates the same number of mask images as the number of layers of the RGB image. For example, if a three-dimensional space is represented by 192 layers, 16 mask images are generated.

[0058] Then, as shown in Figures 11 and 12, this mask image is used to perform atlas transformation on the RGB image to generate an atlas image (RGB). Each mask image is the result of superimposing 12 alpha images, and therefore contains 12 layers of non-zero regions (12 layers of non-zero regions are superimposed). Therefore, by applying this mask image, color distribution information for 12 layers of objects can be extracted from the RGB image, thereby suppressing the loss of information described above. In other words, this suppresses the degradation of 3D data quality (image quality of the rendered image) caused by atlas transformation.

[0059] <Method 1-1-1> When this method 1-1 is applied, the atlas transformation information of the RGB_base image and the RGB_view image may be made common, as shown in the third row from the top of the table in FIG. 7 (Method 1-1-1).

[0060] For example, in the first information processing device described above, the RGB image may include an RGB_base image indicating the distribution of the RGB_base component among the RGB components, and an RGB_view image indicating the distribution of the RGB_view component among the RGB components. Furthermore, the second atlas conversion unit may perform atlas conversion on the RGB_base image and the RGB_view image to generate respective atlas images, and generate atlas conversion information common to the RGB_base component and the RGB_view component. Note that the atlas conversion information may be information indicating the content of the atlas conversion.

[0061] For example, in the second information processing device described above, the RGB image may include an RGB_base image indicating a distribution of the RGB_base component among the RGB components, and an RGB_view image indicating a distribution of the RGB_view component among the RGB components. Furthermore, the second decoding unit may decode the encoded data to generate an atlas image of the RGB_base component and an atlas image of the RGB_view component. Furthermore, the second atlas inverse transform unit may perform inverse atlas transform on the atlas image of the RGB_base component and the atlas image of the RGB_view component using atlas transform information common to the RGB_base component and the RGB_view component. Furthermore, the atlas transform information may be information indicating the content of the atlas transform.

[0062] As shown in FIG. 8 , the second atlas conversion unit of the first information processing apparatus generates atlas conversion information 113 indicating the details of the atlas conversion together with the atlas image 112. This "contents of the atlas conversion" may include, for example, information such as which region (e.g., regions 121 to 124 in FIG. 8 ) extracted from which layer and where is to be placed in the atlas image. Such atlas conversion information 113 is used for atlas inversion by the second atlas inversion unit of the second information processing apparatus. Atlas inversion is the inverse process of atlas conversion. In other words, atlas inversion is a process of extracting each region (e.g., regions 121 to 124 in FIG. 8 ) placed in the atlas image by the atlas conversion and generating (restoring) the original alpha image or RGB image. Therefore, the second atlas inversion unit can correctly generate (restore) the alpha image or RGB image by performing such atlas inversion based on the atlas conversion information indicating the details of the atlas conversion.

[0063] When an RGB image includes an RGB_base image and an RGB_view image, as in the example of FIG. 9 , the atlas conversion unit performs atlas conversion on each of the RGB_base image and the RGB_view image. In these atlas conversions, the atlas conversion unit may extract identical regions from the RGB_base image and the RGB_view image and place them at the same position in the atlas image. In this case, the atlas conversion unit may generate atlas conversion information (atlas conversion information (RGB)) common to the RGB_base component and the RGB_view component, as shown in FIGS. 11 and 12 . This reduces redundancy between the atlas conversion information for the RGB_base component and the atlas conversion information for the RGB_view component when both are generated. This reduces the increase in data volume and the increase in the transmission capacity required for data transmission.

[0064] In this case, for example, in a V3C bitstream, the atlas transformation information (RGB) may be stored in V3C_AD(1) ​​that is common to the RGB_base component and the RGB_view component, as shown in Fig. 13. Then, the atlas image (RGB_base) and the atlas image (RGB_view) may be stored in V3C_AVD linked to that V3C_AD(1).

[0065] <Method 1-2> When the above-mentioned method 1 is applied, a mask may be generated by converting the alpha image to the same size as the RGB image, as shown in the fourth row from the top of the table in FIG. 7 (method 1-2).

[0066] The size (resolution) of the alpha image and the RGB image may differ. In the example of Fig. 14, the size (resolution) of the alpha image and the RGB_view image is what is called 1K, and the size (resolution) of the RGB_base image is what is called 2K. When the size (resolution) of the alpha image and the RGB image differs in this way, the size (resolution) of the mask image generated from the alpha image may differ from the size (resolution) of the RGB image, as described above, and may not be applicable.

[0067] Therefore, by converting the alpha image to the same size as the RGB image, the atlas conversion can be performed correctly, thereby suppressing the increase in the transmission capacity required for data transmission.

[0068] <Method 1-2-1> When this method 1-2 is applied, as shown in the fifth row from the top of the table in FIG. 7, an RGB_base image and an RGB_view image having different sizes may be atlas-transformed independently of each other (Method 1-2-1).

[0069] For example, in the first information processing device described above, the RGB image may include an RGB_base image indicating a distribution of RGB_base components among the RGB components, and an RGB_view image indicating a distribution of RGB_view components among the RGB components, and the second atlas conversion unit may perform atlas conversion independently on the RGB_base image and the RGB_view image, which have different sizes.

[0070] For example, in the second information processing device described above, the RGB image may include an RGB_base image indicating the distribution of the RGB_base component among the RGB components, and an RGB_view image indicating the distribution of the RGB_view component among the RGB components. The RGB_base image and the RGB_view image may have different sizes. The second decoding unit may decode coded data of the atlas image of the RGB_base component and coded data of the atlas image of the RGB_view component, respectively. The second atlas inverse transform unit may perform inverse atlas transform on the atlas image of the RGB_base component and the atlas image of the RGB_view component independently of each other.

[0071] 14, the sizes (resolutions) of the RGB_base image and the RGB_view image may differ. In such cases, it may be difficult to generate common atlas transformation information for the atlas transformation. It may also be difficult to correctly perform an inverse atlas transformation using the common atlas transformation information.

[0072] Therefore, as described above, the RGB_base image and the RGB_view image may be atlas-transformed independently of each other. Also, the atlas image of the RGB_base component and the atlas image of the RGB_view component may be inverse-atlas-transformed independently of each other.

[0073] By doing so, the atlas transform and inverse atlas transform of the RGB_base components and RGB_view components can be performed correctly.

[0074] <Method 1-2-1-1> When this method 1-2-1 is applied, atlas transformation information for the RGB_base image and atlas transformation information for the RGB_view image may be generated as shown in the sixth row from the top of the table in Figure 7 (Method 1-2-1-1).

[0075] For example, in the first information processing device described above, the second atlas conversion unit may generate atlas conversion information for RGB_base components by atlas conversion of an RGB_base image. The second atlas conversion unit may also generate atlas conversion information for RGB_view components by atlas conversion of an RGB_view image. The atlas conversion information may be information indicating the content of the atlas conversion.

[0076] For example, in the second information processing device described above, the second atlas inverse transform unit may perform inverse atlas transform on an atlas image of RGB_base components using atlas transform information on RGB_base components. The second atlas transform unit may also perform inverse atlas transform on an atlas image of RGB_view components using atlas transform information on RGB_view components. The atlas transform information may be information indicating the content of the atlas transform.

[0077] For example, as shown in Fig. 14, atlas transformation may be performed to generate atlas transformation information (RGB_base) and atlas transformation information (RGB_view). In this way, atlas transformation of the RGB_base component and the RGB_view component can be performed independently. Furthermore, atlas inverse transformation may be performed using the atlas transformation information (RGB_base) and the atlas transformation information (RGB_view). Atlas inverse transformation of the RGB_base component and the RGB_view component can be performed independently.

[0078] In this case, as shown in Fig. 15, for example, in a V3C bitstream, the atlas transformation information (RGB_base) may be stored in V3C_AD(1) ​​for the RGB_base component, and the atlas transformation information (RGB_view) may be stored in V3C_AD(2) for the RGB_view component (these may be stored in different V3C_ADs).The atlas image (RGB_base) may then be stored in a V3C_AVD associated with V3C_AD(1).The atlas image (RGB_view) may also be stored in a V3C_AVD associated with V3C_AD(2).

[0079] <Method 1-2-1-2> Furthermore, when the above-described method 1-2-1 is applied, intermediate data (masks) may be generated for each of the RGB_base image and the RGB_view image, as shown in the seventh row from the top of the table in FIG. 7 (Method 1-2-1-2).

[0080] For example, the above-described first information processing device may further include a mask image generation unit that generates, for each of an RGB_base image and an RGB_view image, a mask image that combines non-zero regions of alpha images of multiple consecutive layers. In the first information processing device, a second atlas conversion unit may perform atlas conversion on the RGB_base image using a mask image for the RGB_base image. Furthermore, the second atlas conversion unit may perform atlas conversion on the RGB_view image using a mask image for the RGB_view image.

[0081] When the RGB_base image and the RGB_view image have different sizes, as in the example of FIG. 14 , the size of at least one of them differs from the size of the alpha image. Therefore, there is a risk that a mask image generated from the alpha image cannot be applied at its original size. Therefore, intermediate data (masks) may be generated for each of the RGB_base image and the RGB_view image. By generating mask images separately in this way, mask images that match the sizes of both the RGB_base image and the RGB_view image can be generated. Therefore, atlas conversion can be performed correctly.

[0082] <Method 1-2-1-3> Furthermore, when the above-described method 1-2-1 is applied, alpha and RGB_view may be reduced as shown in the eighth row from the top of the table in FIG. 7 (method 1-2-1-3).

[0083] For example, the first information processing device may further include a reduction processing unit that reduces the alpha image and the RGB_view image. In the first information processing device, a first atlas conversion unit may perform atlas conversion on the reduced alpha image. A second atlas conversion unit may perform atlas conversion on the RGB_base image and the reduced RGB_view image independently of each other.

[0084] The second information processing device may further include an enlargement processing unit that enlarges the alpha image and the RGB_view image.

[0085] As shown in the example of Fig. 9, for an alpha image, an RGB_base image, and an RGB_view image that are all the same size (e.g., 2K), the alpha image and the RGB_view image may be reduced in size (e.g., to 1K), and then atlas transformed and encoded as shown in the example of Fig. 14. By reducing the image for transmission in this way, and then enlarging it after transmission to restore it to its original size, the amount of data transmitted can be reduced. This makes it possible to further suppress an increase in the transmission capacity required for data transmission.

[0086] Because the alpha image is information indicating the presence of an object, reducing its resolution has a relatively small impact on the subjective image quality of the rendered image. Furthermore, because the RGB_view image is blended using a direction-dependent blend ratio, it has a relatively small impact on the subjective image quality of the rendered image compared to the RGB_base image. Therefore, by reducing the alpha image and RGB_view image as described above, it is possible to further reduce the increase in the transmission capacity required for data transmission while minimizing the reduction in the subjective image quality of the rendered image.

[0087] <Method 1-2-2> When the above-mentioned Method 1-2 is applied, if the sizes of RGB_base and view are different as shown in the ninth row from the top of the table in Fig. 7, atlas transformation information for one of the RGB sizes may be transmitted. Alternatively, atlas transformation information for the other size may not be transmitted (Method 1-2-2).

[0088] For example, in the first information processing device described above, the RGB image may include an RGB_base image indicating the distribution of the RGB_base component among the RGB components, and an RGB_view image indicating the distribution of the RGB_view component among the RGB components. The RGB_base image and the RGB_view image may have different sizes. In the first information processing device, the second atlas conversion unit may generate atlas conversion information for the RGB_base component or the RGB_view component by atlas conversion of the RGB_base image and the RGB_view image. The atlas conversion information may be information indicating the content of the atlas conversion.

[0089] For example, in the second information processing device described above, the RGB image may include an RGB_base image indicating a distribution of the RGB_base component among the RGB components, and an RGB_view image indicating a distribution of the RGB_view component among the RGB components. The RGB_base image and the RGB_view image may have different sizes. The second decoding unit may decode coded data of an atlas image of the RGB_base component and coded data of an atlas image of the RGB_view component, respectively. The second atlas inverse transform unit may perform inverse atlas transform on the atlas image of the RGB_base component and the atlas image of the RGB_view component using atlas transform information for the RGB_base component or the RGB_view component.

[0090] For example, the above-described second information processing device may further include a reduction processing unit that reduces the atlas transformation information of the RGB_base component or the RGB_view component.

[0091] Even when the sizes of the RGB_base image and the RGB_view image are different, as in the example of FIG. 14 , the atlas transformation of both images is performed using a mask image generated from the same alpha image, so the layout of one atlas image and the layout of the other atlas image are similar, as in the example of FIG. 16 . That is, by converting the size of one atlas transformation information, it is possible to make it match the other atlas transformation information. Therefore, it is also possible to generate the other atlas transformation information by converting the size of one atlas transformation information. That is, it is also possible to transmit one atlas transformation information (without transmitting the other atlas transformation information), and then, during the other atlas inverse transformation, to convert the size (scale) of the transmitted one atlas transformation information to generate and apply the other atlas transformation information. This reduces the amount of data transmitted. Therefore, it is possible to further suppress an increase in the transmission capacity required for data transmission.

[0092] The atlas transformation information to be transmitted may be for the RGB_base component or for the RGB_view component. Also, atlas transformation information for a larger image size or atlas transformation information for a smaller image size may be transmitted.

[0093] When transmitting atlas transformation information for a larger image size, the destination may reduce the size of the transmitted atlas transformation information to generate atlas transformation information for a smaller image size. This allows for the transmission of more accurate atlas transformation information and more accurate atlas inverse transformation.

[0094] Furthermore, when transmitting atlas conversion information with a smaller image size, the destination may enlarge the transmitted atlas conversion information to generate atlas conversion information with a larger image size. This can further reduce the amount of data transmitted. Therefore, it is possible to further suppress an increase in the transmission capacity required for data transmission.

[0095] <Method 1-2-2-1> When the above-described method 1-2 is applied, the atlas conversion information, reduction instructions, reduction ratio, etc. to be used may be defined in a file, as shown in the tenth row from the top of the table in FIG.

[0096] For example, assume that the RGB_view image is smaller than the RGB_base image as in the example of FIG. 14 , and the atlas transformation information (RGB_base) is reduced to generate the atlas transformation information (RGB_view). In this case, as shown in FIG. 17 , for example, in a V3C bitstream, the V3C_AVD that stores the atlas image (RGB_base) or the atlas image (RGB_view) may store flag information indicating whether or not reduction processing is to be performed, and, if reduction processing is to be performed, information indicating the reduction rate. That is, this information may be stored in the bitstream and transmitted. Then, when reduction processing is to be performed, the V3C_AD associated with that V3C_AVD may be generated by reducing the other V3C_AD (V3C_AD(1)).

[0097] <Method 1-3 and Method 1-3-1> When the above-mentioned Method 1 is applied, multiple RGB images may be identified using identification information for RGB images (Method 1-3), as shown in the eleventh row from the top of the table in Fig. 7. When this Method 1-3 is applied, multiple RGB images may be identified using attribute_id, as shown in the bottom row of the table in Fig. 7 (Method 1-3-1).

[0098] For example, in the first information processing device described above, the RGB image may include an RGB_base image indicating a distribution of the RGB_base component among the RGB components, and an RGB_view image indicating a distribution of the RGB_view component among the RGB components. Furthermore, a second atlas conversion unit may perform atlas conversion on the RGB_base image and the RGB_view image to generate respective atlas images. Furthermore, the first information processing device may further include a multiplexing unit that generates a bitstream by multiplexing, using identification information, coded data of the atlas image of the alpha component, coded data of the atlas image of the RGB_base component, and coded data of the atlas image of the RGB_view component. Furthermore, in the first information processing device described above, the identification information may be information for identifying the RGB_base component and the RGB_view component.

[0099] For example, in the second information processing device described above, the RGB image may include an RGB_base image indicating a distribution of the RGB_base component among the RGB components, and an RGB_view image indicating a distribution of the RGB_view component among the RGB components. The second information processing device may further include a demultiplexing unit that demultiplexes the bitstream to generate coded data of an atlas image of an alpha component, coded data of an atlas image of an RGB_base component, and coded data of an atlas image of an RGB_view component.

[0100] For example, in the second information processing device described above, the demultiplexing unit may demultiplex the bitstream using identification information for identifying the RGB_base component and the RGB_view component.

[0101] When multiple RGB images exist, as in the case of NeX MPI, each RGB image must be identified. Therefore, as described above, multiple RGB images may be identified using identification information for RGB images. Note that any specification for this identification information is acceptable. For example, the identification information (attribute_id) that identifies an attribute may be used as this identification information. For example, in the case of a V3C bitstream, the vuh_attribute_index stored in the V3C unit header may be used as this identification information.

[0102] By using such identification information, it becomes easy to identify each RGB, and it is possible to suppress an increase in the load of the reproduction process (decoding process).

[0103] <Regarding the Method> Each of the above-described methods can be applied in combination with other methods as long as no contradictions arise. For example, Method 1-3 shown in the table of FIG. 7 may be applied in combination with Method 1-1 or Method 1-2. Also, any two or more of Methods 1-2-1-1 to 1-2-1-3 shown in the table of FIG. 7 may be applied in combination. Of course, combinations other than these are also possible. Also, the method shown in FIG. 7 may be applied in combination with other methods.

[0104] In the description of this specification, a higher-level method may include a lower-level method. In other words, applying method 1 may also include applying any one or more of methods 1-1 to 1-3, 1-1-1, 1-2-1 and 1-2-2, 1-3-1, 1-2-1-1 to 1-2-1-3, and 1-2-2-1. Similarly, applying method 1-1 may also include applying method 1-1-1. Similarly, applying method 1-2 may also include applying any one or more of methods 1-2-1 and 1-2-2, 1-2-1-1 to 1-2-1-3, and 1-2-2-1. Similarly, applying method 1-2-1 may also include applying any one or more of methods 1-2-1-1 to 1-2-1-3. Similarly, applying Method 1-2-2 may include applying Method 1-2-2-1, and similarly, applying Method 1-3 may include applying Method 1-3-1.

[0105] Therefore, for example, when it is described as "applying method 1," this may include applying any one or more of methods 1-1 to 1-3, method 1-1-1, method 1-2-1 and 1-2-2, method 1-3-1, method 1-2-1-1 to 1-2-1-3, and method 1-2-2-1.

[0106] 18 is a block diagram showing a main configuration example of an information processing system to which the present technology described above in <3. Encoding and decoding using atlas transformation> is applied. The information processing system 300 shown in Fig. 18 is a system that generates NeX MPI data, encodes and transmits it, decodes it to restore the NeX MPI data, and generates and displays a rendering image.

[0107] 18, the information processing system 300 includes, as transmitting devices (devices that provide NeX data), a NeX learning device 311, a NeX MPI generation device 312, a post-processing device 313, a packing device 314, and an encoding device 315. The information processing system 300 also includes, as receiving devices (devices that receive and play back NeX data), a decoding device 316, an unpacking device 317, a rendering device 318, a display pre-processing device 319, and a display device 320.

[0108] <Flow of Transmission Process> The transmitting devices (devices that provide NeX data), namely, the NeX learning device 311, the NeX MPI generation device 312, the post-processing device 313, the packing device 314, and the encoding device 315, execute transmission processes and perform processing related to the provision of NeX data. An example of the flow of this transmission process will be described with reference to the flowchart in FIG. 19.

[0109] When the transmission process begins, the NeX training device 311 performs NeX training and sets MLP parameters in step S301. In step S302, the NeX MPI generation device 312 generates a NeX MPI using the MLP. In step S303, the post-processing device 313 performs post-processing on the NeX MPI. In step S304, the packing device 314 packs the post-processed NeX MPI. In step S304, the encoding device 315 performs encoding, encodes the packed NeX MPI, and generates a bitstream. In this case, the encoding device 315 applies the above-mentioned method 1 to atlas-transform and encode alpha images and RGB images. In step S305, the encoding device 315 transmits the generated bitstream.

[0110] <Flow of playback process> As receiving side devices (devices that receive and play back NeX data), the decoding device 316, unpacking device 317, rendering device 318, display pre-processing device 319, and display device 320 execute playback processes and perform processing related to the playback of NeX data. An example of the flow of the playback process will be described with reference to the flowchart in Fig. 20.

[0111] When the playback process starts, in step S351, the decoding device 316 receives the bitstream transmitted from the encoding device 315. In step S352, the decoding device 316 executes a decoding process to decode the received bitstream and generate (restore) an alpha image or an RGB image. In step S353, the unpacking device 317 unpacks the alpha image or the RGB image. In step S354, the rendering device 318 performs rendering using the unpacked alpha image or the RGB image to generate a rendered image of a desired viewpoint. In step S355, the display pre-processing device 319 performs display pre-processing on the rendered image. In step S356, the display device 320 displays the rendered image.

[0112] By executing each process in this manner, the information processing system 300 can suppress an increase in the transmission capacity required for transmitting 3D data that uses MPI, such as NeX.

[0113] <Encoding device> Fig. 21 is a block diagram showing an example of the main configuration of the encoding device 315. As shown in Fig. 21 , the encoding device 315 has a non-zero region extraction unit 511, an atlas transformation unit 512, encoding units 513A and 513B, a mask image generation unit 514, an atlas transformation unit 515, encoding units 516A and 516B, and a multiplexing unit 517.

[0114] The non-zero region extraction unit 511 performs processing related to extraction of non-zero regions of the alpha image. For example, the non-zero region extraction unit 511 may obtain an alpha image to be supplied to the encoding device 315. The non-zero region extraction unit 511 may also extract non-zero regions from the alpha image. The non-zero region extraction unit 511 may also supply the extracted non-zero regions to the atlas conversion unit 512. The non-zero region extraction unit 511 may also supply the alpha image to the mask image generation unit 514. The non-zero region extraction unit 511 may also perform such processing for each layer of a group of layers representing a three-dimensional space.

[0115] The atlas conversion unit 512 performs processing related to atlas conversion of the alpha component. For example, the atlas conversion unit 512 may acquire non-zero regions of the alpha component supplied from the non-zero region extraction unit 511. The atlas conversion unit 512 may perform atlas conversion of the alpha component and arrange the non-zero regions in an atlas image (alpha). The atlas conversion unit 512 may also generate atlas conversion information for the atlas conversion. The atlas conversion unit 512 may also supply the atlas image (alpha) that is the generated atlas image of the alpha component to the encoding unit 513A. The atlas conversion unit 512 may also supply the atlas conversion information (alpha) that is the generated atlas conversion information of the alpha component to the encoding unit 513B. The atlas conversion unit 512 may perform such processing for each layer of a group of layers representing a three-dimensional space.

[0116] The encoding unit 513A and the encoding unit 513B perform processing related to encoding of the alpha component. For example, the encoding unit 513A may acquire an atlas image (alpha) supplied from the atlas conversion unit 512. The encoding unit 513A may encode the atlas image (alpha) using a 2D encoding method (e.g., JPEG (Joint Photographic Experts Group) or MPEG) to generate encoded data. The encoding unit 513A may supply the encoded data (also referred to as encoded data (alpha)) of the generated atlas image (alpha) to the multiplexing unit 517. The encoding unit 513B may also acquire atlas transformation information (alpha) supplied from the atlas conversion unit 512. The encoding unit 513B may encode the atlas transformation information (alpha) to generate encoded data. Any method may be used to encode this atlas transformation information (alpha). For example, the encoding unit 513B may encode the atlas transformation information (alpha) according to the MIV / V3C standard. The encoding unit 513B may supply the encoded data of the generated atlas transformation information (alpha) (which may be included in the encoded data (alpha)) to the multiplexing unit 517. Note that the encoding units 513A and 513B may perform such processing for each layer of a group of layers representing a three-dimensional space. Note that the encoding units 513A and 513B are also referred to as encoding units 513.

[0117] The mask image generation unit 514 performs processing related to the generation of a mask image. For example, the mask image generation unit 514 may acquire an alpha image supplied from the non-zero region extraction unit 511. The mask image generation unit 514 may also generate a mask image using the acquired alpha image. In this case, the mask image generation unit 514 may generate intermediate data (mask image) by combining the alpha images of each of multiple consecutive layers corresponding to a single RGB image. The mask image generation unit 514 may also supply the generated mask image to the atlas conversion unit 515.

[0118] The atlas conversion unit 515 performs processing related to atlas conversion of RGB components. For example, the atlas conversion unit 515 may acquire RGB images (RGB_base image and RGB_view image) supplied to the encoding device 315. Alternatively, the atlas conversion unit 515 may acquire a mask image supplied from the mask image generation unit 514. Furthermore, the atlas conversion unit 515 may perform atlas conversion of the RGB components using the mask image and the RGB image. For example, the atlas conversion unit 515 may extract areas corresponding to non-zero areas of the mask image from the RGB_base image and arrange them in the atlas image (RGB_base). Alternatively, the atlas conversion unit 515 may extract areas corresponding to non-zero areas of the mask image from the RGB_view image and arrange them in the atlas image (RGB_view). Alternatively, the atlas conversion unit 515 may generate atlas conversion information (RGB) that is atlas conversion information common to these atlas conversions. The atlas conversion unit 515 may also supply an atlas image (RGB_base) that is an atlas image of the generated RGB_base components and an atlas image (RGB_view) that is an atlas image of the generated RGB_view components to the encoding unit 516A. The atlas conversion unit 515 may also supply atlas conversion information (RGB) that is atlas conversion information of the generated RGB components to the encoding unit 516B. Note that the atlas conversion unit 515 may perform such processing for each of "plural consecutive layers corresponding to a single RGB image" in a group of layers that represent a three-dimensional space.

[0119] The encoding unit 516A and the encoding unit 516B perform processing related to encoding of the RGB_base component and the RGB_view component (which may also be referred to as the RGB component). For example, the encoding unit 516A may acquire an atlas image (RGB_base) supplied from the atlas conversion unit 515. The encoding unit 516A may encode the atlas image (RGB_base) using a 2D encoding method (e.g., JPEG, MPEG, etc.) to generate encoded data. The encoding unit 516A may supply encoded data of the generated atlas image (RGB_base) (also referred to as encoded data (RGB_base)) to the multiplexing unit 517. The encoding unit 516A may also acquire an atlas image (RGB_view) supplied from the atlas conversion unit 515. The encoding unit 516A may encode the atlas image (RGB_view) using a 2D encoding method (e.g., JPEG, MPEG, etc.) to generate encoded data. The encoding unit 516A may supply encoded data of the generated atlas image (RGB_view) (also referred to as encoded data (RGB_view)) to the multiplexing unit 517. The encoding unit 516B may also acquire atlas transformation information (RGB) supplied from the atlas transformation unit 515. The encoding unit 516B may encode the atlas transformation information (RGB) to generate encoded data. Any method may be used to encode the atlas transformation information (RGB). For example, the encoding unit 516B may encode the atlas transformation information (RGB) in accordance with the MIV / V3C standard. The encoding unit 516B may supply encoded data of the generated atlas transformation information (RGB) (also referred to as encoded data (RGB)) to the multiplexing unit 517. The encoding units 516A and 516B may be executed for each of "plural consecutive layers corresponding to a single RGB image" in a group of layers representing a three-dimensional space. The encoding units 516A and 516B are also referred to as encoding units 516.

[0120] The multiplexing unit 517 performs processing related to data multiplexing. For example, the multiplexing unit 517 may acquire coded data (alpha) (coded data of atlas image (alpha) and atlas transformation information (alpha)) supplied from the coding unit 513A and the coding unit 513B. The multiplexing unit 517 may also acquire coded data (RGB_base) and coded data (RGB_view) supplied from the coding unit 516A. The multiplexing unit 517 may also acquire coded data of atlas transformation information (RGB) (coded data (RGB)) supplied from the coding unit 516B. The multiplexing unit 517 may also multiplex the acquired information to generate a bitstream. The multiplexing unit 517 may also transmit the generated bitstream.

[0121] The present technology described above in <3. Encoding / Decoding Using Atlas Transformation> may be applied to the encoding device 315 configured as described above. For example, the above-described method 1 may be applied to the encoding device 315. For example, the atlas transformation unit 512 may perform atlas transformation on an alpha image of 3D data to generate an atlas image of the alpha component. Furthermore, the encoding unit 513A may encode the atlas image of the alpha component. Furthermore, the atlas transformation unit 515 may perform atlas transformation on an RGB image of 3D data to generate an atlas image of the RGB components. Furthermore, the encoding unit 516A may encode the atlas image of the RGB components.

[0122] Alternatively, the above-described method 1-1 may be applied to the encoding device 315. For example, the mask image generation unit 514 may generate a mask image by combining non-zero regions of alpha images of multiple consecutive layers. Furthermore, the atlas transformation unit 515 may perform atlas transformation on the RGB image using the mask image.

[0123] Alternatively, the above-described method 1-1-1 may be applied to the encoding device 315. For example, the RGB image may include an RGB_base image indicating the distribution of the RGB_base component among the RGB components, and an RGB_view image indicating the distribution of the RGB_view component among the RGB components. Alternatively, the atlas conversion unit 515 may perform atlas conversion on the RGB_base image and the RGB_view image to generate respective atlas images, and generate atlas conversion information common to the RGB_base component and the RGB_view component.

[0124] The above-described method 1-3 may also be applied to the encoding device 315. The above-described method 1-3-1 may also be applied to the encoding device 315. For example, the RGB image may include an RGB_base image indicating the distribution of the RGB_base component among the RGB components, and an RGB_view image indicating the distribution of the RGB_view component among the RGB components. The atlas conversion unit 515 may perform atlas conversion on the RGB_base image and the RGB_view image to generate respective atlas images. The multiplexing unit 517 may multiplex the coded data of the atlas image of the alpha component, the coded data of the atlas image of the RGB_base component, and the coded data of the atlas image of the RGB_view component using identification information to generate a bitstream. The identification information may also be information for identifying the RGB_base component and the RGB_view component.

[0125] With this configuration, the encoding device 315 can suppress an increase in the transmission capacity required for transmitting 3D data that uses MPI, such as NeX. Note that the processing up to the atlas conversion may be performed when packing the post-processed NeX MPI. In other words, the packing device 314 and the encoding device 315 may be integrated.

[0126] <Flow of Encoding Process> An example of the flow of the encoding process executed by the encoding device 315 will be described with reference to the flowchart of FIG.

[0127] When the encoding process starts, in step S501, the non-zero region extraction unit 511 extracts non-zero regions from the alpha image. In step S502, the atlas transformation unit 512 performs atlas transformation on the alpha component to generate an atlas image (alpha) and atlas transformation information (alpha). In step S503, the encoding unit 513A encodes the atlas image of the alpha component to generate encoded data (alpha). In addition, the encoding unit 513B encodes the atlas transformation information of the alpha component to generate encoded data of the atlas transformation information (alpha) (encoded data (alpha)).

[0128] In step S504, the mask image generator 514 generates a mask image using the non-zero regions of the main layer and sublayer of the alpha image.

[0129] In step S505, the atlas conversion unit 515 performs atlas conversion on the RGB_base using the mask image to generate an atlas image (RGB_base). In step S506, the encoding unit 516A encodes the atlas image of the RGB_base components to generate encoded data (RGB_base).

[0130] In step S507, the atlas conversion unit 515 performs atlas conversion on the RGB_view using the mask image to generate an atlas image (RGB_view). In step S508, the encoding unit 516A encodes the atlas image of the RGB_view components to generate encoded data (RGB_view).

[0131] The atlas transformation unit 515 generates atlas transformation information (RGB) in the process of step S505 or step S507. In step S509, the encoding unit 516B encodes the atlas transformation information of the RGB components to generate encoded data of the atlas transformation information (RGB) (encoded data (RGB)).

[0132] In step S510, the multiplexing unit 517 multiplexes the coded data (alpha) (coded data of the atlas image (alpha) and the atlas transformation information (alpha)), coded data (RGB_base), coded data (RGB_view), and coded data (RGB) to generate a bitstream.

[0133] When the process of step S509 ends, the encoding process ends and the process returns to FIG.

[0134] In this way, in each process, the present technology described above in <3. Encoding / Decoding Using Atlas Transform> may be applied. For example, the above-described method 1 may be applied. The above-described method 1-1 may be applied. The above-described method 1-1-1 may be applied. The above-described method 1-3 may be applied. The above-described method 1-3-1 may be applied.

[0135] By performing each process in this manner, the encoding device 315 can suppress an increase in the transmission capacity required for transmitting 3D data that uses MPI, such as NeX.

[0136] <Decoding device> Fig. 23 is a block diagram showing an example of the main configuration of the decoding device 316. As shown in Fig. 23 , the decoding device 316 has a demultiplexing unit 551, a decoding unit 552A, a decoding unit 552B, an atlas inverse transform unit 553, a decoding unit 554A, a decoding unit 554B, and an atlas inverse transform unit 555.

[0137] The demultiplexing unit 551 performs processing related to demultiplexing. For example, the demultiplexing unit 551 may acquire a bitstream to be supplied to the decoding device 316. The demultiplexing unit 551 may demultiplex the bitstream to generate coded data of the atlas image (alpha) (coded data (alpha)), coded data of the atlas transformation information (alpha) (coded data (alpha)), coded data of the atlas image (RGB_base) (coded data (RGB_base)), coded data of the atlas image (RGB_view) (coded data (RGB_view)), and coded data of the atlas transformation information (RGB) (coded data (RGB)). The demultiplexing unit 551 may also supply the coded data of the atlas image (alpha) (coded data (alpha)) to the decoding unit 552A. The demultiplexing unit 551 may also supply the coded data of the atlas transformation information (alpha) (coded data (alpha)) to the decoding unit 552B. The demultiplexing unit 551 may also supply the coded data of the atlas image (RGB_base) (coded data (RGB_base)) and the coded data of the atlas image (RGB_view) (coded data (RGB_view)) to the decoding unit 554A. The demultiplexing unit 551 may also supply the coded data of the atlas transformation information (RGB) (coded data (RGB)) to the decoding unit 554B.

[0138] The decoding unit 552A and the decoding unit 552B perform processing related to decoding of the alpha component. For example, the decoding unit 552A may acquire coded data (coded data (alpha)) of the atlas image (alpha) supplied from the demultiplexing unit 551. The decoding unit 552A may decode the coded data (alpha) using a 2D decoding method (e.g., JPEG, MPEG, etc.) to generate the atlas image (alpha). The decoding unit 552A may supply the atlas image (alpha) to the atlas inverse transform unit 553. The decoding unit 552B may also acquire coded data (coded data (alpha)) of the atlas transformation information (alpha) supplied from the demultiplexing unit 551. The decoding unit 552B may decode the coded data (alpha) using a 2D decoding method (e.g., JPEG, MPEG, etc.) to generate the atlas transformation information (alpha). Any decoding method may be used. For example, the decoding unit 552B may decode the coded data (alpha) in accordance with the MIV / V3C standard. The decoding unit 552B may also supply the atlas transformation information (alpha) to the atlas inverse transformation unit 553. The decoding unit 552A and the decoding unit 552B are also referred to as the decoding unit 552.

[0139] The atlas inverse transformation unit 553 performs processing related to the atlas inverse transformation of the alpha component. For example, the atlas inverse transformation unit 553 may acquire an atlas image (alpha) supplied from the decoding unit 552A. The atlas inverse transformation unit 553 may also acquire atlas transformation information (alpha) supplied from the decoding unit 552B. The atlas inverse transformation unit 553 may also perform inverse atlas transformation on the atlas image (alpha) using the atlas transformation information (alpha) to generate (restore) an alpha image. The atlas inverse transformation unit 553 may also supply the generated alpha image to the unpacking device 317.

[0140] The decoding unit 554A and the decoding unit 554B perform processing related to the decoding of RGB components. For example, the decoding unit 554A may acquire the coded data (RGB_base) and the coded data (RGB_view) supplied from the demultiplexing unit 551. The decoding unit 554A may decode the coded data (RGB_base) and the coded data (RGB_view) using a 2D decoding method (e.g., JPEG, MPEG, etc.) to generate an atlas image (RGB_base) and an atlas image (RGB_view). The decoding unit 554A may supply the atlas image (RGB_base) and the atlas image (RGB_view) to the atlas inverse transform unit 555. The decoding unit 554B may acquire the coded data (RGB) supplied from the demultiplexing unit 551. The decoding unit 554B may decode the coded data (RGB) using a 2D decoding method (e.g., JPEG, MPEG, etc.) to generate atlas transform information (RGB). Any decoding method may be used. For example, the decoding unit 554B may decode the coded data (RGB) in accordance with the MIV / V3C standard. The decoding unit 554B may also supply the atlas transformation information (RGB) to the atlas inverse transformation unit 555. The decoding units 554A and 554B are also referred to as the decoding unit 554.

[0141] The atlas inverse transform unit 555 performs processing related to the atlas inverse transform of RGB components. For example, the atlas inverse transform unit 555 may acquire an atlas image (RGB_base) and an atlas image (RGB_view) supplied from the decoding unit 554A. The atlas inverse transform unit 555 may also acquire atlas transformation information (RGB) supplied from the decoding unit 554B. The atlas inverse transform unit 555 may also perform inverse atlas transform on the atlas image (RGB_base) and the atlas image (RGB_view) using the atlas transformation information (RGB) to generate (restore) RGB images (RGB_base image and RGB_view image). The atlas inverse transform unit 553 may supply the generated RGB images (RGB_base image and RGB_view image) to the unpacking device 317.

[0142] The present technology described above in <3. Encoding and Decoding Using Atlas Transform> may be applied to the decoding device 316 configured as described above. For example, the above-described method 1 may be applied to the decoding device 316. For example, the decoding unit 552A may decode the encoded data and generate an atlas image of an alpha component. Furthermore, the atlas inverse transform unit 553 may perform an atlas inverse transform on the atlas image of the alpha component to generate an alpha image of 3D data. Furthermore, the decoding unit 554A may decode the encoded data and generate an atlas image of RGB components. Furthermore, the atlas inverse transform unit 555 may perform an atlas inverse transform on the atlas image of the RGB components to generate an RGB image of 3D data.

[0143] The above-described method 1-1-1 may also be applied to the decoding device 316. For example, the RGB image may include an RGB_base image indicating the distribution of the RGB_base component among the RGB components, and an RGB_view image indicating the distribution of the RGB_view component among the RGB components. The decoding unit 554A may also decode the encoded data to generate an atlas image of the RGB_base component and an atlas image of the RGB_view component. The atlas inverse transform unit 555 may also perform inverse atlas transform on the atlas image of the RGB_base component and the atlas image of the RGB_view component using atlas transform information common to the RGB_base component and the RGB_view component.

[0144] The above-described method 1-3 may also be applied to the decoding device 316. The above-described method 1-3-1 may also be applied to the decoding device 316. For example, the RGB image may include an RGB_base image indicating the distribution of the RGB_base component among the RGB components, and an RGB_view image indicating the distribution of the RGB_view component among the RGB components. The demultiplexing unit 551 may also demultiplex the bitstream to generate coded data of an atlas image of the alpha component, coded data of an atlas image of the RGB_base component, and coded data of an atlas image of the RGB_view component. The demultiplexing unit 551 may also demultiplex the bitstream using identification information for identifying the RGB_base component and the RGB_view component.

[0145] With this configuration, the decoding device 316 can suppress an increase in the transmission capacity required for transmitting 3D data that uses MPI, such as NeX. Note that the atlas inverse transform may be performed when unpacking an alpha image or an RGB image (RGB_base image and RGB_view image). In other words, the decoding device 316 and the unpacking device 317 may be integrated.

[0146] <Flow of Decoding Process> An example of the flow of the decoding process executed by the decoding device 316 will be described with reference to the flowchart of FIG.

[0147] When the decoding process starts, in step S551, the demultiplexing unit 551 demultiplexes the bit stream to generate coded data of the atlas image (alpha), coded data of the atlas transformation information (alpha), coded data of the atlas image (RGB_base), coded data of the atlas image (RGB_view), and coded data of the atlas transformation information (RGB).

[0148] In step S552, the decoding unit 552A decodes the coded data of the alpha component to generate an atlas image (alpha). Furthermore, the decoding unit 552B decodes the coded data of the atlas transformation information (alpha) (coded data (alpha)) to generate the atlas transformation information (alpha). In step S553, the atlas inverse transformation unit 553 performs inverse atlas transformation on the atlas image (alpha) using the atlas transformation information (alpha) to generate an alpha image.

[0149] In step S554, the decoding unit 554B decodes the coded data (RGB) to generate atlas transformation information (RGB). In step S555, the decoding unit 554A decodes the coded data of the RGB_base component to generate an atlas image (RGB_base). In step S556, the atlas inverse transform unit 555 performs an inverse atlas transform on the atlas image (RGB_base) using the atlas transformation information (RGB) to generate an RGB_base image.

[0150] In step S557, the decoding unit 554A decodes the coded data of the RGB_view components to generate an atlas image (RGB_view). In step S558, the atlas inverse transform unit 555 performs inverse atlas transform on the atlas image (RGB_view) using the atlas transformation information (RGB) to generate an RGB_view image.

[0151] When the process of step S558 ends, the decoding process ends and the process returns to FIG.

[0152] In this way, in each process, the present technology described above in <3. Encoding / Decoding Using Atlas Transform> may be applied. For example, the above-described method 1 may be applied. The above-described method 1-1 may be applied. The above-described method 1-1-1 may be applied. The above-described method 1-3 may be applied. The above-described method 1-3-1 may be applied.

[0153] By performing each process in this manner, the decoding device 316 can suppress an increase in the transmission capacity required for transmitting 3D data that uses MPI, such as NeX.

[0154] <Encoding device> Furthermore, the method 1-2 described above in <3. Encoding and decoding using atlas transformation> may be applied to the encoding device 315. In that case, an example of the main configuration of the encoding device 315 is shown in FIG.

[0155] In this case, the encoding device 315 also has basically the same configuration as the example in Fig. 21 . However, in the example in Fig. 25 , the non-zero region extraction unit 511 includes a non-zero region extraction unit 611 and a non-zero region extraction unit 612. Furthermore, the mask image generation unit 514 includes a mask image generation unit 613 and a mask image generation unit 614. Furthermore, the atlas transformation unit 515 includes an atlas transformation unit 615 and an atlas transformation unit 616. Furthermore, the encoding unit 516 includes an encoding unit 617A, an encoding unit 617B, an encoding unit 618A, and an encoding unit 618B. Furthermore, in this example, the encoding device 315 further includes a reduction processing unit 621 and a reduction processing unit 622.

[0156] The non-zero region extraction unit 611 may obtain a reduced alpha image (also referred to as a reduced alpha image) supplied from the reduction processing unit 621. The non-zero region extraction unit 611 may also extract non-zero regions from the reduced alpha image. The non-zero region extraction unit 611 may also supply the extracted non-zero regions to the atlas conversion unit 512. The non-zero region extraction unit 611 may also supply the reduced alpha image to the mask image generation unit 613.

[0157] The non-zero region extraction unit 612 may obtain an alpha image to be supplied to the encoding device 315. The non-zero region extraction unit 612 may also extract non-zero regions from the alpha image. The non-zero region extraction unit 612 may also supply the extracted non-zero regions and the alpha image to the mask image generation unit 614.

[0158] The mask image generation unit 613 may acquire the reduced alpha image supplied from the non-zero region extraction unit 611. The mask image generation unit 613 may also generate a mask image (also referred to as a reduced mask image) using the acquired reduced alpha image. In this case, the mask image generation unit 613 may generate intermediate data (reduced mask image) by synthesizing the reduced alpha images of each of multiple consecutive layers corresponding to a single RGB image. The mask image generation unit 613 may also supply the generated reduced mask image to the atlas conversion unit 615.

[0159] The mask image generation unit 614 may acquire the alpha image and its nonzero regions supplied from the nonzero region extraction unit 612. The mask image generation unit 614 may also generate a mask image using the acquired alpha image and its nonzero regions. In this case, the mask image generation unit 614 may generate intermediate data (mask image) by synthesizing the alpha images of each of multiple consecutive layers corresponding to a single RGB image. The mask image generation unit 614 may also supply the generated mask image to the atlas conversion unit 616.

[0160] The atlas conversion unit 615 may acquire a reduced RGB_view image (also referred to as a reduced RGB_view image) supplied from the reduction processing unit 622. The atlas conversion unit 615 may also acquire a reduced mask image supplied from the mask image generation unit 613. The atlas conversion unit 615 may also perform atlas conversion of the RGB_view components using the reduced mask image and the reduced RGB_view image. For example, the atlas conversion unit 615 may extract, from the reduced RGB_view image, an area corresponding to a non-zero area in the reduced mask image and arrange the extracted area in an atlas image (RGB_view). The atlas conversion unit 615 may also generate atlas conversion information (RGB_view) related to the atlas conversion. The atlas conversion unit 615 may also supply the generated atlas image (RGB_view), which is an atlas image of the RGB_view components, to the encoding unit 617A. The atlas conversion unit 615 may also supply the generated atlas conversion information (RGB_view) to the encoding unit 617B. The atlas conversion unit 615 may perform such processing for each of "plural consecutive layers corresponding to a single RGB image" in the layer group that represents the three-dimensional space.

[0161] The atlas conversion unit 616 may acquire an RGB_base image supplied to the encoding device 315. Alternatively, the atlas conversion unit 616 may acquire a mask image supplied from the mask image generation unit 614. Alternatively, the atlas conversion unit 616 may perform atlas conversion of the RGB_base component using the mask image and the RGB_base image. For example, the atlas conversion unit 616 may extract an area corresponding to a non-zero area of ​​the mask image from the RGB_base image and place the extracted area in an atlas image (RGB_base). Alternatively, the atlas conversion unit 616 may generate atlas conversion information (RGB_base) related to the atlas conversion. Alternatively, the atlas conversion unit 616 may supply the generated atlas image (RGB_base), which is an atlas image of the RGB_base component, to the encoding unit 618A. Alternatively, the atlas conversion unit 616 may supply the generated atlas conversion information (RGB_base) to the encoding unit 618B. The atlas conversion unit 616 may perform this processing for each of "plural consecutive layers corresponding to a single RGB image" in the layer group that represents the three-dimensional space.

[0162] The encoding unit 617A and the encoding unit 617B perform processing related to encoding of the RGB_view component. For example, the encoding unit 617A may acquire an atlas image (RGB_view) supplied from the atlas transformation unit 615. The encoding unit 617A may also encode the atlas image (RGB_view) using a 2D encoding method (e.g., JPEG, MPEG, etc.) to generate encoded data. The encoding unit 617A may also supply encoded data (encoded data (RGB_view)) of the generated atlas image (RGB_view) to the multiplexing unit 517. The encoding unit 617B may also acquire atlas transformation information (RGB_view) supplied from the atlas transformation unit 515. The encoding unit 617B may also encode the atlas transformation information (RGB_view) to generate encoded data. Any method may be used to encode the atlas transformation information (RGB_view). For example, the encoding unit 617B may encode the atlas transformation information (RGB_view) in accordance with the MIV / V3C standard. The encoding unit 617B may supply encoded data of the generated atlas transformation information (RGB_view) (also referred to as encoded data (RGB_view)) to the multiplexing unit 517. The encoding units 617A and 617B may be executed for each "plurality of consecutive layers corresponding to a single RGB image" in a group of layers representing a three-dimensional space. The encoding units 617A and 617B are also referred to as the encoding unit 617.

[0163] The encoding units 618A and 618B perform processing related to encoding of the RGB_base component. For example, the encoding unit 618A may acquire an atlas image (RGB_base) supplied from the atlas conversion unit 616. The encoding unit 618A may encode the atlas image (RGB_base) using a 2D encoding method (e.g., JPEG, MPEG, etc.) to generate encoded data. The encoding unit 618A may supply encoded data (encoded data (RGB_base)) of the generated atlas image (RGB_base) to the multiplexing unit 517. The encoding unit 618B may also acquire atlas transformation information (RGB_base) supplied from the atlas conversion unit 515. The encoding unit 618B may encode the atlas transformation information (RGB_base) to generate encoded data. Any method may be used to encode the atlas transformation information (RGB_base). For example, the encoding unit 618B may encode the atlas transformation information (RGB_base) according to the MIV / V3C standard. The encoding unit 618B may supply the generated encoded data of the atlas transformation information (RGB_base) (also referred to as encoded data (RGB_base)) to the multiplexing unit 517. The encoding units 618A and 618B may be executed for each "plurality of consecutive layers corresponding to a single RGB image" in a group of layers expressing a three-dimensional space. The encoding units 618A and 618B are also referred to as the encoding unit 618.

[0164] In this case, the multiplexing unit 517 may acquire coded data (alpha) (coded data of the atlas image (alpha)) supplied from the coding unit 513A. The multiplexing unit 517 may also acquire coded data (alpha) (coded data of the atlas transformation information (alpha)) supplied from the coding unit 513B. The multiplexing unit 517 may also acquire coded data (RGB_view) (coded data of the atlas image (RGB_view)) supplied from the coding unit 617A. The multiplexing unit 517 may also acquire coded data (RGB_view) (coded data of the atlas transformation information (RGB_view)) supplied from the coding unit 617B. The multiplexing unit 517 may also acquire coded data (RGB_base) (coded data of the atlas image (RGB_base)) supplied from the coding unit 618A. The multiplexing unit 517 may also acquire coded data (RGB_base) (coded data of the atlas transformation information (RGB_base)) supplied from the coding unit 617B. The multiplexing unit 517 may multiplex the acquired information to generate a bit stream, and may transmit the generated bit stream.

[0165] The reduction processing unit 621 may generate a reduced alpha image by reducing the alpha image supplied to the encoding device 315. The reduction processing unit 621 may supply the generated reduced alpha image to the non-zero region extraction unit 611.

[0166] The reduction processing unit 622 may generate a reduced RGB_view image by reducing the RGB_view image supplied to the encoding device 315. The reduction processing unit 622 may supply the generated reduced RGB_view image to the atlas conversion unit 615.

[0167] The present technology described above in <3. Encoding and Decoding Using Atlas Transformation> may be applied to the encoding device 315 configured as above. For example, the above-described method 1-2 may be applied to this encoding device 315. For example, the alpha image may be converted to the same size as the RGB image.

[0168] The above-described method 1-2-1 may also be applied to this encoding device 315. For example, an RGB image may include an RGB_base image indicating the distribution of RGB_base components among the RGB components, and an RGB_view image indicating the distribution of RGB_view components among the RGB components. Then, the atlas conversion unit 515 (atlas conversion unit 615 and atlas conversion unit 616) may perform atlas conversion independently on the RGB_base image and the RGB_view image, which have different sizes.

[0169] The above-described method 1-2-1-1 may also be applied to this encoding device 315. For example, the atlas transformation unit 616 may generate atlas transformation information for the RGB_base components by performing atlas transformation on the RGB_base image. Also, the atlas transformation unit 615 may generate atlas transformation information for the RGB_view components by performing atlas transformation on the RGB_view image.

[0170] Furthermore, the above-described method 1-2-1-2 may be applied to this encoding device 315. For example, the mask image generation unit 514 (the mask image generation unit 613 and the mask image generation unit 614) may generate mask images for each of the RGB_base image and the RGB_view image by combining non-zero regions of alpha images of multiple consecutive layers. Then, the atlas conversion unit 616 may perform atlas conversion on the RGB_base image using the mask image for the RGB_base image. Furthermore, the atlas conversion unit 615 may perform atlas conversion on the RGB_view image using the mask image for the RGB_view image.

[0171] The above-described method 1-2-1-3 may also be applied to this encoding device 315. For example, the reduction processing unit 621 may reduce the alpha image. Alternatively, the reduction processing unit 622 may reduce the RGB_view image. Alternatively, the atlas conversion unit 512 may perform atlas conversion on the reduced alpha image. Alternatively, the atlas conversion unit 515 (the atlas conversion unit 615 and the atlas conversion unit 616) may perform atlas conversion on the RGB_base image and the reduced RGB_view image independently of each other.

[0172] With this configuration, the encoding device 315 can suppress an increase in the transmission capacity required for transmitting 3D data that uses MPI, such as NeX. In this case, the processing up to the atlas conversion may also be performed when packing the post-processed NeX MPI. In other words, the packing device 314 and the encoding device 315 may be integrated.

[0173] <Flow of Encoding Process> An example of the flow of the encoding process executed by the encoding device 315 will be described with reference to the flowchart of FIG.

[0174] When the encoding process starts, the reduction processing unit 621 reduces the alpha image in step S601. In step S602, the non-zero region extraction unit 611 extracts non-zero regions from the reduced alpha image. In step S603, the atlas transformation unit 512 performs atlas transformation on the alpha component to generate an atlas image (alpha) and atlas transformation information (alpha). In step S604, the encoding unit 513A encodes the atlas image of the alpha component to generate encoded data (alpha). In addition, the encoding unit 513B encodes the atlas transformation information of the alpha component to generate encoded data (alpha).

[0175] In step S605, the mask image generation unit 613 generates a reduced mask image using the non-zero regions of the main layer and sub-layer of the reduced alpha image. In step S606, the reduction processing unit 622 reduces the RGB_view image.

[0176] In step S607, the atlas transformation unit 615 performs atlas transformation on the RGB_view using the reduced mask image to generate an atlas image (RGB_view). In step S608, the encoding unit 617A encodes the atlas image of the RGB_view components to generate encoded data (RGB_view). Furthermore, the encoding unit 617B encodes the atlas transformation information of the RGB_view components to generate encoded data (RGB_view).

[0177] In step S609, the non-zero region extraction unit 612 extracts non-zero regions from the alpha image. In step S610, the mask image generation unit 614 generates a mask image using the non-zero regions of the main layer and sub-layer of the alpha image.

[0178] In step S611, the atlas conversion unit 616 performs atlas conversion on the RGB_base using the mask image to generate an atlas image (RGB_base). In step S612, the encoding unit 618A encodes the atlas image of the RGB_base components to generate encoded data (RGB_base). In addition, the encoding unit 618B encodes the atlas conversion information of the RGB_base components to generate encoded data (RGB_base).

[0179] In step S613, the multiplexing unit 517 multiplexes the coded data of the atlas image (alpha), the coded data of the atlas transformation information (alpha), the coded data of the atlas image (RGB_base), the coded data of the atlas image (RGB_view), the coded data of the atlas transformation information (RGB_base), and the coded data of the atlas transformation information (RGB_view) to generate a bitstream.

[0180] When the process of step S613 ends, the encoding process ends and the process returns to FIG.

[0181] In this way, in each process, the present technology described above in <3. Encoding / Decoding Using Atlas Transform> may be applied. For example, the above-described method 1-2 may be applied. The above-described method 1-2-1 may also be applied. The above-described method 1-2-1-1 may also be applied. The above-described method 1-2-1-2 may also be applied. The above-described method 1-2-1-3 may also be applied.

[0182] By performing each process in this manner, the encoding device 315 can suppress an increase in the transmission capacity required for transmitting 3D data that uses MPI, such as NeX.

[0183] <Decoding device> Furthermore, the method 1-2 described above in <3. Encoding and decoding using atlas transformation> may be applied to the decoding device 316. In that case, an example of the main configuration of the decoding device 316 is shown in FIG.

[0184] In this case, the decoding device 316 also has basically the same configuration as the example in FIG. 23 . However, in the example in FIG. 27 , the decoding device 316 has a decoding unit 652A, a decoding unit 652B, a decoding unit 654A, and a decoding unit 654B instead of the decoding unit 554. Furthermore, the decoding device 316 has an atlas inverse transformation unit 653 and an atlas inverse transformation unit 655 instead of the atlas inverse transformation unit 555. Furthermore, the decoding device 316 also has an enlargement processing unit 651 and an enlargement processing unit 656. In this case, the demultiplexing unit 551 may acquire and demultiplex a bitstream as in the case of FIG. 23 . The demultiplexing unit 551 may supply coded data (coded data (alpha)) of the generated atlas image (alpha) to the decoding unit 552A. The demultiplexing unit 551 may supply coded data (coded data (alpha)) of the generated atlas transformation information (alpha) to the decoding unit 552B. The demultiplexing unit 551 may supply the coded data (coded data (RGB_base)) of the generated atlas image (RGB_base) to the decoding unit 652A. The demultiplexing unit 551 may supply the coded data (coded data (RGB_base)) of the generated atlas transformation information (RGB_base) to the decoding unit 652B. The demultiplexing unit 551 may supply the coded data (coded data (RGB_view)) of the generated atlas image (RGB_view) to the decoding unit 654A. The demultiplexing unit 551 may supply the coded data (coded data (RGB_view)) of the generated atlas transformation information (RGB_view) to the decoding unit 654B. The decoding unit 552A may decode the coded data (alpha) in the same manner as in the case of FIG. 23 and supply the generated atlas image (alpha) to the atlas inverse transformation unit 553. The decoding unit 552B may decode the coded data (alpha) in the same manner as in the case of FIG. 23 and supply the generated atlas transformation information (alpha) to the atlas inverse transformation unit 553.

[0185] In this case, the atlas inverse transformation unit 553 may perform inverse atlas transformation on the atlas image (alpha) to generate a reduced alpha image. That is, the atlas inverse transformation unit 553 may acquire the atlas image (alpha) supplied from the decoding unit 552A. The atlas inverse transformation unit 553 may also acquire atlas transformation information (alpha) supplied from the decoding unit 552B. The atlas inverse transformation unit 553 may also perform inverse atlas transformation on the atlas image (alpha) using the atlas transformation information (alpha) to generate (restore) a reduced alpha image. The atlas inverse transformation unit 553 may supply the reduced alpha image to the enlargement processing unit 651. The enlargement processing unit 651 performs processing related to enlargement of the reduced alpha image. For example, the enlargement processing unit 651 may acquire the reduced alpha image supplied from the atlas inverse transformation unit 553. The enlargement processing unit 651 may enlarge the reduced alpha image to generate (restore) an alpha image. The enlargement processing unit 651 may supply the generated alpha image to the unpacking device 317.

[0186] The decoding unit 652A and the decoding unit 652B perform processing related to decoding of the RGB_base component. For example, the decoding unit 652A may acquire coded data (coded data (RGB_base)) of the atlas image (RGB_base) supplied from the demultiplexing unit 551. The decoding unit 652A may decode the coded data (RGB_base) using a 2D decoding method (e.g., JPEG, MPEG, etc.) to generate the atlas image (RGB_base). The decoding unit 652A may supply the atlas image (RGB_base) to the atlas inverse transform unit 653. The decoding unit 652B may acquire coded data (coded data (RGB_base)) of the atlas transformation information (RGB_base) supplied from the demultiplexing unit 551. The decoding unit 652B may decode the coded data (RGB_base) to generate the atlas transformation information (RGB_base). Any decoding method may be used. For example, the decoding unit 652B may decode the coded data (RGB_base) in accordance with the MIV / V3C standard. The decoding unit 652B may also supply the atlas transformation information (RGB_base) to the atlas inverse transformation unit 653. The decoding unit 652A and the decoding unit 652B are also referred to as the decoding unit 652.

[0187] The atlas inverse transform unit 653 performs processing related to the atlas inverse transform of the RGB_base component. For example, the atlas inverse transform unit 653 may acquire an atlas image (RGB_base) supplied from the decoding unit 652A. The atlas inverse transform unit 653 may also acquire atlas transform information (RGB_base) supplied from the decoding unit 652B. The atlas inverse transform unit 653 may also perform an atlas inverse transform on the atlas image (RGB_base) using the atlas transform information (RGB_base) to generate (restore) an RGB_base image. The atlas inverse transform unit 653 may also supply the generated RGB_base image to the unpacking device 317.

[0188] The decoding unit 654A and the decoding unit 654B perform processing related to decoding of the RGB_view component. For example, the decoding unit 654A may acquire coded data (coded data (RGB_view)) of the atlas image (RGB_view) supplied from the demultiplexing unit 551. The decoding unit 654A may decode the coded data (RGB_view) using a 2D decoding method (e.g., JPEG, MPEG, etc.) to generate the atlas image (RGB_view). The decoding unit 654A may supply the atlas image (RGB_view) to the atlas inverse transform unit 655. The decoding unit 654B may acquire coded data (coded data (RGB_view)) of the atlas transformation information (RGB_view) supplied from the demultiplexing unit 551. The decoding unit 654B may decode the coded data (RGB_view) to generate the atlas transformation information (RGB_view). Any decoding method may be used. For example, the decoding unit 654B may decode the coded data (RGB_view) in accordance with the MIV / V3C standard. The decoding unit 654B may also supply the atlas transformation information (RGB_view) to the atlas inverse transformation unit 655. The decoding unit 654A and the decoding unit 654B are also referred to as the decoding unit 654.

[0189] The atlas inverse transform unit 655 may, for example, acquire an atlas image (RGB_view) supplied from the decoding unit 654A. The atlas inverse transform unit 655 may also acquire atlas transformation information (RGB_view) supplied from the decoding unit 654B. The atlas inverse transform unit 655 may also perform inverse atlas transformation on the atlas image (RGB_view) using the atlas transformation information (RGB_view) to generate (restore) a reduced RGB_view image. The atlas inverse transform unit 655 may also supply the generated reduced RGB_view image to the enlargement processing unit 656.

[0190] The enlargement processing unit 656 may obtain the reduced RGB_view image supplied from the atlas inverse transformation unit 655. The enlargement processing unit 656 may enlarge the reduced RGB_view image to generate (restore) an RGB_view image. The enlargement processing unit 656 may supply the generated RGB_view image to the unpacking device 317.

[0191] The present technology described above in <3. Encoding and Decoding Using Atlas Transformation> may be applied to the decoding device 316 configured as above. For example, the above-described method 1-2 may be applied to this decoding device 316. That is, the alpha image may be converted to the same size as the RGB image to generate a mask.

[0192] The above-described method 1-2-1 may also be applied to this decoding device 316. For example, the RGB image may include an RGB_base image indicating the distribution of the RGB_base component among the RGB components, and an RGB_view image indicating the distribution of the RGB_view component among the RGB components. The RGB_base image and the RGB_view image may have different sizes. The decoding unit 652A and the decoding unit 654A may decode the coded data of the atlas image of the RGB_base component and the coded data of the atlas image of the RGB_view component, respectively. The atlas inverse transform unit 653 and the atlas inverse transform unit 655 may perform inverse atlas transforms on the atlas image of the RGB_base component and the atlas image of the RGB_view component independently of each other.

[0193] Furthermore, the above-described method 1-2-1-1 may be applied to this decoding device 316. For example, the atlas inverse transform unit 653 may perform inverse atlas transform on an atlas image of the RGB_base component using atlas transform information for the RGB_base component. Furthermore, the atlas inverse transform unit 655 may perform inverse atlas transform on an atlas image of the RGB_view component using atlas transform information for the RGB_view component. The atlas transform information may be information indicating the content of the atlas transform.

[0194] Moreover, the above-described method 1-2-1-3 may be applied to this decoding device 316. For example, the enlargement processing unit 651 may enlarge the reduced alpha image, and the enlargement processing unit 656 may enlarge the reduced RGB_view image.

[0195] With this configuration, the decoding device 316 can suppress an increase in the transmission capacity required for transmitting 3D data that uses MPI, such as NeX. In this case, the atlas inverse transform may also be performed when unpacking the alpha image, RGB_base image, and RGB_view image. In other words, the decoding device 316 and the unpacking device 317 may be integrated.

[0196] <Flow of Decoding Process> An example of the flow of the decoding process executed by the decoding device 316 will be described with reference to the flowchart of FIG.

[0197] When the decoding process starts, in step S651, the demultiplexing unit 551 demultiplexes the bit stream to generate coded data of the atlas image (alpha), coded data of the atlas transformation information (alpha), coded data of the atlas image (RGB_base), coded data of the atlas transformation information (RGB_base), coded data of the atlas image (RGB_view), and coded data of the atlas transformation information (RGB_view).

[0198] In step S652, the decoding unit 552A decodes the coded data of the atlas image of the alpha component to generate an atlas image (alpha). The decoding unit 552B also decodes the coded data of the atlas transformation information of the alpha component to generate atlas transformation information (alpha). In step S653, the atlas inverse transformation unit 553 performs inverse atlas transformation on the atlas image (alpha) using the atlas transformation information (alpha) to generate a reduced alpha image. In step S654, the enlargement processing unit 651 enlarges the reduced alpha image.

[0199] In step S655, the decoding unit 652A decodes the coded data of the atlas image of the RGB_base component to generate an atlas image (RGB_base). Furthermore, the decoding unit 652B decodes the coded data of the atlas transformation information of the RGB_base component to generate atlas transformation information (RGB_base). In step S656, the atlas inverse transformation unit 653 performs an inverse atlas transformation on the atlas image (RGB_base) using the atlas transformation information (RGB_base) to generate an RGB_base image.

[0200] In step S657, the decoding unit 654A decodes the coded data of the atlas image of the RGB_view components to generate an atlas image (RGB_view). Furthermore, the decoding unit 654B decodes the coded data of the atlas transformation information of the RGB_view components to generate atlas transformation information (RGB_view). In step S658, the atlas inverse transformation unit 655 performs inverse atlas transformation on the atlas image (RGB_view) using the atlas transformation information (RGB_view) to generate a reduced RGB_view image. In step S659, the enlargement processing unit 656 enlarges the reduced RGB_view image.

[0201] When the process of step S659 ends, the decoding process ends and the process returns to FIG.

[0202] In this way, in each process, the present technology described above in <3. Encoding / Decoding Using Atlas Transform> may be applied. For example, the above-described method 1-2 may be applied. The above-described method 1-2-1 may also be applied. The above-described method 1-2-1-1 may also be applied. The above-described method 1-2-1-2 may also be applied. The above-described method 1-2-1-3 may also be applied.

[0203] By performing each process in this manner, the decoding device 316 can suppress an increase in the transmission capacity required for transmitting 3D data that uses MPI, such as NeX.

[0204] <Encoding device> Furthermore, the method 1-2-2 described above in <3. Encoding and decoding using atlas transformation> may be applied to the encoding device 315. In that case, an example of the main configuration of the encoding device 315 is shown in FIG.

[0205] As shown in Fig. 29, in this case too, the encoding device 315 basically has the same configuration as in the example of Fig. 25. However, in the example of Fig. 29, the atlas transformation unit 615 does not generate atlas transformation information (RGB_view) (or does not supply the atlas transformation information (RGB_view) to the encoding unit 617B).

[0206] That is, the encoding unit 617B may be omitted as in the example of Fig. 29. In this case, the multiplexing unit 517 may obtain the encoded data (alpha), the atlas transformation information (alpha), the encoded data (RGB_view), the encoded data (RGB_base), and the atlas transformation information (RGB_base), and multiplex them to generate a bitstream.

[0207] The present technology described above in <3. Encoding and Decoding Using Atlas Transform> may be applied to the encoding device 315 configured as described above. For example, Method 1-2-2 described above may be applied to this encoding device 315. For example, the RGB image may include an RGB_base image indicating the distribution of the RGB_base component among the RGB components, and an RGB_view image indicating the distribution of the RGB_view component among the RGB components. The RGB_base image and the RGB_view image may have different sizes. The atlas transformation unit 515 (atlas transformation unit 615 and atlas transformation unit 616) may generate atlas transformation information for the RGB_base component or the RGB_view component by atlas transformation of the RGB_base image and the RGB_view image. The atlas transformation information may be information indicating the content of the atlas transformation.

[0208] Moreover, the above-mentioned method 1-2-2-1 may be applied to this encoding device 315. For example, the atlas conversion information, reduction instructions, reduction ratio, etc. to be used may be defined in a file.

[0209] With this configuration, the encoding device 315 can suppress an increase in the transmission capacity required for transmitting 3D data that uses MPI, such as NeX. In this case, the processing up to the atlas conversion may also be performed when packing the post-processed NeX MPI. In other words, the packing device 314 and the encoding device 315 may be integrated.

[0210] Note that the flow of the encoding process in this case is basically the same as the example in Fig. 26 , except that in step S608, the atlas image (RGB_view) is encoded by the encoding unit 617A, and encoding of the atlas transformation information (RGB_view) is omitted (skip). Then, in step S613, the multiplexing unit 517 multiplexes the encoded data of the atlas image (alpha), the encoded data of the atlas transformation information (alpha), the encoded data of the atlas image (RGB_base), the encoded data of the atlas image (RGB_view), and the encoded data of the atlas transformation information (RGB_base) to generate a bitstream.

[0211] <Decoding device> Furthermore, the method 1-2-2 described above in <3. Encoding and decoding using atlas transformation> may be applied to the decoding device 316. In that case, an example of the main configuration of the decoding device 316 is shown in FIG.

[0212] In this case, the decoding device 316 basically has the same configuration as in the example of Fig. 27. However, in the example of Fig. 30, the decoding device 316 further includes a reduction processing unit 681.

[0213] In this example, the demultiplexing unit 551 demultiplexes the bitstream and supplies the generated atlas transformation information (RGB_base) to the decoding unit 652B and the decoding unit 654B.

[0214] In this case, like the decoding unit 652B, the decoding unit 654B obtains coded data of the atlas transformation information (RGB_base) from the demultiplexing unit 551 and decodes it. Then, the decoding unit 654B supplies the atlas transformation information (RGB_base) generated by this decoding to the reduction processing unit 681. Note that the decoding unit 654B may be omitted, and the atlas transformation information (RGB_base) may be supplied from the decoding unit 652B to the reduction processing unit 681. The reduction processing unit 681 may obtain the atlas transformation information (RGB_base) supplied from the decoding unit 654B (or the decoding unit 652B). Furthermore, the reduction processing unit 681 may reduce (downscale) the atlas transformation information (RGB_base) and supply it to the atlas inverse transformation unit 655.

[0215] The atlas inverse transform unit 655 may use the reduced atlas transformation information (RGB_base) to perform an inverse atlas transform on the atlas image (RGB_view) to generate a reduced RGB_view image.

[0216] The present technology described above in <3. Encoding and Decoding Using Atlas Transform> may be applied to a decoding device 316 configured as described above. For example, Method 1-2-2 described above may be applied to this decoding device 316. For example, the RGB image may include an RGB_base image indicating the distribution of the RGB_base component among the RGB components, and an RGB_view image indicating the distribution of the RGB_view component among the RGB components. The RGB_base image and the RGB_view image may have different sizes. The decoding unit 652A may decode the coded data of the atlas image of the RGB_base component, and the decoding unit 654A may decode the coded data of the atlas image of the RGB_view component. The atlas inverse transform unit 653 may perform an inverse atlas transform on the atlas image of the RGB_base component using the atlas transform information of the RGB_base component, and the atlas inverse transform unit 655 may perform an inverse atlas transform on the atlas image of the RGB_view component using the atlas transform information of the RGB_view component. The reduction processing unit 681 may reduce the atlas transform information of the RGB_base component or the RGB_view component.

[0217] Moreover, the above-mentioned method 1-2-2-1 may be applied to this decoding device 316. For example, the atlas transformation information to be used, the reduction instruction, the reduction rate, etc. may be set according to the file definition.

[0218] With this configuration, the decoding device 316 can suppress an increase in the transmission capacity required for transmitting 3D data that uses MPI, such as NeX. In this case, the atlas inverse transform may also be performed when unpacking the alpha image, RGB_base image, and RGB_view image. In other words, the decoding device 316 and the unpacking device 317 may be integrated.

[0219] <Flow of Decoding Process> An example of the flow of the decoding process executed by the decoding device 316 will be described with reference to the flowchart of FIG.

[0220] When the decoding process starts, in step S681, the demultiplexing unit 551 demultiplexes the bit stream.

[0221] In step S682, the decoding unit 552A decodes the coded data of the atlas image of the alpha component to generate an atlas image (alpha). The decoding unit 552B also decodes the coded data of the atlas transformation information of the alpha component to generate atlas transformation information (alpha). In step S683, the atlas inverse transformation unit 553 performs inverse atlas transformation on the atlas image (alpha) using the atlas transformation information (alpha) to generate a reduced alpha image. In step S684, the enlargement processing unit 651 enlarges the reduced alpha image.

[0222] In step S685, the decoding unit 652A decodes the coded data of the atlas image of the RGB_base component to generate an atlas image (RGB_base). Furthermore, the decoding unit 652B decodes the coded data of the atlas transformation information of the RGB_base component to generate atlas transformation information (RGB_base). In step S686, the atlas inverse transformation unit 653 performs an inverse atlas transformation on the atlas image (RGB_base) using the atlas transformation information (RGB_base) to generate an RGB_base image.

[0223] In step S687, the decoding unit 654A decodes the coded data of the atlas image for the RGB_view component to generate an atlas image (RGB_view). The decoding unit 654B decodes the coded data of the atlas transformation information for the RGB_base component to generate atlas transformation information (RGB_base). In step S688, the reduction processing unit 681 reduces the atlas transformation information (RGB_base) to generate atlas transformation information for the RGB_view component (reduced atlas transformation information (RGB_base)).

[0224] In step S689, the atlas inverse transform unit 655 performs inverse atlas transform on the atlas image (RGB_view) using the atlas transformation information for the RGB_view component (reduced atlas transformation information (RGB_base)), to generate a reduced RGB_view image. In step S690, the enlargement processing unit 656 enlarges the reduced RGB_view image.

[0225] When the process of step S690 ends, the decoding process ends and the process returns to FIG.

[0226] In this way, in each process, the present technology described above in <3. Encoding and decoding using atlas transformation> may be applied. For example, the above-described method 1-2-2 may be applied. Also, the above-described method 1-2-2-1 may be applied.

[0227] By performing each process in this manner, the decoding device 316 can suppress an increase in the transmission capacity required for transmitting 3D data that uses MPI, such as NeX.

[0228] 5. Supplementary Notes Computer The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, the term computer includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0229] FIG. 32 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0230] In a computer 900 shown in FIG. 32, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0231] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0232] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0233] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0234] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.

[0235] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0236] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0237] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0238] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

[0239] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0240] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0241] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0242] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.

[0243] Furthermore, various types of information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, the associated pieces of data may be combined into one piece of data or may be separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to only part of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0244] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0245] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0246] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0247] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0248] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0249] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0250] Furthermore, for example, the multiple technical elements included in the present technology can be implemented independently and individually, as long as no contradiction occurs. Of course, any multiple technical elements among them can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the present technology described above can be implemented in combination with other technologies not described above.

[0251] Note that the present technology can also be configured as follows: (1) An information processing device comprising: a first atlas conversion unit that performs atlas conversion on an alpha image of 3D data and generates an atlas image of an alpha component, a first encoding unit that encodes the atlas image of the alpha component, a second atlas conversion unit that performs atlas conversion on an RGB image of the 3D data and generates an atlas image of the RGB components, and a second encoding unit that encodes the atlas image of the RGB components, wherein the alpha image is distribution information of the alpha component that indicates the presence of an object in a single layer of a layer group that represents a three-dimensional space, the RGB image is information that indicates the distribution of the RGB components that indicate color information of the object in a plurality of consecutive layers of the layer group, the 3D data has the alpha image for each of the single layer and the RGB image for each of the consecutive layers for the three-dimensional space, and the atlas conversion is processing that arranges, in the atlas image, non-zero regions included in the single alpha image or non-zero regions included in the RGB image that correspond to non-zero regions included in the single or multiple alpha images. (2) The information processing device according to (1), further comprising: a mask image generation unit configured to generate a mask image by combining non-zero regions of the alpha images of the consecutive layers, wherein the second atlas conversion unit is configured to perform atlas conversion on the RGB image using the mask image. (3) The information processing device according to (2), further comprising: an RGB_base image indicating a distribution of RGB_base components of the RGB components; and an RGB_view image indicating a distribution of RGB_view components of the RGB components. The information processing device according to (2), further comprising: a mask image generation unit configured to generate a mask image by combining non-zero regions of the alpha images of the consecutive layers, wherein the second atlas conversion unit is configured to perform atlas conversion on the RGB_base image and the RGB_view image, and to generate atlas conversion information common to the RGB_base component and the RGB_view component, wherein the atlas conversion information is information indicating content of the atlas conversion.(4) The information processing device according to (1), wherein the RGB image includes an RGB_base image indicating a distribution of RGB_base components among the RGB components and an RGB_view image indicating a distribution of RGB_view components among the RGB components, and the second atlas conversion unit is configured to perform atlas conversion on the RGB_base image and the RGB_view image, which have different sizes, independently of each other. (5) The information processing device according to (4), wherein the second atlas conversion unit is configured to generate atlas conversion information for the RGB_base components by atlas conversion of the RGB_base image and generate atlas conversion information for the RGB_view components by atlas conversion of the RGB_view image, and the atlas conversion information is information indicating content of the atlas conversion. (6) The information processing device according to (4) or (5), further comprising: a mask image generation unit configured to generate, for each of the RGB_base image and the RGB_view image, a mask image obtained by combining non-zero regions of the alpha images of the consecutive layers, wherein the second atlas conversion unit is configured to atlas-convert the RGB_base image using the mask image for the RGB_base image, and to atlas-convert the RGB_view image using the mask image for the RGB_view image. (7) The information processing device according to any of (4) to (6), further comprising: a downscaling processing unit configured to downsize the alpha image and the RGB_view image, wherein the first atlas conversion unit is configured to atlas-convert the downsized alpha image, and the second atlas conversion unit is configured to atlas-convert the RGB_base image and the downsized RGB_view image independently of each other.(8) The information processing device according to (1), wherein the RGB image includes an RGB_base image indicating a distribution of RGB_base components among the RGB components and an RGB_view image indicating a distribution of RGB_view components among the RGB components, the RGB_base image and the RGB_view image have different sizes, the second atlas transformation unit is configured to generate atlas transformation information of the RGB_base components or the RGB_view components by atlas transformation of the RGB_base image and the RGB_view image, and the atlas transformation information is information indicating the content of the atlas transformation. (9) The information processing device according to any one of (1) to (8), wherein the RGB image includes an RGB_base image indicating a distribution of RGB_base components among the RGB components, and an RGB_view image indicating a distribution of RGB_view components among the RGB components; the second atlas conversion unit is configured to atlas convert the RGB_base image and the RGB_view image to generate respective atlas images; and the information processing device further includes a multiplexing unit that multiplexes coded data of the atlas image of the alpha component, coded data of the atlas image of the RGB_base component, and coded data of the atlas image of the RGB_view component using identification information to generate a bitstream; and the identification information is information for identifying the RGB_base component and the RGB_view component.(10) An information processing method comprising: atlas-transforming an alpha image of 3D data to generate an atlas image of an alpha component; encoding the atlas image of the alpha component; atlas-transforming an RGB image of the 3D data to generate an atlas image of an RGB component; and encoding the atlas image of the RGB components; the alpha image is distribution information of the alpha component indicating the presence of an object in a single layer of a group of layers representing a three-dimensional space; the RGB image is information indicating the distribution of the RGB component indicating color information of the object in a plurality of consecutive layers of the group of layers; the 3D data has the alpha image for each of the single layer and the RGB image for each of the consecutive layers for the three-dimensional space; and the atlas transformation is a process of arranging, in the atlas image, non-zero areas included in the single alpha image or non-zero areas included in the RGB image corresponding to the non-zero areas included in the single or multiple alpha images.

[0252] (11) A data processing system including: a first decoding unit that decodes encoded data and generates an atlas image of an alpha component; a first inverse atlas transformation unit that performs inverse atlas transformation on the atlas image of the alpha component and generates an alpha image of 3D data; a second decoding unit that decodes encoded data and generates an atlas image of RGB components; and a second inverse atlas transformation unit that performs inverse atlas transformation on the atlas image of the RGB components and generates an RGB image of the 3D data, wherein the alpha image is distribution information of the alpha component indicating the presence of an object in a single layer of a layer group representing a three-dimensional space; the RGB image is information indicating the distribution of the RGB components indicating color information of the object in a plurality of consecutive layers of the layer group; the 3D data has the alpha image for each of the single layer and the RGB image for each of the consecutive layers for the three-dimensional space; and the inverse atlas transformation is an inverse process of atlas transformation that arranges, in an atlas image, a non-zero region included in the single alpha image or a non-zero region included in the RGB image corresponding to a non-zero region included in one or more of the alpha images. (12) The information processing device according to (11), wherein the RGB image includes an RGB_base image indicating a distribution of an RGB_base component among the RGB components and an RGB_view image indicating a distribution of an RGB_view component among the RGB components, the second decoding unit is configured to decode encoded data to generate an atlas image of the RGB_base component and an atlas image of the RGB_view component, the second atlas inverse transform unit is configured to perform inverse atlas transform on the atlas image of the RGB_base component and the atlas image of the RGB_view component using atlas transform information common to the RGB_base component and the RGB_view component, and the atlas transform information is information indicating content of the atlas transform.(13) The information processing device according to (11), wherein the RGB image includes an RGB_base image indicating a distribution of an RGB_base component of the RGB components and an RGB_view image indicating a distribution of an RGB_view component of the RGB components, the RGB_base image and the RGB_view image have different sizes, the second decoding unit is configured to decode coded data of the atlas image of the RGB_base component and coded data of the atlas image of the RGB_view component, respectively, and the second atlas inverse transform unit is configured to perform inverse atlas transform on the atlas image of the RGB_base component and the atlas image of the RGB_view component independently of each other. (14) The information processing device according to (13), wherein the second atlas inverse transform unit is configured to perform inverse atlas transform on the atlas image of the RGB_base component using atlas transform information of the RGB_base component and to perform inverse atlas transform on the atlas image of the RGB_view component using atlas transform information of the RGB_view component, and the atlas transform information is information indicating content of the atlas transform. (15) The information processing device according to (13) or (14), further comprising an enlargement processing unit that enlarges the alpha image and the RGB_view image. (16) The RGB image includes an RGB_base image indicating a distribution of an RGB_base component of the RGB components and an RGB_view image indicating a distribution of an RGB_view component of the RGB components, the RGB_base image and the RGB_view image having mutually different sizes, the second decoding unit is configured to decode coded data of an atlas image of the RGB_base component and coded data of an atlas image of the RGB_view component, respectively, and the second atlas inverse transform unit is configured to perform inverse atlas transform on the atlas image of the RGB_base component and the atlas image of the RGB_view component using atlas transform information of the RGB_base component or the RGB_view component. (17) The information processing device according to (16), further comprising a reduction processing unit that reduces the atlas transform information of the RGB_base component or the RGB_view component.(18) The information processing device according to any one of (11) to (17), wherein the RGB image includes an RGB_base image indicating a distribution of an RGB_base component among the RGB components and an RGB_view image indicating a distribution of an RGB_view component among the RGB components, and further comprises a demultiplexing unit that demultiplexes a bitstream to generate coded data of an atlas image of the alpha component, coded data of the atlas image of the RGB_base component, and coded data of an atlas image of the RGB_view component. (19) The information processing device according to (18), wherein the demultiplexing unit demultiplexes the bitstream using identification information for identifying the RGB_base component and the RGB_view component. (20) An information processing method comprising: decoding encoded data to generate an atlas image of an alpha component; performing an inverse atlas transform on the atlas image of the alpha component to generate an alpha image of 3D data; decoding encoded data to generate an atlas image of RGB components; performing an inverse atlas transform on the atlas image of the RGB components to generate an RGB image of the 3D data; the alpha image is distribution information of the alpha component indicating the presence of an object in a single layer of a layer group representing a three-dimensional space; the RGB image is information indicating the distribution of the RGB components indicating color information of the object in a plurality of consecutive layers of the layer group; the 3D data has the alpha image for each of the single layer and the RGB image for each of the consecutive layers for the three-dimensional space; and the inverse atlas transform is an inverse process of atlas transform that arranges, in an atlas image, non-zero areas included in the single alpha image or non-zero areas included in the RGB image corresponding to the non-zero areas included in the single or multiple alpha images.

[0253] 300 Information processing system, 311 NeX learning device, 312 NeX MPI generation device, 313 Post-processing device, 314 Packing device, 315 Encoding device, 316 Decoding device, 317 Unpacking device, 318 Rendering device, 319 Display pre-processing device, 320 Display device, 511 Non-zero region extraction unit, 512 Atlas conversion unit, 513 Encoding unit, 514 Mask image generation unit, 515 Atlas conversion unit, 516 Encoding unit, 517 Multiplexing unit, 551 Demultiplexing unit, 552 Decoding unit, 553 Atlas inverse conversion unit, 554 Decoding unit, 555 Atlas inverse conversion unit, 611 Non-zero region extraction unit, 612 Non-zero region extraction unit, 613 Mask image generation unit, 614 Mask image generation unit, 615 atlas conversion unit, 616 atlas conversion unit, 617 encoding unit, 618 encoding unit, 651 enlargement processing unit, 652 decoding unit, 653 atlas inverse conversion unit, 654 decoding unit, 655 atlas inverse conversion unit, 656 enlargement processing unit, 681 reduction processing unit, 900 computer

Claims

1. An information processing device comprising: a first atlas conversion unit that performs atlas conversion on an alpha image of 3D data and generates an atlas image of an alpha component; a first encoding unit that encodes the atlas image of the alpha component; a second atlas conversion unit that performs atlas conversion on an RGB image of the 3D data and generates an atlas image of an RGB component; and a second encoding unit that encodes the atlas image of the RGB component.

2. The information processing device according to claim 1, further comprising a mask image generation unit that generates a mask image by combining non-zero areas of the alpha images of the multiple consecutive layers, and the second atlas conversion unit is configured to perform atlas conversion on the RGB image using the mask image.

3. The information processing device according to claim 2, wherein the RGB image includes an RGB_base image indicating a distribution of RGB_base components among the RGB components, and an RGB_view image indicating a distribution of RGB_view components among the RGB components; the second atlas transformation unit is configured to perform atlas transformation on the RGB_base image and the RGB_view image to generate respective atlas images, and to generate atlas transformation information common to the RGB_base components and the RGB_view components; and the atlas transformation information is information indicating content of the atlas transformation.

4. The information processing device according to claim 1, wherein the RGB image includes an RGB_base image indicating a distribution of RGB_base components among the RGB components, and an RGB_view image indicating a distribution of RGB_view components among the RGB components, and the second atlas conversion unit is configured to perform atlas conversion on the RGB_base image and the RGB_view image, which have different sizes, independently of each other.

5. The information processing device according to claim 4, wherein the second atlas transformation unit is configured to generate atlas transformation information for the RGB_base component by atlas transformation of the RGB_base image, and to generate atlas transformation information for the RGB_view component by atlas transformation of the RGB_view image, and the atlas transformation information is information indicating the content of the atlas transformation.

6. The information processing device according to claim 4, further comprising a mask image generation unit configured to generate, for each of the RGB_base image and the RGB_view image, a mask image obtained by combining non-zero regions of the alpha images of the consecutive layers, and wherein the second atlas conversion unit is configured to atlas convert the RGB_base image using the mask image for the RGB_base image, and to atlas convert the RGB_view image using the mask image for the RGB_view image.

7. The information processing device according to claim 4, further comprising a reduction processing unit that reduces the alpha image and the RGB_view image, wherein the first atlas conversion unit is configured to perform atlas conversion on the reduced alpha image, and the second atlas conversion unit is configured to perform atlas conversion on the RGB_base image and the reduced RGB_view image independently of each other.

8. The information processing device according to claim 1, wherein the RGB image includes an RGB_base image indicating a distribution of RGB_base components among the RGB components, and an RGB_view image indicating a distribution of RGB_view components among the RGB components, the RGB_base image and the RGB_view image have mutually different sizes, the second atlas transformation unit is configured to generate atlas transformation information of the RGB_base components or the RGB_view components by atlas transformation of the RGB_base image and the RGB_view image, and the atlas transformation information is information indicating content of the atlas transformation.

9. The information processing device according to claim 1, wherein the RGB image includes an RGB_base image indicating a distribution of RGB_base components among the RGB components, and an RGB_view image indicating a distribution of RGB_view components among the RGB components; the second atlas conversion unit is configured to perform atlas conversion on the RGB_base image and the RGB_view image to generate respective atlas images; and the information processing device further comprises a multiplexing unit that generates a bitstream by multiplexing encoded data of the atlas image of the alpha component, encoded data of the atlas image of the RGB_base component, and encoded data of the atlas image of the RGB_view component using identification information, wherein the identification information is information for identifying the RGB_base component and the RGB_view component.

10. An information processing method comprising: atlas-transforming an alpha image of 3D data, generating an atlas image of an alpha component; encoding the atlas image of the alpha component; atlas-transforming an RGB image of the 3D data, generating an atlas image of an RGB component; and encoding the atlas image of the RGB component.

11. An information processing device comprising: a first decoding unit that decodes encoded data and generates an atlas image of an alpha component; a first atlas inverse transformation unit that performs inverse atlas transformation on the atlas image of the alpha component and generates an alpha image of 3D data; a second decoding unit that decodes encoded data and generates an atlas image of RGB components; and a second atlas inverse transformation unit that performs inverse atlas transformation on the atlas image of the RGB components and generates an RGB image of the 3D data.

12. The information processing device according to claim 11, wherein the RGB image includes an RGB_base image indicating a distribution of an RGB_base component among the RGB components, and an RGB_view image indicating a distribution of an RGB_view component among the RGB components; the second decoding unit is configured to decode encoded data to generate an atlas image of the RGB_base component and an atlas image of the RGB_view component; the second atlas inverse transform unit is configured to perform inverse atlas transform on the atlas image of the RGB_base component and the atlas image of the RGB_view component using atlas transformation information common to the RGB_base component and the RGB_view component; and the atlas transformation information is information indicating content of the atlas transformation.

13. The information processing device according to claim 11, wherein the RGB image includes an RGB_base image indicating a distribution of RGB_base components among the RGB components, and an RGB_view image indicating a distribution of RGB_view components among the RGB components, the RGB_base image and the RGB_view image have mutually different sizes, the second decoding unit is configured to respectively decode encoded data of an atlas image of the RGB_base components and encoded data of an atlas image of the RGB_view components, and the second atlas inverse transform unit is configured to perform atlas inverse transform on the atlas image of the RGB_base components and the atlas image of the RGB_view components independently of each other.

14. The information processing device according to claim 13, wherein the second atlas inverse transformation unit is configured to perform an atlas inverse transformation of an atlas image of the RGB_base component using atlas transformation information of the RGB_base component, and to perform an atlas inverse transformation of an atlas image of the RGB_view component using atlas transformation information of the RGB_view component, and the atlas transformation information is information indicating the content of the atlas transformation.

15. An information processing method comprising: decoding encoded data, generating an atlas image of an alpha component, performing an inverse atlas transform on the atlas image of the alpha component, and generating an alpha image of 3D data; decoding encoded data, generating an atlas image of RGB components, performing an inverse atlas transform on the atlas image of the RGB components, and generating an RGB image of the 3D data.

Images