Image processing device and method

Abstract

The present disclosure relates to an image processing device and method for suppressing a reduction in image quality. The image processing device and method generate: a base video frame to which a base patch is disposed, the base patch being obtained by projecting a point cloud, which represents an object having a three-dimensional shape as a point set, onto a two-dimensional plane for each partial area; and an additional video frame to which an additional patch is disposed, the additional patch being obtained by projecting a partial area including at least a portion of a partial area corresponding to the base patch of the point cloud into the same two-dimensional plane as that of the base patch with at least a portion of parameters changed to that of the base patch. In addition, the image processing device and method encode the generated base video frame and additional video frame to generate the encoded data. The present disclosure may be applied, for example, to an image processing device, an electronic apparatus, an image processing method, a program, or the like.

Description

[Technical field]

[0001] The present disclosure relates to an image processing device and method, and more particularly to an image processing device and method capable of suppressing degradation of image quality. [Background technology]

[0002] Conventionally, the Moving Picture Experts Group (MPEG) has been working on standardization of encoding and decoding of point cloud data, which represents a three-dimensional object as a set of points (for example, see Non-Patent Document 1).

[0003] In addition, a method has been proposed in which the geometry data and attribute data of a point cloud are projected onto a two-dimensional plane for each small region, the image (patch) projected onto the two-dimensional plane is placed within a frame image, and the frame image is encoded using an encoding method for two-dimensional images (hereinafter also referred to as a video-based approach) (see, for example, Non-Patent Documents 2 to 4). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "Information technology - MPEG-I (Coded Representation of Immersive Media) - Part 9: Geometry-based Point Cloud Compression", ISO / IEC 23090-9:2019(E) [Non-Patent Document 2] Tim Golla and Reinhard Klein, "Real-time Point Cloud Compression," IEEE, 2015 [Non-Patent Document 3] K. Mammou, “Video-based and Hierarchical Approaches Point Cloud Compression”, MPEG m41649, Oct. 2017 [Non-Patent Document 4] K. Mammou,“PCC Test Model Category 2 v0,” N17248 MPEG output document, October 2017 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the case of the video-based approach described in Non-Patent Documents 2 to 4, the accuracy of information is set uniformly for all patches. In other words, the accuracy of information cannot be changed locally. Therefore, compared with the case where the accuracy of information can be changed locally, there is a risk that the quality of the point cloud for the same amount of information may be reduced. Therefore, there is a risk that the subjective image quality of the display image in which the point cloud reconstructed by decoding the encoded data generated by this video-based approach is projected onto a two-dimensional plane may be reduced.

[0006] The present disclosure has been made in consideration of such circumstances, and makes it possible to suppress degradation in image quality of a two-dimensional image for display of 3D data. [Means for solving the problem]

[0007] An image processing device according to one aspect of the present technology is an image processing device including: a video frame generation unit that generates a base video frame in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each partial region; and an additional video frame in which an additional patch is arranged by projecting a partial region including at least a part of the partial region corresponding to the base patch of the point cloud onto the same two-dimensional plane as in the case of the base patch, with at least some parameters changed from those of the base patch; and an encoding unit that encodes the base video frame and the additional video frame generated by the video frame generation unit, and generates encoded data.

[0008] An image processing method according to one aspect of the present technology is an image processing method that generates a base video frame in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each partial region, and an additional video frame in which an additional patch is arranged by projecting a partial region including at least a part of the partial region of the point cloud corresponding to the base patch onto the same two-dimensional plane as the base patch, with at least some parameters changed from those of the base patch, and encodes the generated base video frame and the additional video frame to generate encoded data.

[0009] An image processing device according to another aspect of the present technology is an image processing device including: a decoding unit that decodes encoded data and generates a base video frame in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each partial region; and an additional video frame in which an additional patch is arranged by projecting a partial region including at least a part of the partial region corresponding to the base patch of the point cloud onto the same two-dimensional plane as in the case of the base patch, with at least some parameters changed from those of the base patch; and a reconstruction unit that reconstructs the point cloud using the base video frame generated by the decoding unit and the additional video frame.

[0010] An image processing method according to another aspect of the present technology is an image processing method that decodes encoded data, generates a base video frame in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each partial region, and an additional video frame in which an additional patch is arranged by projecting a partial region including at least a part of the partial region of the point cloud corresponding to the base patch onto the same two-dimensional plane as the base patch, with at least some parameters changed from those of the base patch, and reconstructs the point cloud using the generated base video frame and the additional video frame.

[0015] In an image processing device and method according to one aspect of the present technology, a base video frame is generated in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each partial region, and an additional video frame is generated in which an additional patch is arranged by projecting a partial region including at least a part of the partial region corresponding to the base patch of the point cloud onto the same two-dimensional plane as the base patch, with at least some parameters changed from those of the base patch, and the generated base video frame and additional video frame are encoded to generate encoded data.

[0016] In an image processing device and method according to another aspect of the present technology, encoded data is decoded, and a base video frame is generated in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each subregion, and an additional video frame is generated in which an additional patch is arranged by projecting a subregion including at least a portion of the subregion corresponding to the base patch of the point cloud onto the same two-dimensional plane as the base patch, with at least some parameters changed from those of the base patch, and a point cloud is reconstructed using the generated base video frame and the additional video frame. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating data of a video-based approach. [Diagram 2] FIG. 13 is a diagram illustrating transmission of an additional patch. [Diagram 3] FIG. 13 is a diagram illustrating an additional patch. [Figure 4] FIG. 1 is a diagram for explaining the target and manner of action of each method. [Diagram 5] FIG. 13 is a diagram illustrating generation of an additional patch. [Figure 6] 13A and 13B are diagrams illustrating an example of the operation of an additional patch. [Figure 7] 13A and 13B are diagrams illustrating an example of the operation of an additional patch. [Figure 8] FIG. 4 is a diagram illustrating an example of a patch configuration. [Figure 9] FIG. 2 is a block diagram showing an example of the main configuration of an encoding device. [Figure 10] 13 is a block diagram showing an example of a main configuration of a packing encoding unit. FIG. [Figure 11] 11 is a flowchart illustrating an example of the flow of an encoding process. [Figure 12] 13 is a flowchart illustrating an example of the flow of a packing encoding process. [Figure 13] FIG. 2 is a block diagram showing an example of the main configuration of a decoding device. [Figure 14] FIG. 2 is a block diagram showing an example of the main configuration of a 3D reconstruction unit. [Figure 15] 13 is a flowchart illustrating an example of the flow of a decoding process. [Figure 16] 13 is a flowchart illustrating an example of the flow of a 3D reconstruction process. [Figure 17] FIG. 13 is a diagram illustrating generation of an additional patch. [Figure 18] 13 is a block diagram showing an example of a main configuration of a packing encoding unit. FIG. [Figure 19] 13 is a flowchart illustrating an example of the flow of a packing encoding process. [Figure 20] FIG. 2 is a block diagram showing an example of the main configuration of a 3D reconstruction unit. [Figure 21] 13 is a flowchart illustrating an example of the flow of a 3D reconstruction process. [Figure 22] 13 is a flowchart illustrating an example of the flow of a packing encoding process. [Diagram 23] 13 is a flowchart illustrating an example of the flow of a 3D reconstruction process. [Figure 24] 13 is a block diagram showing an example of a main configuration of a packing encoding unit. FIG. [Diagram 25] 13 is a flowchart illustrating an example of the flow of a packing encoding process. [Figure 26] FIG. 2 is a block diagram showing an example of the main configuration of a 3D reconstruction unit. [Figure 27] 13 is a flowchart illustrating an example of the flow of a 3D reconstruction process. [Figure 28] 13 is a diagram illustrating a configuration of auxiliary patch information. FIG. [Figure 29] 11A and 11B are diagrams illustrating information indicating an action target of an additional patch. [Diagram 30] 11A and 11B are diagrams illustrating information indicating processing content using an additional patch. [Diagram 31] 13A and 13B are diagrams illustrating information regarding alignment of additional patches. [Diagram 32] 11 is a diagram illustrating size setting information of an additional occupancy map. FIG. [Diagram 33] FIG. 11 is a diagram illustrating transmission information for each method. [Diagram 34] FIG. 1 is a block diagram showing an example of a main configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1.Transmission of additional patches 2. First embodiment (Method 1) 3. Second embodiment (Method 2) 4. Third embodiment (Method 3) 5. Fourth embodiment (method 4) 6. Fifth embodiment (Method 5) 7. Notes

[0021] <1. Transmission of additional patches> <References supporting technical content and technical terms> The scope of disclosure of the present technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent documents that were publicly known at the time of filing the application, and the contents of other documents referenced in the following non-patent documents.

[0022] Non-patent document 1: (mentioned above) Non-patent document 2: (mentioned above) Non-patent document 3: (mentioned above) Non-patent document 4: (mentioned above)

[0023] In other words, the contents described in the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0024] <Point cloud> Previously, there was 3D data such as point clouds that represented three-dimensional structures based on point position information and attribute information.

[0025] For example, in the case of a point cloud, a three-dimensional structure (an object with a three-dimensional shape) is represented as a collection of many points. Point cloud data (also referred to as point cloud data) is composed of position information (also referred to as geometry data) and attribute information (also referred to as attribute data) of each point. The attribute data can include any information. For example, color information, reflectance information, normal information, etc. of each point may be included in the attribute data. In this way, point cloud data has a relatively simple data structure, and by using a sufficiently large number of points, any three-dimensional structure can be represented with sufficient accuracy.

[0026] <Quantization of position information using voxels> Since such point cloud data has a relatively large amount of data, an encoding method using voxels was devised to compress the amount of data by encoding, etc. A voxel is a three-dimensional region for quantizing geometry data (position information).

[0027] That is, a three-dimensional region (also called a bounding box) containing a point cloud is divided into small three-dimensional regions called voxels, and each voxel indicates whether it contains a point or not. In this way, the position of each point is quantized in voxel units. Therefore, by converting point cloud data into such voxel data (also called voxel data), it is possible to suppress an increase in the amount of information (typically, to reduce the amount of information).

[0028] <Outline of the video-based approach> In the video-based approach, such geometry data and attribute data of the point cloud are projected onto a two-dimensional plane for each small region (connection component). The image in which this geometry data and attribute data are projected onto the two-dimensional plane is also called a projected image. Moreover, the projected image for each small region is called a patch. For example, in the projected image (patch) of the geometry data, the position information of the points is expressed as position information (depth value) in the vertical direction (depth direction) to the projection plane.

[0029] Then, each patch generated in this way is arranged in a frame image. A frame image in which patches of geometry data are arranged is also called a geometry video frame. A frame image in which patches of attribute data are arranged is also called a color video frame. For example, each pixel value of the geometry video frame indicates the above-mentioned depth value.

[0030] These video frames are then encoded using a two-dimensional image encoding method such as AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding). In other words, point cloud data, which is 3D data representing a three-dimensional structure, can be encoded using a codec for two-dimensional images.

[0031] <Occupancy Map> In addition, in such a video-based approach, an occupancy map can also be used. The occupancy map is map information that indicates the presence or absence of a projected image (patch) for each NxN pixel of a geometry video frame. For example, the occupancy map indicates the area (NxN pixels) where a patch exists in a geometry video frame or a color video frame with a value of "1", and indicates the area (NxN pixels) where a patch does not exist with a value of "0".

[0032] Such an occupancy map is coded as data separate from the geometry video frame and the color video frame, and is transmitted to the decoding side. By referring to this occupancy map, the decoder can determine whether or not a patch exists in the area, so that the effects of noise and the like caused by coding and decoding can be suppressed, and 3D data can be restored more accurately. For example, even if the depth value changes due to coding and decoding, the decoder can ignore the depth value of the area where the patch does not exist (so as not to process it as the position information of the 3D data) by referring to the occupancy map.

[0033] The occupancy map can also be transmitted as a video frame, similar to a geometry video frame or a color video frame.

[0034] In other words, in the case of the video-based approach, a geometry video frame 11 in which patches 11A of geometry data are arranged, a color video frame 12 in which patches 12A of attribute data are arranged, and an occupancy map 13 in which patches 13A of occupancy map are arranged, as shown in A of Figure 1, are transmitted.

[0035] <Support patch information> Furthermore, in the case of the video-based approach, information about the patch (also referred to as auxiliary patch information) is transmitted as metadata. An example of this auxiliary patch information is shown in auxiliary patch information 14 in FIG. 1B. The auxiliary patch information 14 includes information about each patch. For example, as shown in FIG. 1B, the information includes patch identification information (patchIndex), the position (u0, v0) of the patch on a 2D projection plane (a two-dimensional plane onto which a connection component (small region) of a point cloud is projected), the position (u, v, d) of the projection plane in three-dimensional space, the width (width), the height (Height) of the patch, the projection direction (Axis) of the patch, and the like.

[0036] <Video Images> In the following, it is assumed that (objects in) a point cloud can change in the time direction, like a two-dimensional video. In other words, the geometry data and attribute data have a concept of time direction, and are data sampled at predetermined time intervals, like a two-dimensional video. It is noted that, like a video frame of a two-dimensional image, data at each sampling time is called a frame. In other words, the point cloud data (geometry data and attribute data) is assumed to be composed of multiple frames, like a two-dimensional video.

[0037] <Video-based approach to quality reduction> However, in the case of this video-based approach, there is a risk of point loss due to projection of the point cloud (small region) or smoothing processing. For example, if the projection direction is at an unfavorable angle with respect to the small region 3D shape, the projection may cause point loss. In addition, point loss may occur due to changes in the shape of the patch caused by smoothing processing, etc. Therefore, there is a risk of a decrease in the subjective image quality of a display image obtained by projecting 3D data reconstructed by decoding the encoded data generated by this video-based approach onto a 2D plane.

[0038] However, in the case of the video-based approaches described in Non-Patent Documents 2 to 4, the accuracy of information is set uniformly for all patches. Therefore, for example, when trying to improve the accuracy of a part of a patch, it is necessary to improve the accuracy of the entire patch for all patches, which unnecessarily increases the amount of information and may reduce the coding efficiency.

[0039] In other words, since the accuracy of information cannot be changed locally, the quality of the point cloud may be reduced for the same amount of information compared to the case where the accuracy of information can be changed locally. Therefore, the subjective image quality of the display image in which the point cloud reconstructed by decoding the encoded data generated by such a video-based approach is projected onto a two-dimensional plane may be reduced.

[0040] For example, if the accuracy of the occupancy map is low, burrs may occur at the boundaries of the patches, which may reduce the quality of the reconstructed point cloud. Although it is possible to improve the accuracy to suppress the occurrence of these burrs, in that case, it is difficult to control the accuracy locally, so it is necessary to improve the accuracy of the entire occupancy map. As a result, there is a risk that the amount of information will increase unnecessarily and the coding efficiency will decrease.

[0041] As a method for reducing such burrs, that is, as a method for suppressing deterioration in the quality of the reconstructed point cloud, it has been considered to perform a smoothing process on the geometry data. However, this smoothing process requires a large amount of processing, which may increase the load. In addition, searching for the parts where the smoothing process should be performed also requires a large amount of processing, which may increase the load.

[0042] In addition, since it is difficult to locally control the accuracy of information, it is necessary to reconstruct distant objects and near objects with the same accuracy (resolution) with respect to the viewpoint position. For example, when matching the accuracy (resolution) of distant objects to that of near objects, there is a risk that the amount of information of distant objects will increase unnecessarily. Conversely, when matching the accuracy (resolution) of near objects to that of distant objects, there is a risk that the quality of near objects will decrease, resulting in a decrease in the subjective image quality of the display image.

[0043] Furthermore, it was difficult to locally control the quality of the reconstructed point cloud based on, for example, user authority (locally control the subjective image quality of the display image). For example, it was difficult to provide the entire point cloud with the original quality (high resolution) to users who paid a high usage fee or users with administrator authority, but provide a portion of it with lower quality (low resolution) to users who paid a low usage fee or users with guest authority (i.e., provide it in a state where some areas of the 2D image are pixelated). This made it difficult to realize a variety of services.

[0044] <Transmission of additional patches> Therefore, in the above-mentioned video-based approach, an additional patch is transmitted as shown in Table 20 of Fig. 2. The patch in the video-based approach described in Non-Patent Documents 2 to 4 is called a base patch. This base patch is a patch that is always used to reconstruct a partial region of a point cloud that includes a small region corresponding to the base patch.

[0045] In contrast, patches other than the base patch are called additional patches. These additional patches are optional patches and are not essential for reconstructing a subregion of the point cloud that includes the small region corresponding to the additional patch. In other words, the point cloud can be reconstructed using only the base patch, or both the base patch and the additional patch.

[0046] That is, as shown in Fig. 3, a base patch 30 and an additional patch 40 are transmitted. The base patch 30 is composed of a patch 31A of geometry data arranged in a geometry video frame 31, a patch 32A of attribute data arranged in a color video frame 32, and a patch 33A of an occupancy map arranged in an occupancy map 33, as in the case of Fig. 1.

[0047] Similarly, the additional patch 40 may be composed of a patch 41A of geometry data, a patch 42A of attribute data, and a patch 43A of an occupancy map, but some of these may be omitted. For example, the additional patch 40 may be composed of any of the patch 41A of geometry data, the patch 42A of attribute data, and the patch 43A of an occupancy map, or any of the patch 41A of geometry data, the patch 42A of attribute data, and the patch 43A of an occupancy map may be omitted. Note that the small area of ​​the point cloud corresponding to the additional patch 40 is arbitrary, and may include at least a part of the small area of ​​the point cloud corresponding to the base patch 30, or may include an area other than the small area of ​​the point cloud corresponding to the base patch 30. Of course, the small area corresponding to the additional patch 40 may completely match the small area corresponding to the base patch 30, or may not overlap with the small area corresponding to the base patch 30.

[0048] Although the base patch 30 and the additional patch 40 can be arranged in the same video frame, in the following description, for convenience of explanation, it is assumed that the base patch 30 and the additional patch 40 are arranged in different video frames. Also, the video frame in which the additional patch is arranged is also referred to as an additional video frame. For example, the additional video frame in which the patch 41A is arranged is also referred to as an additional geometry video frame 41. Also, the additional video frame in which the patch 42A is arranged is also referred to as an additional color video frame 42. Furthermore, the additional video frame (occupancy map) in which the patch 43A is arranged is also referred to as an additional occupancy map 43.

[0049] The additional patch may be used to update the information of the base patch, or in other words, the additional patch may be composed of information used to update the information of the base patch.

[0050] For example, as in "Method 1" shown in Table 20 of FIG. 2, the additional patch may be used for local control (partial control) of the accuracy of the information of the base patch. In other words, the additional patch may be composed of information used for local control of the accuracy of the information of the base patch. For example, an additional patch composed of information with higher accuracy than the base patch may be transmitted together with the base patch, and the information of the base patch may be updated using the additional patch on the receiving side, thereby locally improving the accuracy of the information of the base patch. In this way, the quality of the point cloud reconstructed using the base patch with its updated information may be locally improved.

[0051] The parameter for controlling the accuracy is arbitrary, and may be, for example, a resolution or a bit depth. As shown in "Method 1-1" in Table 20 of FIG. 2, the additional patch may be a patch of an occupancy map. That is, the additional video frame may be an additional occupancy map. As shown in "Method 1-2" in Table 20 of FIG. 2, the additional patch may be a patch of geometry data. That is, the additional video frame may be an additional geometry video frame. As shown in "Method 1-3" in Table 20 of FIG. 2, the additional patch may be a patch of attribute data. That is, the additional video frame may be an additional color video frame. These "Method 1-1" to "Method 1-3" may be applied in any combination.

[0052] Also, for example, as in "Method 2" shown in Table 20 of FIG. 2, the additional patch may be used as a substitute for smoothing. In other words, the additional patch may be configured with information corresponding to the result of smoothing. For example, such an additional patch may be transmitted together with a base patch, and the additional patch may be used on the receiving side to update the information of the base patch, thereby obtaining the base patch after smoothing. In this way, it is possible to reconstruct a point cloud reflecting the smoothing process without performing the smoothing process on the receiving side. In other words, it is possible to suppress an increase in the load due to the smoothing process.

[0053] Furthermore, for example, as in "Method 3" shown in Table 20 of FIG. 2, this additional patch may be used to specify the range of processing to be performed on the base patch. In other words, the additional patch may be configured with information specifying the range of processing to be performed on the base patch. The content of this processing is arbitrary. For example, the range of smoothing processing may be specified by this additional patch. For example, such an additional patch and base patch may be transmitted, and smoothing processing may be performed on the receiving side for the range of the base patch specified by the additional patch. In this way, it becomes unnecessary to search for the area to be subjected to smoothing processing, and an increase in the load can be suppressed.

[0054] In each of the above "Method 1" to "Method 3," the additional patch differs from the base patch in at least some parameters, such as the accuracy of information, the corresponding small area, etc. Also, the additional patch may be configured by geometry data or attribute data projected onto the same projection surface as the projection surface of the base patch, or an occupancy map corresponding to them.

[0055] Also, for example, as in "Method 4" shown in Table 20 of FIG. 2, the additional patch may be used to reconstruct the point cloud in the same way as the base patch. In other words, the additional patch may be composed of information used to reconstruct the point cloud in the same way as the base patch. For example, such an additional patch may be transmitted together with the base patch, and the receiving side may select whether to reconstruct the point cloud using only the base patch or to reconstruct the point cloud using the base patch and the additional patch. In this way, the quality of the point cloud can be controlled according to various conditions. In this case, the attribute data may be omitted from the additional patch. That is, the additional patch may be composed of a patch of geometry data and a patch of an occupancy map. That is, the additional video frame may be composed of a geometry video frame and an occupancy map.

[0056] Furthermore, for example, as in "Method 5" shown in Table 20 of FIG. 2, information about the additional patch may be transmitted as auxiliary patch information. By referring to this information, the receiving side can grasp the characteristics of the additional patch more accurately. The content of this information about the additional patch is arbitrary. For example, flag information indicating whether it is an additional patch may be transmitted as auxiliary patch information as information about the additional patch. By referring to this flag information, the receiving side can more easily distinguish between an additional patch and a base patch.

[0057] This "Method 5" can be applied in combination with each of the above-mentioned "Method 1" to "Method 4". In the case of each of "Method 1" to "Method 3", the information about the base patch included in the supplemental patch information may also be applied to the additional patch. In that case, the information about the additional patch may be omitted.

[0058] <Effects of additional patches> Table 50 shown in FIG. 4 summarizes the target and the way in which each of the above-mentioned methods works. For example, in the case of "method 1-1" in which an additional patch is used to locally improve the accuracy (resolution) of the occupancy map, the additional patch is a patch of the occupancy map, and acts on a base patch of the occupancy map with a pixel (resolution) coarser than that of the additional patch. For example, the information of the base patch is updated by performing a logical operation (e.g., logical sum (OR) or logical product (AND)) with the additional patch for each bit. For example, the area indicated by the additional patch is added to the area indicated by the base patch, or the area indicated by the additional patch is deleted from the area indicated by the base patch. In other words, this logical operation can locally improve the accuracy (resolution) of the occupancy map.

[0059] In the case of "Method 1-2" in which an additional patch is used to locally improve the precision (resolution) of geometry data, the additional patch is a patch of geometry data, and acts on a base patch of the geometry data having a coarser value (bit depth) than the additional patch. For example, the information of the base patch is updated by adding the value of the base patch to the value of the additional patch, subtracting the value of the additional patch from the value of the base patch, or replacing the value of the base patch with the value of the additional patch. In other words, such calculations and replacements can locally improve the precision (bit depth) of the geometry data.

[0060] Furthermore, in the case of "Method 1-3" in which an additional patch is used to locally improve the precision (resolution) of the attribute data, the additional patch is a patch of attribute data, and acts on a base patch of attribute data having a coarser value (bit depth) than the additional patch. For example, the information of the base patch is updated by adding the value of the base patch to the value of the additional patch, subtracting the value of the additional patch from the value of the base patch, or replacing the value of the base patch with the value of the additional patch. In other words, such calculations and replacements can locally improve the precision (bit depth) of the attribute data.

[0061] In the case of "Method 2" in which the smoothing process result is obtained using an additional patch, the additional patch is a patch of the occupancy map, and acts on a base patch of the occupancy map with the same pixels (resolution) as the additional patch, or on a base patch of the occupancy map with a coarser pixel (resolution) than the additional patch. For example, the information of the base patch is updated by performing a bit-by-bit logical operation (e.g., logical sum (OR) or logical product (AND)) with the additional patch. For example, a base patch that has been subjected to smoothing process is obtained by adding the area indicated by the additional patch to the area indicated by the base patch, or deleting the area indicated by the additional patch from the area indicated by the base patch. This makes it possible to suppress an increase in the load.

[0062] Furthermore, in the case of "Method 3" where the processing range is specified using an additional patch, the additional patch is a patch of the occupancy map, and acts on a base patch of the occupancy map with the same pixels (resolution) as the additional patch, or on a base patch of the occupancy map with a coarser pixel (resolution) than the additional patch. For example, the additional patch sets a flag in the processing range (e.g., the smoothing processing range), and the smoothing processing is performed on the range of the base patch indicated by the additional patch. This makes it possible to suppress an increase in the load.

[0063] In the case of "Method 4" where the point cloud is reconstructed using an additional patch, the additional patch is a patch used for reconstruction of the point cloud, similar to the base patch, and acts on the point cloud reconstructed using the base patch. For example, the additional patch is composed of an occupancy map patch and a geometry data patch, and in order to reconstruct the attribute data, a recolor process is performed using the point cloud reconstructed using the base patch.

[0064] <2. First embodiment (method 1)> <Method 1-1> In this embodiment, the above-mentioned "Method 1" will be described. First, "Method 1-1" will be described. In the case of this "Method 1-1", patches of occupancy maps with multiple types of accuracy are generated from patches of geometry data.

[0065] For example, from a patch of geometry data as shown in A of FIG. 5, a patch of a low-precision occupancy map as shown in B of FIG. 5 is generated. This patch is designated as a base patch. By doing so, it is possible to improve the coding efficiency. However, in this case, the accuracy of the range of the geometry data indicated by the occupancy map decreases. If this base patch is expressed with the accuracy of the patch of geometry data, it will become as shown in C of FIG. 5.

[0066] Incidentally, if an occupancy map is generated from the patch of geometry data shown in A of Fig. 5 with the same accuracy (same resolution) as the geometry data, it will look like D of Fig. 5. In this case, the occupancy map can more accurately represent the range of the geometry data, but the amount of information in the occupancy map increases.

[0067] Therefore, the difference between the patch shown in D of FIG. 5 and the base patch shown in C of FIG. 5 is derived (E of FIG. 5), and this is set as an additional patch. That is, the base patch shown in B of FIG. 5 and the additional patch shown in E of FIG. 5 are transmitted. On the receiving side, a patch such as that shown in D of FIG. 5 can be obtained from these patches. That is, the accuracy of the base patch can be improved. That is, by transmitting the additional patch, the accuracy of the point cloud can be locally improved.

[0068] This difference (area indicated by the additional patch) may be an area to be deleted from the area indicated by the base patch, or an area to be added to the area indicated by the base patch. When the additional patch indicates an area to be deleted from the area indicated by the base patch, for example, as shown in Fig. 6, a bitwise logical AND is performed between an occupancy map 71 of the base patch and an occupancy map 72 of the additional patch to derive an area obtained by deleting the area indicated by the additional patch from the area indicated by the base patch. When the additional patch indicates an area to be added to the area indicated by the base patch, for example, as shown in Fig. 7, a bitwise logical OR is performed between an occupancy map 81 of the base patch and an occupancy map 82 of the additional patch to derive an area obtained by adding the area indicated by the additional patch to the area indicated by the base patch.

[0069] For example, when there are bits of "0" or "1" locally in the occupancy map 91 as shown in A of FIG. 8, an occupancy map in which all bits are "1" or an occupancy map 92 in which all bits are "0" may be used as the occupancy map of the base patch as shown in B of FIG. 8. Then, an occupancy map 93 (B of FIG. 8) of bits with locally different values ​​in the occupancy map 91 may be used as the occupancy map of the additional patch. In this case, the occupancy map 92 (B of FIG. 8) of the base patch may be known also on the receiving side, and the transmission of the occupancy map may be omitted. That is, only the occupancy map 93 shown in B of FIG. 8 may be transmitted. In this way, the increase in the amount of code of the occupancy map can be suppressed.

[0070] <Encoding device> Next, an encoding device that performs such "Method 1-1" will be described. FIG. 9 is a block diagram showing an example of the configuration of an encoding device to which the present technology is applied. The encoding device 100 shown in FIG. 9 is a device that projects 3D data such as a point cloud onto a two-dimensional plane and performs encoding using an encoding method for two-dimensional images (an encoding device that applies a video-based approach). The encoding device 100 performs such processing by applying "Method 1-1" shown in Table 20 of FIG. 2.

[0071] Note that Fig. 9 shows the main processing units, data flows, etc., and is not limited to all that is shown in Fig. 9. In other words, in encoding device 100, there may be processing units that are not shown as blocks in Fig. 9, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 9.

[0072] As shown in FIG. 9, the encoding device 100 includes a patch decomposition unit 101, a packing encoding unit 102, and a multiplexer 103.

[0073] The patch decomposition unit 101 performs processing related to decomposition of 3D data. For example, the patch decomposition unit 101 acquires 3D data (e.g., a point cloud) representing a three-dimensional structure that is input to the encoding device 100. The patch decomposition unit 101 also decomposes the acquired 3D data into a plurality of small regions (connection components), projects the 3D data for each small region onto a two-dimensional plane, and generates patches of geometry data and patches of attribute data.

[0074] The patch decomposition unit 101 also generates an occupancy map corresponding to the generated patches. At this time, the patch decomposition unit 101 applies the above-mentioned "Method 1-1" to generate a base patch and an additional patch of the occupancy map. In other words, the patch decomposition unit 101 generates an additional patch that locally improves the accuracy (resolution) of the base patch of the occupancy map.

[0075] The patch decomposition unit 101 supplies each of the generated patches (base patches of geometry data and attribute data, and base patches and additional patches of an occupancy map) to a packing encoding unit 102.

[0076] The packing coding unit 102 performs processing related to packing and coding of data. For example, the packing coding unit 102 acquires base patches and additional patches supplied from the patch decomposition unit 101, and arranges each patch in a two-dimensional image to pack it as a video frame. For example, the packing coding unit 102 packs base patches of geometry data as video frames to generate geometry video frames (Geometry video frame(s)). In addition, the packing coding unit 102 packs base patches of attribute data as video frames to generate color video frames (Color video frame(s)). Furthermore, the packing coding unit 102 generates an occupancy map in which the base patches are arranged and an additional occupancy map in which the additional patches are arranged, corresponding to those video frames.

[0077] Furthermore, the packing encoding unit 102 encodes each of the generated video frames (geometry video frame, color video frame, occupancy map, and additional occupancy map) to generate encoded data.

[0078] Furthermore, the packing coding unit 102 generates auxiliary patch information, which is information about the patch, and encodes (compresses) the information to generate encoded data. The packing coding unit 102 supplies the generated encoded data to the multiplexer 103.

[0079] The multiplexer 103 performs processing related to multiplexing. For example, the multiplexer 103 obtains various types of encoded data supplied from the packing encoding unit 102, and multiplexes the data to generate a bit stream. The multiplexer 103 outputs the generated bit stream to the outside of the encoding device 100.

[0080] <Packing coding part> Fig. 10 is a block diagram showing a main configuration example of the packing encoding unit 102. Note that Fig. 10 shows main processing units, data flows, etc., and is not limited to all that are shown in Fig. 10. In other words, in the packing encoding unit 102, there may be processing units not shown as blocks in Fig. 10, and there may be processing and data flows not shown as arrows, etc. in Fig. 10.

[0081] As shown in FIG. 10, the packing encoding unit 102 includes an occupancy map generation unit 121, a geometry video frame generation unit 122, an OMap encoding unit 123, a video encoding unit 124, a geometry video frame decoding unit 125, a geometry data reconstruction unit 126, a geometry smoothing processing unit 127, a color video frame generation unit 128, a video encoding unit 129, an auxiliary patch information generation unit 130, and an auxiliary patch information encoding unit 131.

[0082] The occupancy map generation unit 121 generates an occupancy map corresponding to a video frame, in which the base patches supplied from the patch decomposition unit 111 are arranged. Similarly, the occupancy map generation unit 121 generates an additional occupancy map corresponding to an additional video frame, in which the additional patches supplied from the patch decomposition unit 111 are arranged.

[0083] The occupancy map generation unit 121 supplies the generated occupancy map and additional occupancy map to the OMap encoding unit 123. In addition, the occupancy map generation unit 121 supplies the generated occupancy map to the geometry video frame generation unit 122. Furthermore, the occupancy map generation unit 121 supplies information on the base patch and the additional patch to the auxiliary patch information generation unit 130.

[0084] The geometry video frame generation unit 122 generates a geometry video frame, which is a video frame in which base patches of the geometry data supplied from the patch decomposition unit 111 are arranged. The geometry video frame generation unit 122 supplies the generated geometry video frame to the video encoding unit 124.

[0085] The OMap encoder 123 encodes the occupancy map supplied from the occupancy map generator 121 using a coding method for two-dimensional images to generate encoded data. The OMap encoder 123 also encodes the additional occupancy map supplied from the occupancy map generator 121 using a coding method for two-dimensional images to generate encoded data. The OMap encoder 123 supplies the encoded data to the multiplexer 103.

[0086] The video encoding unit 124 encodes the geometry video frame supplied from the geometry video frame generation unit 122 using a coding method for two-dimensional images to generate encoded data. The video encoding unit 124 supplies the generated encoded data to the multiplexer 103. The video encoding unit 124 also supplies the generated encoded data to the geometry video frame decoding unit 125.

[0087] The geometry video frame decoding unit 125 decodes the coded data supplied from the video coding unit 124 by a decoding method for two-dimensional images corresponding to the coding method applied by the video coding unit 124, and generates (reconstructs) a geometry video frame. The geometry video frame decoding unit 125 supplies the generated (reconstructed) geometry video frame to the geometry data reconstruction unit 126.

[0088] The geometry data reconstruction unit 126 extracts base patches of geometry data from the geometry video frames supplied from the geometry video frame decoding unit 125, and reconstructs the geometry data of the point cloud using the base patches. That is, each point is placed in a three-dimensional space. The geometry data reconstruction unit 126 supplies the reconstructed geometry data to the geometry smoothing processing unit 127.

[0089] The geometry smoothing processing unit 127 performs smoothing processing on the geometry data supplied from the geometry data reconstruction unit 126 to reduce burrs on patch boundaries, etc. The geometry smoothing processing unit 127 supplies the geometry data after the smoothing processing to the color video frame generation unit 128.

[0090] The color video frame generation unit 128 performs recolor processing or the like to make the base patches of the attribute data supplied from the patch decomposition unit 111 correspond to the geometry data supplied from the geometry smoothing processing unit 127, and generates a color video frame, which is a video frame in which the base patches are arranged. The color video frame generation unit 128 supplies the generated color video frame to the video encoding unit 129.

[0091] The video encoding unit 129 encodes the color video frames supplied from the color video frame generation unit 128 using a coding method for two-dimensional images to generate encoded data. The video encoding unit 129 supplies the generated encoded data to the multiplexer 103.

[0092] The auxiliary patch information generating unit 130 generates auxiliary patch information using information on the base patches and additional patches of the occupancy map supplied from the occupancy map generating unit 121. The auxiliary patch information generating unit 130 supplies the generated auxiliary patch information to the auxiliary patch information encoding unit 131.

[0093] The auxiliary patch information encoder 131 encodes the auxiliary patch information supplied from the auxiliary patch information generator 130 using an arbitrary encoding method to generate encoded data. The auxiliary patch information encoder 131 supplies the generated encoded data to the multiplexer 103.

[0094] <Encoding process flow> An example of the flow of the encoding process executed by the encoding device 100 configured as above will be described with reference to the flowchart of FIG.

[0095] When the encoding process is started, in step S101, the patch decomposition unit 101 of the encoding device 100 generates a base patch. In addition, in step S102, the patch decomposition unit 101 generates an additional patch. In this case, the encoding device 100 applies "Method 1-1" in Table 20 of Fig. 2, and therefore generates a base patch and an additional patch of the occupancy map.

[0096] In step S103, the packing encoding unit 102 executes a packing encoding process, packs the base patch and the additional patch, and encodes the generated video frame.

[0097] In step S104, the multiplexer 103 multiplexes the various encoded data generated in step S102 to generate a bit stream. In step S105, the multiplexer 103 outputs the bit stream to the outside of the encoding device 100. When the process of step S105 ends, the encoding process ends.

[0098] <Packing encoding process flow> Next, an example of the flow of the packing encoding process executed in step S103 in FIG. 11 will be described with reference to the flowchart in FIG.

[0099] When the packing encoding process is started, in step S121, the occupancy map generation unit 121 generates an occupancy map using the base patch generated in step S101 of Fig. 11. In addition, in step S122, the occupancy map generation unit 121 generates an additional occupancy map using the additional patch generated in step S102 of Fig. 11. Furthermore, in step S123, the geometry video frame generation unit 122 generates a geometry video frame using the base patch generated in step S101 of Fig. 11.

[0100] In step S124, the OMap encoder 123 encodes the occupancy map generated in step S121 using a coding scheme for two-dimensional images to generate encoded data. In step S125, the OMap encoder 123 encodes the additional occupancy map generated in step S122 using a coding scheme for two-dimensional images to generate encoded data.

[0101] In step S126, the video encoding unit 124 encodes the geometry video frame generated in step S123 using a coding method for two-dimensional images to generate encoded data. In step S127, the geometry video frame decoding unit 125 decodes the encoded data generated in step S126 using a decoding method for two-dimensional images corresponding to the coding method to generate (restore) a geometry video frame.

[0102] In step S128, the geometry data reconstruction unit 126 unpacks the geometry video frame generated (reconstructed) in step S127, and reconstructs the geometry data.

[0103] In step S129, the geometry smoothing processor 127 performs smoothing processing on the geometry data reconstructed in step S128 to suppress burrs and the like on the patch boundaries.

[0104] In step S130, the color video frame generating unit 128 performs recolor processing or the like to make the attribute data correspond to the geometry smoothing processing result, and generates a color video frame in which the base patch is arranged. In step S131, the video encoding unit 129 encodes the color video frame by a two-dimensional image encoding method, and generates encoded data.

[0105] In step S132, the auxiliary patch information generating unit 130 generates auxiliary patch information using information on the base patch and additional patch of the occupancy map. In step S133, the auxiliary patch information encoding unit 131 encodes the generated auxiliary patch information by an arbitrary encoding method to generate encoded data.

[0106] When the process of step S133 ends, the packing encoding process ends, and the process returns to FIG.

[0107] By executing each process as described above, the encoding device 100 can generate an occupancy map and an additional occupancy map for improving the accuracy of the occupancy map. Therefore, the encoding device 100 can locally improve the accuracy of the occupancy map. This makes it possible to suppress a decrease in the quality of the reconstructed point cloud while suppressing a decrease in the encoding efficiency and an increase in the load. In other words, it is possible to suppress a decrease in the image quality of the two-dimensional image for display of the 3D data.

[0108] <Decryption device> Fig. 13 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an image processing device to which the present technology is applied. The decoding device 200 shown in Fig. 13 is a device (a decoding device to which a video-based approach is applied) that decodes encoded data, in which 3D data such as a point cloud is projected onto a two-dimensional plane and encoded, by a decoding method for two-dimensional images and reconstructs 3D data. This decoding device 200 corresponds to the encoding device 100 in Fig. 9, and can decode a bit stream generated by the encoding device 100 and reconstruct 3D data. That is, this decoding device 200 performs such processing by applying "Method 1-1" shown in Table 20 in Fig. 2.

[0109] Note that Fig. 13 shows the main processing units, data flows, and the like, and is not limited to all that is shown in Fig. 13. In other words, in the decoding device 200, there may be processing units that are not shown as blocks in Fig. 13, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 13.

[0110] As shown in FIG. 13, the decoding device 200 includes a demultiplexer 201, an auxiliary patch information decoding unit 202, an OMap decoding unit 203, a video decoding unit 204, a video decoding unit 205, and a 3D reconstruction unit 206.

[0111] The demultiplexer 201 performs processing related to demultiplexing of data. For example, the demultiplexer 201 can obtain a bit stream to be input to the decoding device 200. This bit stream is supplied from the encoding device 100, for example.

[0112] The demultiplexer 201 can also demultiplex the bitstream. For example, the demultiplexer 201 can extract the coded data of auxiliary patch information from the bitstream by demultiplexing. The demultiplexer 201 can also extract the coded data of geometry video frames from the bitstream by demultiplexing. The demultiplexer 201 can also extract the coded data of color video frames from the bitstream by demultiplexing. The demultiplexer 201 can also extract the coded data of occupancy maps and the coded data of additional occupancy maps from the bitstream by demultiplexing.

[0113] Furthermore, the demultiplexer 201 can supply the extracted data to a downstream processing unit. For example, the demultiplexer 201 can supply the extracted coded data of the auxiliary patch information to the auxiliary patch information decoding unit 202. The demultiplexer 201 can also supply the extracted coded data of the geometry video frame to the video decoding unit 204. The demultiplexer 201 can also supply the extracted coded data of the color video frame to the video decoding unit 205. The demultiplexer 201 can also supply the extracted coded data of the occupancy map and the extracted coded data of the additional occupancy map to the OMap decoding unit 203.

[0114] The auxiliary patch information decoding unit 202 performs processing related to decoding of the encoded data of the auxiliary patch information. For example, the auxiliary patch information decoding unit 202 can acquire the encoded data of the auxiliary patch information supplied from the demultiplexer 201. Furthermore, the auxiliary patch information decoding unit 202 can decode the encoded data to generate the auxiliary patch information. This decoding method may be any decoding method that corresponds to the encoding method applied during encoding (for example, the encoding method applied by the auxiliary patch information encoding unit 131). Furthermore, the auxiliary patch information decoding unit 202 supplies the generated auxiliary patch information to the 3D reconstruction unit 206.

[0115] The OMap decoding unit 203 performs processing related to decoding of the coded data of the occupancy map and the coded data of the additional occupancy map. For example, the OMap decoding unit 203 can obtain the coded data of the occupancy map and the coded data of the additional occupancy map supplied from the demultiplexer 201. The OMap decoding unit 203 can also decode the coded data to generate the occupancy map and the additional occupancy map. Furthermore, the OMap decoding unit 203 can supply the occupancy map and the additional occupancy map to the 3D reconstruction unit 206.

[0116] The video decoding unit 204 performs processing related to the decoding of the encoded data of the geometry video frame. For example, the video decoding unit 204 can obtain the encoded data of the geometry video frame supplied from the demultiplexer 201. Furthermore, the video decoding unit 204 can decode the encoded data to generate a geometry video frame. This decoding method can be any decoding method for two-dimensional images that corresponds to the encoding method applied during encoding (for example, the encoding method applied by the video encoding unit 124). Furthermore, the video decoding unit 204 can supply the geometry video frame to the 3D reconstruction unit 206.

[0117] The video decoding unit 205 performs processing related to the decoding of the encoded data of the color video frame. For example, the video decoding unit 205 can obtain the encoded data of the color video frame supplied from the demultiplexer 201. Furthermore, the video decoding unit 205 can decode the encoded data to generate a color video frame. This decoding method can be any decoding method for two-dimensional images that corresponds to the encoding method applied during encoding (for example, the encoding method applied by the video encoding unit 129). Furthermore, the video decoding unit 205 can supply the color video frame to the 3D reconstruction unit 206.

[0118] The 3D reconstruction unit 206 performs processing related to unpacking of video frames and reconstruction of 3D data. For example, the 3D reconstruction unit 206 can acquire auxiliary patch information supplied from the auxiliary patch information decoding unit 202. Also, the 3D reconstruction unit 206 can acquire an occupancy map supplied from the OMap decoding unit 203. Furthermore, the 3D reconstruction unit 206 can acquire a geometry video frame supplied from the video decoding unit 204. Also, the 3D reconstruction unit 206 can acquire a color video frame supplied from the video decoding unit 205. Furthermore, the 3D reconstruction unit 206 can unpack the video frames and reconstruct 3D data (for example, a point cloud). The 3D reconstruction unit 206 outputs the 3D data obtained by such processing to the outside of the decoding device 200. For example, this 3D data is supplied to a display unit to display the image, recorded on a recording medium, or supplied to another device via communication.

[0119] <3D reconstruction part> Fig. 14 is a block diagram showing a main configuration example of the 3D reconstruction unit 206. Note that Fig. 14 shows main processing units, data flows, etc., and is not limited to all of the processing units shown in Fig. 14. In other words, in the 3D reconstruction unit 206, there may be processing units not shown as blocks in Fig. 14, and there may be processing and data flows not shown as arrows in Fig. 14.

[0120] As shown in FIG. 14, the 3D reconstruction unit 206 includes an occupancy map reconstruction unit 221 , a geometry data reconstruction unit 222 , an attribute data reconstruction unit 223 , a geometry smoothing processing unit 224 , and a recolor processing unit 225 .

[0121] The occupancy map reconstruction unit 221 generates a composite occupancy map by performing a bit-by-bit logical operation (by deriving a logical sum or a logical product) on the occupancy map and the additional occupancy map supplied from the OMap decoding unit 203 using the auxiliary patch information supplied from the auxiliary patch information decoding unit 202. The occupancy map generation unit 121 supplies the composite occupancy map to the geometry data reconstruction unit 222.

[0122] The geometry data reconstruction unit 222 unpacks the geometry video frame supplied from the video decoding unit 204 (FIG. 13) using the auxiliary patch information supplied from the auxiliary patch information decoding unit 202 and the composite occupancy map supplied from the occupancy map reconstruction unit 221 to extract a base patch of the geometry data. The geometry data reconstruction unit 222 further reconstructs the geometry data using the base patch and auxiliary patch information. The geometry data reconstruction unit 222 further supplies the reconstructed geometry data and the composite occupancy map to the attribute data reconstruction unit 223.

[0123] The attribute data reconstruction unit 223 unpacks the color video frame supplied from the video decoding unit 205 (FIG. 13) using the auxiliary patch information supplied from the auxiliary patch information decoding unit 202 and the composite occupancy map supplied from the occupancy map reconstruction unit 221 to extract a base patch of attribute data. The attribute data reconstruction unit 223 further reconstructs attribute data using the base patch and auxiliary patch information. The attribute data reconstruction unit 223 supplies various information such as geometry data, the composite occupancy map, and the reconstructed attribute data to the geometry smoothing processing unit 224.

[0124] The geometry smoothing processing unit 224 performs smoothing processing on the geometry data supplied from the attribute data reconstruction unit 223. The geometry smoothing processing unit 224 supplies the geometry data and attribute data that have been subjected to the smoothing processing to a recolor processing unit 225.

[0125] The recolor processing unit 225 acquires the geometry data and attribute data supplied from the geometry smoothing processing unit 224, performs recolor processing using them, and generates (reconstructs) a point cloud by making the attribute data correspond to the geometry data. The recolor processing unit 225 outputs the point cloud to the outside of the decoding device 200.

[0126] <Decryption process flow> An example of the flow of the decoding process executed by the decoding device 200 having such a configuration will be described with reference to the flowchart of FIG.

[0127] When the decoding process starts, in step S201, the demultiplexer 201 of the decoding device 200 demultiplexes the bitstream and extracts auxiliary patch information, occupancy maps, additional occupancy maps, geometry video frames, color video frames, etc. from the bitstream.

[0128] In step S202, the auxiliary patch information decoding unit 202 decodes the coded data of the auxiliary patch information extracted from the bitstream by the process of step S201. In step S203, the OMap decoding unit 203 decodes the coded data of the occupancy map extracted from the bitstream by the process of step S201. In addition, in step S204, the OMap decoding unit 203 decodes the coded data of the additional occupancy map extracted from the bitstream by the process of step S201.

[0129] In step S205, the video decoding unit 204 decodes the coded data of the geometry video frame extracted from the bitstream by the process of step S201. In step S206, the video decoding unit 205 decodes the coded data of the color video frame extracted from the bitstream by the process of step S201.

[0130] In step S207, the 3D reconstruction unit 206 performs 3D reconstruction processing using the information obtained by the above processing, and reconstructs the 3D data. When the processing in step S207 ends, the decoding processing ends.

[0131] <3D reconstruction process flow> Next, an example of the flow of the 3D reconstruction process executed in step S207 in FIG. 15 will be described with reference to the flowchart in FIG.

[0132] When the 3D reconstruction process is started, in step S221, the occupancy map reconstruction unit 221 performs a bitwise logical operation (including, for example, a logical sum or a logical product) on the occupancy map and the additional occupancy map using the auxiliary patch information to generate a composite occupancy map.

[0133] In step S222, the geometry data reconstruction unit 222 unpacks the geometry video frame and reconstructs the geometry data using the auxiliary patch information and the generated composite occupancy map.

[0134] In step S223, the attribute data reconstruction unit 223 unpacks the color video frame using the auxiliary patch information and the generated composite occupancy map, and reconstructs the attribute data.

[0135] In step S224, the geometry smoothing processor 224 performs smoothing processing on the geometry data obtained in step S222.

[0136] In step S225, the recolor processing unit 225 performs recolor processing, and reconstructs the point cloud by making the attribute data reconstructed in step S223 correspond to the geometry data that has been smoothed in step S224.

[0137] When the process of step S225 ends, the 3D reconstruction process ends and the process returns to FIG.

[0138] By executing each process as described above, the decoding device 200 can reconstruct 3D data using the occupancy map and the additional occupancy map for improving the accuracy of the occupancy map. Therefore, the decoding device 200 can locally improve the accuracy of the occupancy map. As a result, the decoding device 200 can suppress a decrease in the quality of the reconstructed point cloud while suppressing a decrease in the encoding efficiency and an increase in the load. In other words, it is possible to suppress a decrease in the image quality of the two-dimensional image for display of the 3D data.

[0139] <Method 1-2> Although "Method 1-1" has been described above, "Method 1-2" can also be realized in a similar manner. In the case of "Method 1-2", an additional patch of geometry data is generated. That is, in this case, the geometry video frame generating unit 122 (FIG. 10) generates a geometry video frame in which a base patch of geometry data is arranged, and an additional geometry video frame in which the additional patch of geometry data is arranged. The video encoding unit 124 encodes the geometry video frame and the additional geometry video frame, respectively, to generate encoded data.

[0140] Furthermore, information regarding the base patch and information regarding the additional patch are supplied from the geometry video frame generator 122 to the auxiliary patch information generator 130, which generates auxiliary patch information based on this information.

[0141] In the case of "Method 1-2", the geometry data reconstruction unit 222 of the decoding device 200 reconstructs the geometry data corresponding to the geometry video frame and the geometry data corresponding to the additional geometry video frame, and synthesizes them to generate synthetic geometry data. For example, the geometry data reconstruction unit 222 may generate synthetic geometry data by replacing the value of the geometry data corresponding to the base patch with the value of the geometry data corresponding to the additional patch. Alternatively, the geometry data reconstruction unit 222 may generate synthetic geometry data by adding or subtracting the value of the geometry data corresponding to the base patch and the value of the geometry data corresponding to the additional patch.

[0142] In this way, the accuracy of the geometry data can be locally improved. Then, by reconstructing a point cloud using such composite geometry data, it is possible to suppress a decrease in the coding efficiency and a decrease in the quality of the reconstructed point cloud while suppressing an increase in the load. In other words, it is possible to suppress a decrease in the image quality of a two-dimensional image for display of 3D data.

[0143] <Method 1-3> Of course, "Method 1-3" can be realized in a similar way. In the case of "Method 1-3", an additional patch of attribute data is generated. In other words, like the case of geometry data, the value of the attribute data corresponding to the base patch and the value of the attribute data corresponding to the additional patch can be added, subtracted, or replaced to generate composite attribute data by combining them.

[0144] In this case, information regarding the base patch and information regarding the additional patch are supplied from the color video frame generator 128 to the auxiliary patch information generator 130, which generates auxiliary patch information based on this information.

[0145] In this way, the accuracy of the attribute data can be locally improved. Then, by reconstructing the point cloud using such composite attribute data, it is possible to suppress the reduction in the coding efficiency and the reduction in the quality of the reconstructed point cloud while suppressing the increase in the load. In other words, it is possible to suppress the reduction in the image quality of the two-dimensional image for display of the 3D data.

[0146] <Combination> The above-mentioned "Method 1" to "Method 3" can be used in any combination. Furthermore, the above-mentioned "Method 1" to "Method 3" can all be applied.

[0147] <3. Second embodiment (method 2)> <Substitute for smoothing processing> In this embodiment, the above-mentioned "Method 2" will be described. In the case of this "Method 2", an additional occupancy map (an additional patch) is generated so that the composite occupancy map corresponds to the result of the smoothing process.

[0148] For example, when a base patch of an occupancy map with lower accuracy than the geometry data, as shown in A of FIG. 17, is represented with the accuracy of the geometry data, it becomes as shown in B of FIG. 17. When a smoothing process is performed on the geometry data, the patch becomes shaped as shown in C of FIG. 17. The hatched area in C of FIG. 17 indicates an area where points are added when B of FIG. 17 is used as a reference. Also, the gray area indicates an area where points are deleted when B of FIG. 17 is used as a reference. When this patch is represented with the same accuracy as the geometry data, the occupancy map becomes as shown in D of FIG. 17. In this case, the range of the geometry data can be accurately represented, but the amount of code for the occupancy map increases.

[0149] Therefore, an occupancy map for adding a point as shown in E of Fig. 17 and an occupancy map for deleting a point as shown in F of Fig. 17 are generated as additional occupancy maps. By transmitting such additional occupancy maps, an occupancy map reflecting the smoothing process can be generated on the decoding side. That is, the smoothing process result of the geometry data can be obtained without performing the smoothing process. That is, since the smoothing process can be omitted, an increase in the load due to the smoothing process can be suppressed.

[0150] <Packing coding part> In this case, too, the encoding device 100 has a configuration basically similar to that in the case of "Method 1-1" (FIG. 9). Moreover, an example of the main configuration of the packing encoding unit 102 in this case is shown in FIG. 18. As shown in FIG. 18, the packing encoding unit 102 in this case has a configuration basically similar to that in the case of "Method 1-1" (FIG. 10). However, in this case, the geometry smoothing processing unit 127 supplies the smoothed geometry data to the occupancy map generation unit 121. The occupancy map generation unit 121 generates an occupancy map corresponding to the base patch, and also generates an additional occupancy map based on the smoothed geometry data.

[0151] The occupancy map generation unit 121 supplies the generated occupancy map and the additional occupancy map to the OMap encoding unit 123. The OMap encoding unit 123 encodes them and generates encoded data thereof.

[0152] In addition, the occupancy map generating unit 121 supplies information relating to the occupancy map and the additional occupancy map to the auxiliary patch information generating unit 130. Based on the information, the auxiliary patch information generating unit 130 generates auxiliary patch information including information relating to the occupancy map and the additional occupancy map. The auxiliary patch information encoding unit 131 encodes the auxiliary patch information generated in the real world.

[0153] <Packing encoding process flow> In this case as well, the encoding process is executed by the encoding device 100 in a flow similar to that of the flowchart in Fig. 11. An example of the flow of the packing encoding process executed in step S103 (Fig. 11) of the encoding process in this case will be described with reference to the flowchart in Fig. 19.

[0154] In this case, when the packing encoding process is started, the processes of steps S301 to S307 are executed similarly to the processes of steps S121, S123, S124, and S126 to S129 in FIG.

[0155] In step S308, the occupancy map generator 121 generates an additional occupancy map based on the smoothing result of step S307. That is, the occupancy map generator 121 generates an additional occupancy map indicating areas to be added and areas to be deleted from the occupancy map so as to more accurately indicate the shape of the patch of the geometry data after the smoothing process, for example, as shown in Fig. 17. In step S309, the OMap encoder 123 encodes the additional occupancy map.

[0156] The processes in steps S310 to S313 are executed in the same manner as the processes in steps S130 to S133 in FIG.

[0157] As described above, by generating and transmitting an additional occupancy map based on the smoothing process result, the receiving side can reconstruct the geometry data using the additional occupancy map and the occupancy map, thereby reconstructing the geometry data to which the smoothing process has been applied. In other words, since the receiving side can reconstruct a point cloud reflecting the smoothing process without performing the smoothing process, it is possible to suppress an increase in the load due to the smoothing process.

[0158] <3D reconstruction part> Next, the receiving side will be described. In this case, too, the decoding device 200 has a configuration basically similar to that in the case of "method 1-1" (FIG. 13). Also, a main configuration example of the 3D reconstruction unit 206 in this case is shown in FIG. 20. As shown in FIG. 20, the 3D reconstruction unit 206 in this case has a configuration basically similar to that in the case of "method 1-1" (FIG. 10). However, in this case, the geometry smoothing processing unit 224 is omitted.

[0159] When the occupancy map reconstruction unit 221 generates a composite occupancy map from the occupancy map and the additional occupancy map, and the geometry data reconstruction unit 222 reconstructs the geometry data using the composite occupancy map, the geometry data that has been subjected to the smoothing process is obtained. Therefore, in this case, the geometry smoothing processing unit 224 can be omitted.

[0160] <3D reconstruction process flow> In this case, the decoding process is also executed by the decoding device 200 in a flow similar to that of the flowchart in Fig. 15. An example of the flow of the 3D reconstruction process executed in step S207 (Fig. 15) of the decoding process in this case will be described with reference to the flowchart in Fig. 21.

[0161] In this case, when the 3D reconstruction process is started, the processes of steps S331 to S334 are executed in the same manner as the processes of steps S221 to S225 in Fig. 16. That is, in this case, the process of step S332 obtains geometry data that has been subjected to the smoothing process. Therefore, the process of step S224 is omitted.

[0162] As described above, smoothing processing is not required on the receiving side, so an increase in the load can be suppressed.

[0163] <4. Third embodiment (Method 3)> <Specify processing range> In this embodiment, the above-mentioned "Method 3" will be described. In the case of this "Method 3", the range to be subjected to processing such as smoothing processing, which is applied to the geometry data and attribute data, is specified by an additional occupancy map.

[0164] <Packing encoding process flow> In this case, the encoding device 100 has a configuration similar to that in the case of "Method 2" (FIGS. 9 and 18). The encoding process executed by the encoding device 100 is also executed in a similar flow to that in the case of "Method 1-1" (FIG. 11).

[0165] An example of the flow of the packing encoding process in this case will be described with reference to the flowchart in FIG.

[0166] When the packing encoding process is started, the processes of steps S351 to S357 are executed in the same manner as the processes of steps S301 to S307 in FIG. 19 (in the case of "method 2").

[0167] In step S358, the occupancy map generating unit 121 generates an additional occupancy map indicating positions to which the smoothing process is applied based on the smoothing process result in step S307. That is, the occupancy map generating unit 121 generates the additional occupancy map so as to set a flag in the area to which the smoothing process is applied.

[0168] Then, the processes of steps S359 to S363 are executed in the same manner as the processes of steps S309 to S313 in FIG.

[0169] As described above, by generating and transmitting an additional occupancy map indicating the range in which the smoothing process is performed based on the smoothing process result, the receiving side can more easily perform the smoothing process in an appropriate range based on the additional occupancy map. In other words, the receiving side does not need to search for the range in which the smoothing process is performed, so that an increase in the load can be suppressed.

[0170] <3D reconstruction process flow> Next, the receiving side will be described. In this case, the decoding device 200 (and the 3D reconstruction unit 206) has a configuration basically similar to that in the case of "method 1-1" (FIGS. 13 and 14). Also, the decoding process in this case is executed by the decoding device 200 in a similar flow to that of the flowchart in FIG. 15. Then, an example of the flow of the 3D reconstruction process executed in step S207 (FIG. 15) of the decoding process in this case will be described with reference to the flowchart in FIG. 22.

[0171] In this case, when the 3D reconstruction process is started, in step S381, the geometry data reconstruction unit 222 unpacks the geometry video frame using the auxiliary patch information and the occupancy map, and reconstructs the geometry data.

[0172] In step S382, the attribute data reconstruction unit 223 unpacks the color video frame using the auxiliary patch information and the occupancy map, and reconstructs the attribute data.

[0173] In step S383, the geometry smoothing unit 224 performs smoothing processing on the geometry data based on the additional occupancy map. That is, the geometry smoothing unit 224 performs smoothing processing on the range specified by the additional occupancy map.

[0174] In step S384, the recolor processing unit 225 performs recolor processing, and reconstructs the point cloud by making the attribute data reconstructed in step S382 correspond to the geometry data that has been smoothed in step S383.

[0175] When the process of step S384 ends, the 3D reconstruction process ends and the process returns to FIG.

[0176] In this way, by performing the smoothing process on the range indicated by the additional occupancy map, the smoothing process can be performed more easily on the appropriate range. In other words, since the receiver does not need to search for the range to apply the smoothing process, the load on the receiver can be reduced.

[0177] <5. Fourth embodiment (method 4)> <Rebuild> In this embodiment, the above-mentioned "Method 4" will be described. In the case of this "Method 4", an additional patch used for reconstruction of the point cloud is generated similarly to the base patch. However, the additional patch is optional and does not need to be used for reconstruction (the additional patch can be omitted and the point cloud can be reconstructed using only the base patch).

[0178] <Packing coding part> In this case, too, the encoding device 100 has a configuration basically similar to that in the case of "Method 1-1" (FIG. 9). Moreover, an example of the main configuration of the packing encoding unit 102 in this case is shown in FIG. 24. As shown in FIG. 24, the packing encoding unit 102 in this case has a configuration basically similar to that in the case of "Method 1-1" (FIG. 10). However, in this case, the patch decomposition unit 101 generates an additional patch for the occupancy map and the geometry data. That is, the patch decomposition unit 101 generates a base patch and an additional patch for the occupancy map and the geometry data.

[0179] Therefore, the occupancy map generation unit 121 of the packing encoding unit 102 generates an occupancy map corresponding to the base patch and an additional occupancy map corresponding to the additional patch, and the geometry video frame generation unit 122 generates a geometry video frame in which the base patch is arranged and an additional geometry video frame in which the additional patch is arranged.

[0180] The auxiliary patch information generating unit 130 obtains information on the base patch and information on the additional patch from each of the occupancy map generating unit 121 and the geometry video frame generating unit 122, and generates auxiliary patch information including them.

[0181] The OMap encoding unit 123 encodes the occupancy map and the additional occupancy map generated in the occupancy map generation unit 121. Furthermore, the video encoding unit 124 encodes the geometry video frame and the additional geometry video frame generated in the geometry video frame generation unit 122. The auxiliary patch information encoding unit 131 encodes the auxiliary patch information to generate encoded data.

[0182] In addition, an additional patch may be generated for the attribute data as well, but as in this example, the attribute data may be omitted from the additional patch, and the attribute data corresponding to the additional patch may be obtained by recoloring processing on the receiving side.

[0183] <Packing encoding process> In this case as well, the encoding process is executed by the encoding device 100 in a flow similar to that of the flowchart in Fig. 11. An example of the flow of the packing encoding process executed in step S103 (Fig. 11) of the encoding process in this case will be described with reference to the flowchart in Fig. 25.

[0184] In this case, when the packing encoding process is started, the processes of steps S401 to S403 are executed in the same manner as the processes of steps S121 to S123 in FIG.

[0185] In step S404, the geometry video frame generating unit 122 generates an additional geometry video frame in which the additional patch is arranged.

[0186] The processes in steps S405 to S407 are executed in the same manner as the processes in steps S124 to S126 in FIG.

[0187] In step S408, the video encoding unit 124 encodes the additional geometry video frame.

[0188] The processes in steps S409 to S415 are executed in the same manner as the processes in steps S127 to S133 in FIG.

[0189] That is, in this case at least additional patches of geometry data and occupancy maps are generated, which can then be used to reconstruct the point cloud.

[0190] <3D reconstruction part> Next, the receiving side will be described. In this case, too, the decoding device 200 has a configuration basically similar to that in the case of "method 1-1" (FIG. 13). Also, a main configuration example of the 3D reconstruction unit 206 in this case is shown in FIG. 26. As shown in FIG. 26, the 3D reconstruction unit 206 in this case has a base patch 3D reconstruction unit 451, a geometry smoothing processing unit 452, a recolor processing unit 453, an additional patch 3D reconstruction unit 454, a geometry smoothing processing unit 455, and a recolor processing unit 456.

[0191] The base patch 3D reconstruction unit 451, the geometry smoothing processing unit 452, and the recolor processing unit 453 perform processing related to the base patch. The base patch 3D reconstruction unit 451 reconstructs a point cloud (a small area corresponding to the base patch) using auxiliary patch information, an occupancy map corresponding to the base patch, a base patch of a geometry video frame, and a base patch of a color video frame. The geometry smoothing processing unit 452 performs a smoothing process on the geometry data corresponding to the base patch. The recolor processing unit 453 performs a recolor process so as to make attribute data correspond to the geometry data on which the smoothing process has been performed.

[0192] The additional patch 3D reconstruction unit 454, the geometry smoothing processing unit 455, and the recolor processing unit 456 perform processing related to the additional patch. The additional patch 3D reconstruction unit 454 reconstructs a point cloud (a small area corresponding to the additional patch) using the auxiliary patch information, the additional occupancy map, and the additional geometry video frame (i.e., using the additional patch). The geometry smoothing processing unit 455 performs a smoothing process on the geometry data corresponding to the base patch. The recolor processing unit 456 performs a recolor process using the recolor processing result by the recolor processing unit 453, i.e., the attribute data of the base patch. As a result, the recolor processing unit 456 synthesizes the point cloud corresponding to the base patch and the point cloud corresponding to the additional patch, generates and outputs a point cloud corresponding to the base patch and the additional patch.

[0193] <3D reconstruction process flow> In this case, the decoding process is also executed by the decoding device 200 in a flow similar to that of the flowchart in Fig. 15. An example of the flow of the 3D reconstruction process executed in step S207 (Fig. 15) of the decoding process in this case will be described with reference to the flowchart in Fig. 27.

[0194] In this case, when the 3D reconstruction process is started, in step S451, the base patch 3D reconstruction unit 451 unpacks the geometry video frame and color video frame for the base patch using the auxiliary patch information and the occupancy map, and reconstructs a point cloud corresponding to the base patch.

[0195] In step S452, the geometry smoothing processor 452 performs smoothing processing on the geometry data of the base patch. That is, the geometry smoothing processor 452 performs smoothing processing on the geometry data of the point cloud corresponding to the base patch obtained in step S451.

[0196] In step S453, the recolor processing unit 453 performs a recolor process on the base patch. That is, the recolor processing unit 453 performs a recolor process so that the attribute data of the point cloud corresponding to the base patch obtained in step S451 corresponds to the geometry data.

[0197] In step S454, the additional patch 3D reconstruction unit 454 determines whether to decode the additional patch based on, for example, the auxiliary patch information, etc. For example, if an additional patch exists and it is determined that the additional patch is to be decoded, the process proceeds to step S455.

[0198] In step S455, the additional patch 3D reconstruction unit 454 unpacks the additional geometry video frame for the additional patch using the auxiliary patch information and the additional occupancy map, and reconstructs geometry data corresponding to the additional patch.

[0199] In step S456, the geometry smoothing processor 455 performs smoothing processing on the geometry data for the additional patch. That is, the geometry smoothing processor 455 performs smoothing processing on the geometry data of the point cloud corresponding to the additional patch obtained in step S455.

[0200] In step S457, the recolor processor 456 performs recolor processing of the additional patch using the attribute data of the base patch. That is, the recolor processor 456 makes the attribute data of the base patch correspond to the geometry data obtained by the smoothing processing in step S456.

[0201] By executing each process in this way, the point cloud corresponding to the base patch and the additional patch is reconstructed. When the process of step S457 is completed, the 3D reconstruction process is completed. Also, when it is determined in step S454 that the additional patch is not to be decoded, the 3D reconstruction process is completed. That is, the point cloud corresponding to the base patch is output.

[0202] As described above, since the point cloud can be reconstructed using additional patches, the point cloud can be reconstructed in a wider variety of ways.

[0203] <6. Fifth embodiment (Method 5)> <Supplementary patch information> As described above, when applying an additional patch, in addition to "1. Information regarding the base patch," the supplemental patch information may also transmit "2. Information regarding the additional patch," for example, as shown in Table 501 in FIG. 28.

[0204] The content of this "2. Information regarding additional patches" is arbitrary. For example, it may include "2-1. Additional patch flag." This additional patch flag is flag information that indicates whether the corresponding patch is an additional patch or not. For example, if this additional patch flag is "true (1)," it indicates that the corresponding patch is an additional patch. By referring to this flag information, it is possible to more easily distinguish between additional patches and base patches.

[0205] Furthermore, "2-2. Information on the use of the additional patch" may be included in "2. Information on the additional patch." This "2-2. Information on the use of the additional patch" may include, for example, "2-2-1. Information indicating the target of the additional patch." This "2-2-1. Information indicating the target of the additional patch" indicates, for example, what data the additional patch will affect depending on the parameter value, as in table 502 in FIG. 29.

[0206] In the example of Figure 29, when the parameter value is "0", it indicates that the target of the additional patch is the occupancy map corresponding to the base patch. When the parameter value is "1", it indicates that the target of the additional patch is the base patch of the geometry data. When the parameter value is "2", it indicates that the target of the additional patch is the base patch of the attribute data. When the parameter value is "3", it indicates that the target of the additional patch is the occupancy map corresponding to the additional patch. When the parameter value is "4", it indicates that the target of the additional patch is the additional patch of the geometry data. When the parameter value is "5", it indicates that the target of the additional patch is the additional patch of the attribute data. When the parameter value is "6", it indicates that the target of the additional patch is both the additional patch of the geometry data and the additional patch of the attribute data.

[0207] 28, "2-2. Information on the use of the additional patch" may include, for example, "2-2-2. Information indicating the processing content using the additional patch." This "2-2-2. Information indicating the processing content using the additional patch" indicates, for example, what processing the additional patch is used for depending on the parameter value, as in table 503 in FIG. 30.

[0208] In the example of FIG. 30, when the parameter value is "0", it indicates that the additional patch is used to reconstruct the point cloud. Also, when the parameter value is "1", it indicates that the additional patch is used to add a point. In other words, in this case, a bitwise logical AND is derived between the base patch and the additional patch. Furthermore, when the parameter value is "2", it indicates that the additional patch is used to delete a point. In other words, in this case, a bitwise logical AND is derived between the base patch and the additional patch.

[0209] Moreover, when the parameter value is "3", it indicates that the value of the additional patch is added to the value of the base patch, and when the parameter value is "4", it indicates that the value of the base patch is replaced with the value of the additional patch.

[0210] A parameter value of 5 indicates that the target point is flagged for smoothing, a parameter value of 6 indicates that recoloring is performed from the reconstructed point cloud corresponding to the base patch, and a parameter value of 7 indicates that the additional patch is decoded according to its distance from the viewpoint.

[0211] Returning to Fig. 28, "2-3. Information regarding alignment of additional patches" may be included in "2. Information regarding additional patches." This "2-3. Information regarding alignment of additional patches" may include, for example, information such as "2-3-1. Target patch ID," "2-3-2. Position information of additional patches," "2-3-3. Position shift information of additional patches," and "2-3-4. Size information of additional patches."

[0212] For example, when the base patch and the additional patch are located at different positions, "2-3-1. Target patch ID" and "2-3-2. Position information of the additional patch" may be included in "2. Information related to the additional patch."

[0213] "2-3-1. Target patch ID" is identification information (patch index) of the target patch. "2-3-2. Position information of additional patch" is information indicating the position of the additional patch on the occupancy map, and is indicated by two-dimensional plane coordinates such as (u0', v0'). For example, in FIG. 31, the additional patch corresponding to the base patch 511 is the additional patch 512. In this case, the coordinates of the upper left point 513 of the additional patch 512 are "2-3-2. Position information of additional patch". Note that this "2-3-2. Position information of additional patch" may be expressed by the shift amount (Δu0, Δv0) from the position (u0, v0) of the base patch, as indicated by the arrow 514 in FIG. 31. Note that Δu0=u0-u0', Δv0=v0-v0'.

[0214] Also, for example, if the base patch and the additional patch have different sizes, "2-3-3. Position shift information of the additional patch" and "2-3-4. Size information of the additional patch" may be included in "2. Information related to the additional patch."

[0215] "2-3-3. Position shift information of additional patch" is the amount of position shift caused by size change. In the example of Fig. 31, arrow 514 corresponds to "2-3-3. Position shift information of additional patch". In other words, this "2-3-3. Position shift information of additional patch" is represented by (Δu0, Δv0).

[0216] "2-3-4. Size information of additional patch" indicates the size of the patch after the change. In other words, it is information indicating the size of the additional patch 512 indicated by the dotted line in FIG. 31, and is indicated by the width and height, for example, w', h', etc. Note that this "2-3-4. Size information of additional patch" may be expressed by the differences Δw, Δh from the base patch. Note that Δw=w-w', Δh=h-h'.

[0217] In addition, by sharing the base patch and patch information, it is possible to omit the transmission of alignment information.

[0218] 28, "2-4. Size setting information for additional occupancy map" may be included in "2. Information related to additional patch." The "2-4. Size setting information for additional occupancy map" may include "2-4-1. Occupancy precision," "2-4-2. Picture size," "2-4-3. Ratio per patch," and the like, which indicate the accuracy of the occupancy map.

[0219] In other words, the accuracy of the additional occupancy map may be expressed as in the past by "2-4-1. Occupancy Precision," or it may be expressed by "2-4-2. Image Size," or it may be expressed by "2-4-3. Ratio per Patch."

[0220] "2-4-2. Image size" is information indicating the size of the occupancy map, and is indicated, for example, by the width and height of the occupancy map. In other words, if the height of the additional occupancy map 522 shown in B of FIG. 32 is 1x and the width is 2x that of the base occupancy map 521 shown in A of FIG. 32, then width=2 and height=1 are specified. In this way, it is possible to collectively control the patches in the occupancy map. This makes it possible to suppress a decrease in the coding efficiency of the auxiliary patch information.

[0221] "2-4-3. Ratio of each patch" is information that specifies the ratio for each patch. For example, as shown in C of FIG. 32, information indicating the respective ratios of patch 531, patch 532, and patch 533 can be transmitted. In this way, the size of each patch can be controlled more flexibly. For example, it is possible to improve the accuracy of only the necessary patches.

[0222] Examples of information to be transmitted in each of the above-mentioned "Method 1" to "Method 4" are shown in Table 551 of Fig. 33. As shown in Table 551, various types of information can be transmitted in each method.

[0223] As described above, by providing additional patches to the base patch, it becomes possible to control the local information accuracy, which can suppress the decrease in coding efficiency, the increase in load, and the decrease in the quality of the reconstructed point cloud.

[0224] Also, for example, an object can be reconstructed with a precision according to the distance from the viewpoint. For example, by controlling whether or not to use an additional patch according to the distance from the viewpoint, an object far from the viewpoint can be reconstructed with the coarse precision of the base patch, and an object close to the viewpoint can be reconstructed with the high precision of the additional patch.

[0225] Furthermore, for example, the quality of the reconstructed point cloud can be locally controlled based on the user's authority, etc. (local control of the subjective image quality of the display image). For example, the entire point cloud can be provided in the original quality (high resolution) to users who have paid a high usage fee or users with administrator authority, but a portion of the point cloud can be provided in a lower quality (low resolution) state to users who have paid a low usage fee or users with guest authority (i.e., the image is provided in a state where some areas of the two-dimensional image are pixelated). This makes it possible to realize a variety of services.

[0226] <7. Notes> <Computer> The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, etc., capable of executing various functions by installing various programs.

[0227] FIG. 34 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0228] In a computer 900 shown in FIG. 34, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0229] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0230] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0231] In the computer configured as above, the series of processes described above are performed by the CPU 901, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data and the like necessary for the CPU 901 to execute various processes as appropriate.

[0232] The program executed by the computer can be applied by recording it on a removable medium 921 such as a package medium, for example. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by mounting the removable medium 921 on the drive 915.

[0233] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0234] In addition, this program can also be installed in advance in the ROM 902 or the storage unit 913.

[0235] <Target of application of this technology> Although the above describes the application of this technology to the encoding and decoding of point cloud data, this technology is not limited to these examples and can be applied to the encoding and decoding of 3D data of any standard. In other words, as long as it does not conflict with the above-mentioned technology, various processes such as encoding and decoding methods and specifications of various data such as 3D data and metadata are arbitrary. In addition, as long as it does not conflict with the present technology, some of the above-mentioned processes and specifications may be omitted.

[0236] Furthermore, although the encoding device 100 and the decoding device 200 have been described above as application examples of the present technology, the present technology can be applied to any configuration.

[0237] For example, the present technology can be applied to various electronic devices, such as transmitters and receivers (e.g., television sets and mobile phones) in satellite broadcasting, wired broadcasting such as cable TV, distribution over the Internet, and distribution to terminals via cellular communication, or devices (e.g., hard disk recorders and cameras) that record images on media such as optical disks, magnetic disks, and flash memories, and play images from these storage media.

[0238] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set in which other functions are added to a unit (e.g., a video set).

[0239] In addition, for example, the present technology can also be applied to a network system composed of multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and cooperate with each other to perform processing via a network. For example, the present technology may be implemented in a cloud service that provides services related to images (video images) to any terminal such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0240] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0241] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. In addition, the applications are also arbitrary.

[0242] <Other> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Thus, the value that this "flag" can take may be, for example, a binary value of 1 / 0, or may be three or more values. In other words, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, it is assumed that identification information (including flags) is not only included in the bit stream, but also includes difference information of the identification information with respect to certain reference information in the bit stream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information with respect to the reference information.

[0243] Also, various information (metadata, etc.) related to the coded data (bit stream) may be transmitted or recorded in any form as long as it is associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, data associated with each other may be combined into one piece of data, or each piece of data may be made into separate data. For example, information associated with coded data (image) may be transmitted on a transmission path different from that of the coded data (image). Also, for example, information associated with coded data (image) may be recorded on a recording medium different from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may be a part of the data, not the entire data. For example, an image and information corresponding to the image may be associated with each other in any unit, such as multiple frames, one frame, or a part of a frame.

[0244] In this specification, the terms "combine," "multiplex," "append," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0245] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0246] For example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0247] Also, for example, the above-mentioned program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0248] Also, for example, each step of one flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0249] Also, for example, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel, or individually at a required timing such as when a call is made. In other words, the processing of each step may be executed in an order different from the order described above, so long as no contradiction occurs. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0250] Also, for example, a plurality of techniques related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any of a plurality of the present technologies can be implemented in combination. For example, a part or all of the present technology described in any of the embodiments can be implemented in combination with a part or all of the present technology described in other embodiments. Also, a part or all of any of the above-mentioned present technologies can be implemented in combination with other technologies not described above.

[0251] The present technology can also be configured as follows. (1) a video frame generating unit that generates a base video frame in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each partial area, and an additional video frame in which an additional patch is arranged by projecting a partial area including at least a part of the partial area corresponding to the base patch of the point cloud onto the same two-dimensional plane as in the case of the base patch, with at least some parameters changed from those of the base patch; an encoding unit that encodes the base video frame and the additional video frame generated by the video frame generating unit to generate encoded data; An image processing device comprising: (2) The additional patch is configured with information of higher accuracy than the base patch. An image processing device as described in (1). (3) the additional video frame is an occupancy map; The additional patch indicates an area to be added to the area indicated by the base patch, or an area to be deleted from the area indicated by the base patch. An image processing device as described in (2). (4) The additional patch represents the result of a smoothing process on the base patch. An image processing device as described in (3). (5) the additional video frame is a geometry video frame or a color video frame; The additional patch is composed of values ​​that are added to the values ​​of the base patch or values ​​that replace the values ​​of the base patch. An image processing device as described in (2). (6) The additional patch indicates a range of the area indicated by the base patch to which a predetermined process is to be applied. An image processing device as described in (1). (7) The additional patch indicates a range of the area indicated by the base patch in which smoothing processing is to be performed. An image processing device according to (6). (8) generating a base video frame in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each partial region, and an additional video frame in which an additional patch is arranged by projecting a partial region including at least a part of the partial region of the point cloud corresponding to the base patch onto the same two-dimensional plane as the base patch, with at least some parameters changed from those of the base patch; The generated base video frame and the additional video frame are encoded to generate encoded data. Image processing methods.

[0252] (9) a decoding unit that decodes encoded data and generates a base video frame in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each partial area, and an additional video frame in which an additional patch is arranged by projecting a partial area including at least a part of the partial area corresponding to the base patch of the point cloud onto the same two-dimensional plane as the base patch, with at least some parameters changed from those of the base patch; a reconstruction unit that reconstructs the point cloud using the base video frames generated by the decoding unit and the additional video frames; An image processing device comprising: (10) Decoding the encoded data, and generating a base video frame in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each partial area, and an additional video frame in which an additional patch is arranged by projecting a partial area including at least a part of the partial area of ​​the point cloud corresponding to the base patch onto the same two-dimensional plane as the base patch, with at least some parameters changed from those of the base patch; Reconstructing the point cloud using the generated base video frames and the additional video frames. Image processing methods.

[0253] (11) An auxiliary patch information generating unit that generates auxiliary patch information, which is information about patches obtained by projecting a point cloud, which represents an object of a three-dimensional shape as a set of points, onto a two-dimensional plane for each partial region, and includes an additional patch flag that indicates whether the additional patch is an additional patch that is not essential for reconstructing the corresponding partial region of the point cloud; and an auxiliary patch information encoding unit that encodes the auxiliary patch information generated by the auxiliary patch information generating unit and generates encoded data; An image processing device comprising: (12) an additional video frame generating unit generating an additional video frame in which the additional patch corresponding to the auxiliary patch information generated by the auxiliary patch information generating unit is arranged; an additional video frame encoding unit that encodes the additional video frame generated by the additional video frame generating unit; Further equipped The image processing device according to (11). (13) The additional video frames are occupancy map and geometry video frames. An image processing device according to (12). (14) The auxiliary patch information further includes information indicating a target of the additional patch. The image processing device according to (11). (15) The auxiliary patch information further includes information indicating a process to be performed using the additional patch. The image processing device according to (11). (16) The auxiliary patch information further includes information regarding alignment of the additional patch. The image processing device according to (11). (17) The auxiliary patch information further includes information regarding a size setting of the additional patch. The image processing device according to (11). (18) Generate auxiliary patch information, which is information about patches obtained by projecting a point cloud, which represents a three-dimensional object as a set of points, onto a two-dimensional plane for each partial region, and includes an additional patch flag indicating whether the additional patch is not essential for reconstructing the corresponding partial region of the point cloud; The generated auxiliary patch information is encoded to generate encoded data. Image processing methods.

[0254] (19) an auxiliary patch information decoding unit that decodes the encoded data and generates auxiliary patch information, which is information about patches obtained by projecting a point cloud, which represents a three-dimensional object as a set of points, onto a two-dimensional plane for each partial region; a reconstruction unit that reconstructs the point cloud using the additional patch based on an additional patch flag that indicates whether the additional patch is an additional patch that is not essential for reconstruction of a corresponding partial region of the point cloud, the additional patch being included in the auxiliary patch information generated by the auxiliary patch information decoding unit; An image processing device comprising: (20) Decoding the encoded data, and generating auxiliary patch information, which is information about patches obtained by projecting a point cloud, which represents a three-dimensional object as a set of points, onto a two-dimensional plane for each subregion; The point cloud is reconstructed using the additional patch based on an additional patch flag that indicates whether the additional patch is not essential for reconstruction of the corresponding partial region of the point cloud, the additional patch being included in the generated auxiliary patch information. Image processing methods. [Explanation of symbols]

[0255] 100 encoding device, 101 patch decomposition unit, 102 packing encoding unit, 103 multiplexer, 121 occupancy map generation unit, 122 geometry video frame generation unit, 123 OMap encoding unit, 124 video encoding unit, 125 geometry video frame decoding unit, 126 geometry data reconstruction unit, 127 geometry smoothing processing unit, 128 color video frame generation unit, 129 video encoding unit, 130 auxiliary patch information generation unit, 131 auxiliary patch information encoding unit, 200 decoding device, 201 demultiplexer, 202 auxiliary patch information decoding unit, 203 OMap decoding unit, 204 and 205 video decoding unit, 206 3D reconstruction unit, 221 occupancy map reconstruction unit, 222 geometry data reconstruction unit, 223 attribute data reconstruction unit, 224 Geometry smoothing processor, 225 Recolor processor, 451 Base patch 3D reconstruction unit, 452 Geometry smoothing processor, 453 Recolor processor, 454 Additional patch 3D reconstruction unit, 455 Geometry smoothing processor, 456 Recolor processor

Claims

1. a video frame generating unit that generates a base video frame in which a base patch is arranged by projecting a point cloud, which represents an object having a three-dimensional shape as a set of points, onto a two-dimensional plane for each partial region, and an additional video frame in which an additional patch is arranged by projecting a partial region including at least a part of the partial region of the point cloud corresponding to the base patch onto the same two-dimensional plane as the base patch, with at least a part of parameters changed from those of the base patch; an encoding unit that encodes the base video frame and the additional video frame generated by the video frame generating unit to generate encoded data; An image processing device comprising:

2. The additional patch is configured with information of higher accuracy than the base patch. The image processing device according to claim 1 .

3. the additional video frame is an occupancy map; The additional patch indicates an area to be added to the area indicated by the base patch, or an area to be deleted from the area indicated by the base patch. The image processing device according to claim 2 .

4. The additional patch represents the result of smoothing the base patch. The image processing device according to claim 3 .

5. the additional video frame is a geometry video frame or a color video frame; The additional patch is composed of values ​​that are added to the values ​​of the base patch or values ​​that replace the values ​​of the base patch. The image processing device according to claim 2 .

6. The additional patch indicates a range of the area indicated by the base patch to which a predetermined process is to be applied. The image processing device according to claim 1 .

7. The additional patch indicates a range in the area indicated by the base patch where smoothing processing is to be performed. The image processing device according to claim 6.

8. generating a base video frame in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each partial region, and an additional video frame in which an additional patch is arranged by projecting a partial region including at least a part of the partial region of the point cloud corresponding to the base patch onto the same two-dimensional plane as the base patch, with at least a part of parameters changed from those of the base patch; The generated base video frame and the additional video frame are encoded to generate encoded data. Image processing methods.

9. a decoding unit that decodes the encoded data and generates a base video frame in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each partial area, and an additional video frame in which an additional patch is arranged by projecting a partial area including at least a part of the partial area corresponding to the base patch of the point cloud onto the same two-dimensional plane as the base patch, with at least a part of parameters changed from those of the base patch; a reconstruction unit that reconstructs the point cloud using the base video frame generated by the decoding unit and the additional video frame; An image processing device comprising:

10. Decoding the encoded data, and generating a base video frame in which a base patch is arranged by projecting a point cloud, which represents a three-dimensional object as a collection of points, onto a two-dimensional plane for each partial region, and an additional video frame in which an additional patch is arranged by projecting a partial region including at least a part of the partial region of the point cloud corresponding to the base patch onto the same two-dimensional plane as the base patch, with at least a part of parameters changed from those of the base patch; Reconstructing the point cloud using the generated base video frames and the additional video frames. Image processing methods.

Images