Image processing device and method

Abstract

The present disclosure relates to an image processing device and method capable of suppressing image quality degradation. Correction information is generated, which is information about the correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image, and the generated correction information is encoded. In addition, the encoded data of the correction information is decoded, and 3D data is constructed using the 2D data and the correction information generated by decoding the encoded data of the correction information, the correction information being information about the correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image. The present disclosure can be applied to, for example, an information processing device, an image processing device, an electronic device, an information processing method, and a program.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus and method, and more particularly to an image processing apparatus and method capable of suppressing degradation in image quality. Background Art

[0002] Conventionally, for example, as a method of encoding 3D data representing a three-dimensional structure such as a point cloud, there has been, for example, encoding using voxels such as an octree (for example, see Non-Patent Literature 1).

[0003] In recent years, for example, the following method (hereinafter also referred to as a video-based method) has been proposed as another encoding method: projecting the position information and attribute information of the point cloud onto a two-dimensional plane for each small area; arranging the image (patch) projected onto the two-dimensional plane in a frame image; and encoding the frame image using an encoding method for two-dimensional images.

[0004] In this video-based method, when an occupancy map indicating whether a patch exists at each position of a frame image is used, encoding efficiency can be improved by reducing resolution and reducing the amount of information.

[0005] However, if the resolution of the occupancy map is reduced, there is a possibility that the patch range indicated by the occupancy map may not be consistent with the actual patch range, and points that originally do not exist may be generated during the reconstruction of the 3D data. Therefore, a method of correcting the position of such points by smoothing is considered. However, even if such smoothing is performed, there is a possibility that the peak signal-to-noise ratio (PSNR) may deteriorate. Therefore, a method of removing blank parts generated by reducing the resolution (N×N accuracy) of the occupancy map has been considered (for example, see non-patent document 5).

[0006] Furthermore, in the case of a high-resolution occupancy map (1×1 precision) without overlapping patches, there is a possibility that if the positions of patch boundary points are distorted, gaps may be formed between adjacent patches, and the gaps may become holes (missing portions) in a two-dimensional image for display generated from 3D data. Therefore, a method of widening the boundary of a high-resolution occupancy map (1×1 precision) is also considered (for example, see Non-Patent Document 6).

[0007] Reference List

[0008] Non-patent literature

[0009] Non-patent literature 1: R. Mekuria, IEEE research member, K. Blom, P. Cesar., IEEE member, "Design, Implementation and Evaluation of a Point Cloud Codec for Tele-Immersive Video", tcsvt_paper_submitted_february.pdf.

[0010] Non-Patent Literature 2: Tim Golla and Reinhard Klein, “Real-time Point Cloud Compression,” IEEE, 2015.

[0011] Non-patent literature 3: K. Mammou, “Video-based and Hierarchical Approaches PointCloud Compression,” MPEG m41649, October 2017.

[0012] Non-patent document 4: K. Mammou, “PCC Test Model Category 2v0,” N17248MPEG output file, October 2017.

[0013] Non-Patent Literature 5: Dejun Zhang, Zheng Liu, Vladyslav Zakharchenko, Jianle Chen, Kangying Cai, “[VPCC] New proposal of an adaptive outlier removing method”, ISO / IEC JTC1 / SC29 / WG11 MPEG2018 / m44766, October 2018, Macau, China.

[0014] Non-Patent Literature 6: Dejun Zhang, Zheng Liu, Vladyslav Zakharchenko, Jianle Chen, Kangying Cai, “[VPCC] New proposal of an adaptive hole filling method for reconstructed point cloud”, ISO / IEC JTC1 / SC29 / WG11MPEG2018 / m44767, October 2018, Macau, China. Summary of the invention

[0015] Problems to be solved by the present invention

[0016] However, since the occupancy map is corrected in the methods described in Non-Patent Document 5 and Non-Patent Document 6, there is a possibility that another patch may be included in the area indicating the presence of one patch and the constructed 3D data may be degraded. Therefore, there is a possibility that the subjective image quality of the display image obtained by projecting the 3D data on the two-dimensional plane may be reduced.

[0017] The present disclosure has been made in view of such circumstances, and can suppress a decrease in the image quality of a two-dimensional image for displaying 3D data.

[0018] Solution to the problem

[0019] An image processing device according to one aspect of the present technology is an image processing device, which includes: a correction information generating unit, which generates correction information, which is information about the correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image; and a correction information encoding unit, which encodes the correction information generated by the correction information generating unit.

[0020] An image processing method according to one aspect of the present technology is an image processing method including: generating correction information which is information about correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image; and encoding the generated correction information.

[0021] An image processing device according to another aspect of the present technology is an image processing device including: a correction information decoding unit that decodes encoded data of correction information, the correction information being information about correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image; and a construction unit that constructs 3D data by using the 2D data and correction information generated by decoding the encoded data of the correction information by the correction information decoding unit.

[0022] An image processing method according to another aspect of the present technology is an image processing method, which includes: decoding encoded data of correction information, which is information about correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image; and constructing 3D data by using the 2D data and the correction information generated by decoding the encoded data of the correction information.

[0023] In an image processing device and an image processing method according to an aspect of the present technology, correction information about correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image is generated and the generated correction information is encoded.

[0024] In an image processing device and an image processing method according to another aspect of the present technology, encoded data of correction information is decoded, and 3D data is constructed using 2D data and correction information generated by decoding the encoded data of the correction information, the correction information being information about correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a diagram for explaining an example of the relationship between the patch range indicated by the occupancy map and the actual patch range.

[0026] Figure 2 It is a diagram for explaining an example of a correction method.

[0027] Figure 3 is a diagram showing candidate examples of removal methods.

[0028] Figure 4 is a diagram showing an example of the removal amount.

[0029] Figure 5 is a diagram showing an example of syntax.

[0030] Figure 6 is a block diagram showing a main configuration example of an encoding device.

[0031] Figure 7 is a block diagram showing a main configuration example of a correction information generating unit.

[0032] Figure 8 is a flowchart for explaining an example of the flow of encoding processing.

[0033] Fig. 9 : is a flowchart for explaining an example of the flow of correction information generation processing.

[0034] Fig.10 is a block diagram showing a main configuration example of a decoding device.

[0035] Fig.11 is a block diagram showing a main configuration example of a 3D reconstruction unit.

[0036] Fig.12 is a flowchart for explaining an example of the flow of decoding processing.

[0037] Fig.13: is a flowchart for explaining an example of the flow of 3D data reconstruction processing.

[0038] Fig.14 is a block diagram showing a main configuration example of a computer. DETAILED DESCRIPTION

[0039] The following description will explain a mode for carrying out the present disclosure (which will be referred to as an embodiment hereinafter). Note that the description will be given in the following order.

[0040] 1. Correction of 3D data reconstruction

[0041] 2. First Embodiment (Encoding Device)

[0042] 3. Second Embodiment (Decoding Device)

[0043] 4. Appendix

[0044] <1. Correction of 3D Data Reconstruction>

[0045] <Documents supporting technical content and technical terms, etc.>

[0046] The scope disclosed in the present technology includes not only the contents described in the embodiments but also the contents described in the following non-patent documents known at the time of filing.

[0047] Non-patent document 1: (as mentioned above)

[0048] Non-patent document 2: ITU (International Telecommunication Union) Telecommunication Standardization Sector, "Advanced video coding for generic audiovisual services", H.264, 04 / 2017

[0049] Non-patent document 3: ITU (International Telecommunication Union) Telecommunication Standardization Sector, "High efficiency video coding", H.265, 12 / 2016

[0050] Non-Patent Literature 4: Jianle Chen, Elena Alshina, Gary J. Sullivan, Jens-Rainer, Jill Boyce, “Algorithm Description of Joint Exploration Test Model 4,” JVET-G1001_v1, Joint Video Exploration Team (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 Seventh Meeting: Torino, IT, 13-21 July 2017

[0051] That is, the contents described in the above non-patent documents are also used as a basis for determining the support requirements. For example, the case where the quadtree block structure described in non-patent document 3 and the quadtree plus binary tree (QTBT) block structure described in non-patent document 4 are not directly described in the implementation also falls within the scope of the disclosure of the present technology and should meet the support requirements of the scope of the claims. In addition, for example, the case where technical terms such as parsing, grammar, or semantics are similarly not directly described in the implementation also falls within the scope of the disclosure of the present technology and should meet the support requirements of the scope of the claims.

[0052] <Point Cloud>

[0053] Conventionally, there has been data such as point clouds that represent a three-dimensional structure based on position information of a point group, attribute information, etc. The point cloud has a relatively simple data structure, and can express an arbitrary three-dimensional structure with sufficient accuracy by using a sufficient number of points.

[0054] <Overview of Video-Based Methods>

[0055] A video-based method has been proposed which projects the location information or attribute information of such a point cloud onto a two-dimensional plane for each small area, arranges the image (patch) projected onto the two-dimensional plane in a frame image, and encodes the frame image using an image encoding method for two-dimensional images.

[0056] In this video-based method, the input point cloud is divided into multiple small areas, and each point is projected on a two-dimensional plane for each small area (generating a patch). Since the point cloud is configured with the position information (geometry) and attribute information (texture) of each point, the projection on the two-dimensional plane is performed on each of the position information and attribute information.

[0057] Then, the patch is arranged in a frame image (two-dimensional image). That is, a frame image (also called a geometric video frame) in which patches of position information are arranged and a frame image (also called a color video frame) in which patches of attribute information are arranged are generated.

[0058] Note that in the geometry video frame, the position information of a point is expressed as position information in the depth direction (depth value (depth)). That is, each pixel value in the frame image indicates the depth value.

[0059] For example, each of the frame images is encoded using an encoding method for two-dimensional plane images such as Advanced Video Coding (AVC) or High Efficiency Video Coding (HEVC).

[0060] The encoded data generated in this way is sent to the decoding side and decoded at the decoding side to generate a frame image. Then, each patch is extracted from the frame image, and 3D data (point cloud, etc.) is reconstructed based on each patch (position information and attribute information).

[0061] <Occupy Map>

[0062] In addition, an occupancy map may also be used in the case of such a video-based method. An occupancy map is map information indicating whether a projection image (patches) of a geometric video frame with predetermined pixel precision exists. For example, an occupancy map indicates whether patches with N×N pixel precision (that is, each area has N×N pixels) exist. For example, in the occupancy map, an area of ​​N×N pixels in which a patch exists is indicated by a value of "1", and an area of ​​N×N pixels in which a patch does not exist is indicated by a value of "0".

[0063] Such an occupancy map is encoded as data independent of the geometry video frame or the color video frame, and is sent to the decoding side. Since the decoder can grasp whether there is a patch in the area by referring to the occupancy map, it is possible to suppress the influence of noise generated by encoding / decoding and restore 3D data more accurately. For example, even if the depth value changes due to encoding / decoding, the decoder can ignore the depth value of the area where there is no patch by referring to the occupancy map (not processing the depth value as the position information of the 3D data).

[0064] When such an occupancy map is used, the encoding efficiency can be improved by lowering the resolution and reducing the amount of information.

[0065] However, if the resolution of the occupancy map is reduced, there is a possibility that the patch range indicated by the occupancy map may not be consistent with the actual patch range, and points that originally did not exist may be generated during the reconstruction of the 3D data. Figure 1As shown in A, a lower-resolution occupied area is generated for pixels 11 near the patch boundary of the position information indicated by small squares. Fig.12 , there is the possibility that the patch may expand and may form a blank area where there was no patch originally ( Figure 1 ). Therefore, a method of moving the position of such a point by performing smoothing to reduce the blank has been considered. However, even if smoothing is performed, there is a possibility that the peak signal-to-noise ratio (PSNR) may deteriorate.

[0066] Therefore, as described in Non-Patent Document 5, a method of removing blank portions formed by reducing the resolution (N×N accuracy) of the occupied map has been considered. However, although the method described in Non-Patent Document 5 can be used to remove blank portions, the blank portions cannot be increased (expanded), and it is difficult to reduce holes (missing portions) formed in the displayed image due to gaps between patches, etc. Moreover, since overlapping of patches is suppressed by reducing the blank portions, there is a possibility that gaps between patches, etc. may be formed.

[0067] Therefore, a method of widening the boundary of a high-resolution occupancy map (1×1 accuracy) is also considered as described in Non-Patent Document 6. However, as described above, there is a possibility that an occupancy map with 1×1 accuracy may increase the amount of information.

[0068] Therefore, it is conceivable to reduce the blank portion at the patch boundary portion as in the method described in Non-Patent Document 5, and further widen the boundary as in the method described in Non-Patent Document 6. However, since the occupancy map is corrected in the methods described in Non-Patent Document 5 and Non-Patent Document 6, there is a possibility that another patch may be included in the area indicating the presence of one patch and the reconstructed 3D data may be degraded. For example, Figure 1 As shown in FIG. 1B , in a case where the positions of the patch 21 and the patch 22 are close to each other, when the area in the occupied map is enlarged, as shown by the dotted circle 23, the area where the patch 21 is located overlaps with the patch 22. In such a case, since the dotted circle 23 is intercepted as the area where the patch 21 is located, there is a possibility that the 3D data reconstructed based on the patch 21 may include information of the patch 22. Therefore, there is a possibility that the subjective image quality of the image (also referred to as the display image) obtained by projecting the reconstructed 3D data on the two-dimensional plane may be reduced.

[0069] Furthermore, since the method described in Non-Patent Document 6 does not support smoothing, it is difficult to suppress the formation of gaps between patches due to dot position shifts caused by smoothing. Therefore, there is a possibility that the gaps may reduce the subjective image quality of the displayed image.

[0070] <Method 1>

[0071] Therefore, if Figure 2 In method 1 in the table of , correction is performed in the conversion from 2D data to 3D data (reconstruction of 3D data) to suppress the formation of holes due to gaps between patches, etc. For example, correction information is generated, the correction information being information about the correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image, and the generated correction information is encoded. For example, the image processing device includes: a correction information generating unit that generates correction information, the correction information being information about the correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image; and a correction information encoding unit that encodes the correction information generated by the correction information generating unit.

[0072] Furthermore, for example, the coded data of the correction information is decoded, and the 3D data is constructed using the 2D data and the correction information generated by decoding the coded data of the correction information, the correction information being information about the correction of the 3D data representing the three-dimensional structure constructed using the 2D data representing the two-dimensional image. For example, the image processing apparatus includes: a correction information decoding unit that decodes the coded data of the correction information, the correction information being information about the correction of the 3D data representing the three-dimensional structure constructed using the 2D data representing the two-dimensional image; and a construction unit that constructs the 3D data by using the 2D data and the correction information generated by decoding the coded data of the correction information by the correction information decoding unit.

[0073] In this way, correction of patch boundaries can be achieved by correction of 3D data. Therefore, as in the case of correction of the occupancy map, degradation of 3D data can be suppressed, such as adding information about other patches from a patch to be processed during reconstruction of 3D data. That is, reduction in subjective image quality of a displayed image can be suppressed.

[0074] The correction information can be generated based on an occupancy map with 1×1 precision and an occupancy map with N×N precision. For example, the occupancy map is reduced in precision from 1×1 precision to N×N precision and encoded, and the correction information is generated based on the occupancy map with 1×1 precision before encoding and the occupancy map with N×N precision generated by decoding the encoded data of the occupancy map. By reducing the precision of the occupancy map and encoding the occupancy map in this way, a reduction in encoding efficiency can be suppressed. In addition, smoothing performed during encoding can be reflected in the correction information by generating the correction information using the occupancy maps before and after encoding. Therefore, a reduction in the subjective image quality of the displayed image due to smoothing can be suppressed.

[0075] In addition, correction information can also be generated based on a margin, which is a set value of the size of a blank, which is a portion of the 3D data that is enlarged by the occupancy map. In this way, the correction amount of the blank can also be controlled by setting the value "margin". For example, depending on the 3D data, a missing portion may be formed in the displayed image that is difficult to handle even by comparing the occupancy maps before and after simple encoding. Therefore, the formation of such a missing portion can be suppressed by setting the margin independently of the comparison result of the occupancy map. In other words, the reduction in the subjective image quality of the displayed image can be further suppressed. Note that the method of setting the margin is arbitrary. For example, the margin can be set based on instructions from the outside, such as a user, can be set based on the analysis results of the 3D data to be encoded, or can be set based on other information.

[0076] In addition, the correction information may include information on the correction of blanks, which are portions of the 3D data that are enlarged by the occupied map. Thus, based on the correction information, the correction of blanks may be reflected in the correction of the 3D data. That is, the correction of the patch boundaries may be achieved by the correction of the 3D data. Therefore, the reduction in the subjective image quality of the displayed image may be suppressed.

[0077] <Method 1-1>

[0078] For example, the information about the correction of the blank may include information indicating the removal method of the blank (information indicating how to remove the blank). Figure 2 The correction method is sent as in method 1-1 shown in the table of . By placing such information in the correction information, it is possible to control how to remove blanks.

[0079] In addition, the information indicating the method of removing the blank may have any content, and may include, for example, information indicating a mode of removing the blank selected from the candidates. For example, a plurality of "modes of removing the blank" may be prepared in advance as candidates, a "mode of removing the blank" may be selected from the candidates for each local area, and the correction information (information indicating the method of removing the blank) may include information indicating the candidate selected for each local area (for example, identification information of the selected candidate, etc.). By placing such information in the correction information, how to remove the blank can be controlled by simpler information. For example, compared with the case where the removal method of all local areas is specifically specified in units of one pixel, the increase in the amount of code can be suppressed.

[0080] <Method 1-2>

[0081] In addition, for example, the information on the correction of the blank may include information indicating the correction amount of the blank (information indicating how much blank is to be corrected). Figure 2By placing such information in the correction information, it is possible to control how much blanking is to be corrected.

[0082] Note that the information indicating the amount of correction of the blank may include information indicating the amount of removal of the blank. Through this information, how much of the blank is to be removed can be controlled. In addition, the information indicating the amount of correction of the blank may include information indicating the amount of increase of the blank. Through this information, how much of the blank is to be increased (increased) can be controlled.

[0083] <Method 1-3 and Method 1-4>

[0084] Furthermore, the accuracy of the occupancy map used to reconstruct the 3D data to be corrected based on the correction information on the decoding side is arbitrary. For example, N×N accuracy may be adopted. That is, Figure 2 The conversion (reconstruction of 3D data) using an occupancy map with N×N precision is corrected as in methods 1-3 shown in the table of . In addition, 1×1 precision may be used. That is, Figure 2 The conversion (reconstruction of 3D data) using an occupancy map with 1×1 accuracy is corrected as in methods 1 to 4 shown in the table of FIG.

[0085] <Whitespace removal mode>

[0086] Next, the candidates of the "blank removal method pattern" described in <Method 1-1> will be described. The candidate patterns (removal methods) can be any patterns. The number is also arbitrary. For example, Figure 3 A to Figure 3 The eight patterns shown in H can be candidates.

[0087] Figure 3 A to Figure 3 H each shows an example of a pattern of blank removal. Figure 3 A to Figure 3 In each of H, each square indicates a block (e.g., pixel) as a data unit occupying a map, and a black block is a block to be removed from the blank. That is, the black block is an empty block, that is, a block to which a value indicating that a patch does not exist is set.

[0088] For example, Figure 3 A shows a mode in which the lower right corner of the local part to be processed is removed. Similarly, Figure 3 B shows the upper right pattern removed, Figure 3 C shows the upper left pattern removed, and Figure 3 D shows the lower left pattern removed.

[0089] In addition, for example, Figure 3 E shows the pattern where the right side of the local part to be processed is removed, Figure 3 F shows the pattern with the upper side removed, Figure 3 G shows the pattern with the left side removed, and Figure 3 The H shows the pattern with the lower side removed. Note that the grey blocks in the figure are fully occupied blocks or empty blocks.

[0090] That is, in the case of this example, which portion is to be removed is selected depending on which candidate is selected.

[0091] <Blank Correction Amount>

[0092] Next, the "amount of blanks removed" described in <Method 1-2> will be described. The amount of blanks removed can be any value. For example, Figure 4 The right side of the pattern shown in A is removed ( Figure 3 In the case of the mode in E of FIG. 1 and in the case where the correction amount of the blank of the area to be processed (current area) including 4×4 pixels is “0” (that is, the initial value (Default)), as Figure 4 As shown in B, the right half pixel (the 2×4 pixel area on the right) is removed. Similarly, when the blank correction amount is "-1", as shown in Figure 4 As shown in C, the rightmost pixel string of the current area (the rightmost 1×4 pixel area) is removed. Similarly, when the blank correction amount is "-2", as shown in Figure 4 As shown in D, the current area is not corrected (the rightmost 0×4 pixel area is removed). In addition, when the blank correction amount is "-3", as shown in Figure 4 As shown in E, a 1×4 pixel area is added to the right end of the current area. In other words, the blank space is expanded.

[0093] In this manner, the correction amount of the blank may indicate the amount of blank removal or the amount of blank addition.

[0094] <Syntax>

[0095] exist Figure 5 An example of the syntax used for the above correction information is shown in A of . The gray part is the syntax related to the correction information. The removal method (type) of blanks at the frame level and patch level and the processing of each type are signaled by omapFilterTypeFrameLevel / omapFilterTypePatchLevel. In addition, the correction amount (amount to be removed / amount to be added) at the frame level and each patch is signaled by omapFilterValueFrameLevel / patchList[i].omapFilterValue. As shown in Figure 5As shown in the conversion table shown in B, this value can be used as an index representing the difference from the initial value (Default).

[0096] <2. First Embodiment>

[0097] <Encoding device>

[0098] Figure 6 : is a block diagram showing an example of the configuration of an encoding device as one aspect of an image processing device to which the present technology is applied. Figure 6 The illustrated encoding device 100 is a device that projects 3D data such as a point cloud onto a two-dimensional plane and encodes it using an encoding method for two-dimensional images (an encoding device to which a video-based method is applied).

[0099] Notice, Figure 6 The main processing units, main data streams, etc. are shown, and may not be all shown. That is, in the encoding device 100, there may be Figure 6 The processing units shown in the form of blocks in the figure may exist or may not be Figure 6 100. The processing or data flow is shown in the form of arrows, etc. This is similar in other figures used to explain the processing units, etc. in the encoding device 100.

[0100] like Figure 6 As shown, the encoding device 100 includes a patch decomposition unit 111, a packing unit 112, an auxiliary patch information compression unit 113, a video encoding unit 114, a video encoding unit 115, an OMap encoding unit 116, a multiplexer 117, a decoding unit 118 and a correction information generation unit 119.

[0101] The patch decomposition unit 111 performs processing related to decomposition of 3D data. For example, the patch decomposition unit 111 acquires 3D data (e.g., point cloud) representing a three-dimensional structure input to the encoding device 100 (arrow 121). In addition, the patch decomposition unit 111 decomposes the acquired 3D data into a plurality of small areas (connected components), projects the 3D data on a two-dimensional plane for each small area, and generates a patch of position information or a patch of attribute information.

[0102] The patch decomposing unit 111 supplies information on each generated patch to the packing unit 112 (arrow 122 ). Furthermore, the patch decomposing unit 111 supplies auxiliary patch information as information on the decomposition to the auxiliary patch information compressing unit 113 (arrow 123 ).

[0103] The packing unit 112 performs processing related to data packing. For example, the packing unit 112 obtains information about the patch provided from the patch decomposition unit 111 (arrow 122). In addition, the packing unit 112 arranges each acquired patch on the two-dimensional image and packs it into a video frame. For example, the packing unit 112 packs patches of position information (geometry) indicating the position of a point into a video frame and generates a geometry video frame. In addition, the packing unit 112 packs patches of attribute information (texture) such as color information added to the position information into a video frame and generates a color video frame. In addition, the packing unit 112 generates an occupancy map with 1×1 precision indicating whether a patch exists.

[0104] The packing unit 112 supplies them to the subsequent processing units (arrow 124). For example, the packing unit 112 supplies the geometry video frame to the video encoding unit 114, supplies the color video frame to the video encoding unit 115, and supplies the occupancy map (1×1Omap) with 1×1 precision to the OMap encoding unit 116. In addition, the packing unit 112 supplies control information about packing to the multiplexer 117 (arrow 125). In addition, the packing unit 112 supplies the occupancy map with 1×1 precision to the correction information generation unit 119.

[0105] The auxiliary patch information compression unit 113 performs processing related to compression of the auxiliary patch information. For example, the auxiliary patch information compression unit 113 obtains the auxiliary patch information provided from the patch decomposition unit 111 (arrow 123). In addition, the auxiliary patch information compression unit 113 obtains the correction information provided from the correction information generation unit 119 (arrow 133). The auxiliary patch information compression unit 113 encodes (compresses) the obtained auxiliary patch information, and generates encoded data of the auxiliary patch information. In addition, the auxiliary patch information compression unit 113 encodes (compresses) the obtained correction information to generate encoded data of the correction information, and places the encoded data of the correction information in the encoded data of the auxiliary patch information. The auxiliary patch information compression unit 113 provides the encoded data of the auxiliary patch information generated in this way to the multiplexer 117 (arrow 126).

[0106] The video encoding unit 114 performs processing related to encoding of the geometry video frame. For example, the video encoding unit 114 obtains the geometry video frame provided from the packing unit 112 (arrow 124). In addition, for example, the video encoding unit 114 encodes the obtained geometry video frame using any encoding method for two-dimensional images such as AVC or HEVC, and generates encoded data of the geometry video frame. The video encoding unit 114 provides the generated encoded data of the geometry video frame to the multiplexer 117 (arrow 127). In addition, the video encoding unit 114 provides the encoded data of the geometry video frame to the decoding unit 118 (arrow 134).

[0107] The video encoding unit 115 performs processing related to the encoding of the color video frame. For example, the video encoding unit 115 obtains the color video frame provided from the packing unit 112 (arrow 124). In addition, the video encoding unit 115 obtains the decoded geometry video frame provided from the decoding unit 118 (arrow 135). Then, the video encoding unit 115 recolors the acquired color video frame by using the acquired geometry video frame, and makes the attribute information correspond to the position information after decoding. That is, the video encoding unit 115 associates the attribute information with the position information updated by smoothing in the encoding. In addition, for example, the video encoding unit 115 encodes the color video frame recolored in this way using any encoding method for two-dimensional images such as AVC or HEVC, and generates encoded data of the color video frame. The video encoding unit 115 provides the encoded data of the generated color video frame to the multiplexer 117 (arrow 128).

[0108] The OMap encoding unit 116 performs processing related to encoding of a video frame of an occupancy map. For example, the OMap encoding unit 116 obtains an occupancy map with 1×1 precision provided from the packing unit 112 (arrow 124). The OMap encoding unit 116 reduces the precision of the obtained occupancy map with 1×1 precision and generates an occupancy map with N×N precision. Then, for example, the OMap encoding unit 116 encodes the generated occupancy map with N×N precision using any encoding method such as arithmetic coding, and generates encoded data of the occupancy map with N×N precision. In addition, the OMap encoding unit 116 provides the generated encoded data of the occupancy map with N×N precision to the multiplexer 117 (arrow 129). In addition, the OMap encoding unit 116 also provides the encoded data to the decoding unit 118 (arrow 131).

[0109] The multiplexer 117 performs processing related to multiplexing. For example, the multiplexer 117 obtains the encoded data of the auxiliary patch information provided from the auxiliary patch information compression unit 113 (arrow 126). In addition, for example, the multiplexer 117 obtains the control information about the packing provided from the packing unit 112 (arrow 125). In addition, for example, the multiplexer 117 obtains the encoded data of the geometry video frame provided from the video encoding unit 114 (arrow 127). In addition, for example, the multiplexer 117 obtains the encoded data of the color video frame provided from the video encoding unit 115 (arrow 128). In addition, for example, the multiplexer 117 obtains the encoded data of the occupancy map provided from the OMap encoding unit 116 (arrow 129).

[0110] The multiplexer 117 multiplexes the acquired information to generate a bit stream. The multiplexer 117 outputs the generated bit stream to the outside of the encoding device 100.

[0111] The decoding unit 118 performs processing related to decoding of the encoded data generated as described above. For example, the decoding unit 118 obtains the encoded data of the occupancy map with N×N precision provided from the OMap encoding unit 116 (arrow 131). In addition, the decoding unit 118 obtains the encoded data of the geometry video frame provided from the video encoding unit 114 (arrow 134). Then, the decoding unit 118 has a configuration similar to that of the decoding device 200 described later, and performs similar processing. For example, the decoding unit 118 decodes the encoded data of the occupancy map with N×N precision and generates an occupancy map with N×N precision. The decoding unit 118 provides the occupancy map with N×N precision to the correction information generation unit 119 (arrow 132). In addition, the decoding unit 118 decodes the encoded data of the geometry video frame and generates a geometry video frame. The decoding unit 118 provides the geometry video frame to the video encoding unit 115 (arrow 135).

[0112] The correction information generation unit 119 performs processing related to the generation of correction information, which is information about the correction of 3D data representing a three-dimensional structure reconstructed using 2D data representing a two-dimensional image. For example, the correction information generation unit 119 obtains an occupancy map with 1×1 precision provided from the packing unit 112 (arrow 124). In addition, the correction information generation unit 119 obtains an occupancy map with N×N precision provided from the decoding unit 118 (arrow 132). In addition, the correction information generation unit 119 obtains the setting of the margin to be input from the outside (for example, a user, etc.) (not shown). The correction information generation unit 119 generates correction information based on the information. For example, the correction information generation unit 119 generates correction information including information about the correction of a blank, which is a portion of the 3D data enlarged by the occupancy map. The correction information generation unit 119 provides the generated correction information to the auxiliary patch information compression unit 113 (arrow 133).

[0113] In this way, the encoding device 100 can cause the decoding side to correct the 3D data based on the correction information. Therefore, the degradation of the 3D data can be suppressed, and the reduction in the subjective image quality of the display image can be suppressed.

[0114] Note that these processing units (from the patch decomposition unit 111 to the correction information generation unit 119) have any configuration. For example, each processing unit may be configured with a logic circuit that implements the above-mentioned processing. In addition, each processing unit may have, for example, a central processing unit (CPU), a read-only memory (ROM), a random access memory (RAM), etc., and use them to execute a program to implement the above-mentioned processing. Of course, each processing unit may have both configurations, and part of the above-mentioned processing may be implemented by a logic circuit, while another part may be implemented by executing a program. The configurations of the individual processing units may be independent of each other, and for example, part of the above-mentioned processing may be implemented by the logic circuits of some processing units, the above-mentioned processing may be implemented by some other processing units that execute a program, and the above-mentioned processing may be implemented by another other processing unit that executes both the logic circuit and the program.

[0115] <Correction Information Generating Unit>

[0116] Figure 7 1 is a block diagram showing a main configuration example of the correction information generating unit 119. Figure 7 As shown, the correction information generating unit 119 has a type setting unit 151 and a removal amount setting unit 152 .

[0117] The type setting unit 151 sets a blank removal method (blank correction type). For example, the type setting unit 151 selects an optimal mode from among candidates of "blank removal method modes" prepared in advance. For example, the type setting unit 151 selects such a mode (selects a type) based on the comparison result of the occupation map or the setting of the margin as described above.

[0118] The removal amount setting unit 152 sets the correction amount of blank (how much blank is to be corrected). For example, the removal amount setting unit 152 sets the removal amount of blank (how much blank is to be removed). In addition, for example, the removal amount setting unit 152 sets the increase amount of blank (how much blank is to be increased). The removal amount setting unit 152 sets such a correction amount based on the comparison result of the occupation map as described above or the setting of the margin.

[0119] The correction information generation unit 119 supplies information indicating the correction type or correction amount of the blank as described above to the auxiliary patch information compression unit 113 as correction information.

[0120] <Encoding Process Flow>

[0121] The processing to be performed by such an encoding device 100 will be described. First, reference will be made to Figure 8 The flowchart describes an example of the flow of the encoding process.

[0122] When the encoding process starts, in step S101 , the patch decomposition unit 111 of the encoding device 100 performs patch decomposition processing to decompose 3D data into patches and project the data of each patch onto a two-dimensional plane.

[0123] In step S102, the packing unit 112 packs the 3D data projected on the two-dimensional plane for each patch passed through the decomposition unit 111 and generates a geometry video frame or a color video frame. In addition, the packing unit 112 generates an occupancy map with 1×1 precision.

[0124] In step S103, the OMap encoding unit 116 reduces the occupancy map with 1×1 precision generated in step S102 in precision to generate an occupancy map with N×N precision, and encodes the occupancy map with N×N precision to generate encoded data of the occupancy map with N×N precision.

[0125] In step S104 , the decoding unit 118 decodes the encoded data of the occupancy map with N×N precision generated in step S103 , and generates an occupancy map with N×N precision.

[0126] In step S105, the correction information generation unit 119 performs correction information generation processing and generates correction information based on the occupancy map with 1×1 precision generated in step S102, the occupancy map with N×N precision after decoding generated in step S104, and the margin. This correction information generation processing will be described later.

[0127] In step S106 , the packing unit 112 expands the geometric video frame generated in step S102 .

[0128] In step S107 , the video encoding unit 114 encodes the geometry video frame expanded in step S106 and generates encoded data of the geometry video frame.

[0129] In step S108 , the video encoding unit 114 decodes the encoded data of the geometry video frame generated in step S107 , and generates a decoded geometry video frame.

[0130] In step S109 , the packing unit 112 expands the color video frame generated in step S102 .

[0131] In step S110 , the video encoding unit 115 recolors the color video frame expanded in step S109 by using the decoded geometry video frame generated in step S108 , and makes the attribute information correspond to the decoded position information.

[0132] In step S111 , the video encoding unit 115 encodes the color video frame recolored in step S110 , and generates encoded data of the color video frame.

[0133] In step S112 , the auxiliary patch information compressing unit 113 encodes (compresses) the auxiliary patch information including the correction information generated by the process of step S105 , and generates encoded data.

[0134] In step S113, the multiplexer 117 multiplexes various information generated as described above and generates a bit stream including the information. In step S114, the multiplexer 117 outputs the bit stream generated by the process of step S113 to the outside of the encoding device 100. When the process of step S113 is completed, the encoding process is completed.

[0135] <Flow of Correction Information Generation Processing>

[0136] Will refer to Fig. 9 A flowchart is used to describe the Figure 8105. When the correction information generation process starts, in step S131, the type setting unit 151 of the correction information generation unit 119 sets the correction type of the blank. In step S132, the removal amount setting unit 152 sets the correction amount of the blank (e.g., the removal amount). In step S133, the correction information generation unit 119 sets such information as correction information and provides the information to the auxiliary patch information compression unit 113.

[0137] When the processing of step S133 ends, the correction information generation processing ends, and the processing returns to Figure 8 .

[0138] By performing each process as described above, the encoding device 100 can generate correction information for 3D data and cause the decoding side to correct the 3D data based on the correction information. Therefore, degradation of 3D data can be suppressed, and reduction in subjective image quality of the displayed image can be suppressed.

[0139] <3. Second Embodiment>

[0140] <Decoding device>

[0141] Fig.10 : is a block diagram showing an example of the configuration of a decoding device as one aspect of an image processing device to which the present technology is applied. Fig.10 The decoding device 200 shown is a device that decodes coded data obtained by projecting 3D data such as a point cloud on a two-dimensional plane using a decoding method for two-dimensional images and reconstructs 3D data (a decoding device to which a video-based method is applied). Figure 6 The decoding device corresponds to the encoding device 100 in the embodiment, and can decode the bit stream generated by the encoding device 100 to reconstruct the 3D data.

[0142] Notice, Fig.10 The main processing units, main data streams, etc. are shown, but not all of them may be shown. That is, in the decoding device 200, there may be Fig.10 The processing units shown in the form of blocks in the figure may exist or may not be Fig.10 200. The processing or data flow is shown in the form of arrows, etc. This is similar in other figures used to explain the processing units, etc. in the decoding device 200.

[0143] like Fig.10 As shown, the decoding device 200 includes a demultiplexer 211, an auxiliary slice information decoding unit 212, a video decoding unit 213, a video decoding unit 214, an OMap decoding unit 215, a depacketizing unit 216 and a 3D reconstruction unit 217.

[0144] The demultiplexer 211 performs processing related to data demultiplexing. For example, the demultiplexer 211 obtains a bit stream input to the decoding device 200. The bit stream is provided, for example, from the encoding device 100. The demultiplexer 211 demultiplexes the bit stream, extracts the encoded data of the auxiliary patch information, and provides it to the auxiliary patch information decoding unit 212. In addition, the demultiplexer 211 extracts the encoded data of the geometry video frame from the bit stream by demultiplexing, and provides it to the video decoding unit 213. In addition, the demultiplexer 211 extracts the encoded data of the color video frame from the bit stream by demultiplexing, and provides it to the video decoding unit 214. In addition, the demultiplexer 211 extracts the encoded data of the occupancy map from the bit stream by demultiplexing, and provides it to the OMap decoding unit 215. In addition, the demultiplexer 211 extracts control information about packing from the bit stream by demultiplexing, and provides it to the unpacking unit 216.

[0145] The auxiliary patch information decoding unit 212 performs processing related to decoding of the encoded data of the auxiliary patch information (including the encoded data of the correction information). For example, the auxiliary patch information decoding unit 212 obtains the encoded data of the auxiliary patch information provided from the demultiplexer 211. In addition, the auxiliary patch information decoding unit 212 decodes the encoded data and generates auxiliary patch information or correction information. The correction information is information generated on the encoding side and sent from the encoding side, and includes, for example, information about correction of a blank, which is a portion of the 3D data enlarged by the occupation map. The auxiliary patch information decoding unit 212 provides the auxiliary patch information to the 3D reconstruction unit 217.

[0146] The video decoding unit 213 performs processing related to decoding of the encoded data of the geometry video frame. For example, the video decoding unit 213 obtains the encoded data of the geometry video frame provided from the demultiplexer 211. In addition, for example, the video decoding unit 213 decodes the encoded data and generates a geometry video frame. The video decoding unit 213 provides the geometry video frame to the depacketizing unit 216.

[0147] The video decoding unit 214 performs processing related to decoding of the encoded data of the color video frame. For example, the video decoding unit 214 obtains the encoded data of the color video frame provided from the demultiplexer 211. In addition, for example, the video decoding unit 214 decodes the encoded data and generates a color video frame. The video decoding unit 214 provides the color video frame to the depacketizing unit 216.

[0148] The OMap decoding unit 215 performs processing related to decoding of the encoded data of the occupancy map. For example, the OMap decoding unit 215 obtains the encoded data of the occupancy map provided from the demultiplexer 211. In addition, for example, the OMap decoding unit 215 decodes the encoded data and generates an occupancy map. The OMap decoding unit 215 provides the occupancy map to the depacketizing unit 216.

[0149] The unpacking unit 216 performs processing related to unpacking. For example, the unpacking unit 216 obtains control information about packing provided from the demultiplexer 211. In addition, the unpacking unit 216 obtains a geometry video frame provided from the video decoding unit 213. In addition, the unpacking unit 216 obtains a color video frame provided from the video decoding unit 214. In addition, the unpacking unit 216 obtains an occupancy map provided from the OMap decoding unit 215. The unpacking unit 216 unpacks the geometry video frame or the color video frame based on the obtained control information or occupancy map, and extracts patches of position information or attribute information, etc. The unpacking unit 216 provides patches of position information or attribute information, etc. to the 3D reconstruction unit 217.

[0150] The 3D reconstruction unit 217 performs processing related to the reconstruction of 3D data. For example, the 3D reconstruction unit 217 obtains the auxiliary patch information and the correction information provided from the auxiliary patch information decoding unit 212. In addition, the 3D reconstruction unit 217 obtains the patch of position information or attribute information, etc. provided from the unpacking unit 216. In addition, the 3D reconstruction unit 217 reconstructs 3D data (for example, point cloud) based on such information. For example, the 3D reconstruction unit 217 reconstructs 3D data based on the auxiliary patch information, the patch of position information or attribute information, etc., and further corrects the 3D data based on the correction information. The 3D reconstruction unit 217 outputs the 3D data obtained by such processing to the outside of the decoding device 200.

[0151] For example, the 3D data is provided to a display unit so that an image is displayed, recorded on a recording medium, or provided to another device through communication.

[0152] In this way, the decoding device 200 can correct the 3D data based on the correction information provided from the encoding side. Therefore, the degradation of the 3D data can be suppressed, and the reduction in the subjective image quality of the display image can be suppressed.

[0153] Note that these processing units (from the demultiplexer 211 to the 3D reconstruction unit 217) have any configuration. For example, each processing unit may be configured with a logic circuit that implements the above-mentioned processing. In addition, each processing unit may, for example, have a CPU, ROM, RAM, etc., and use them to execute a program to implement the above-mentioned processing. Of course, each processing unit may have both configurations, and part of the above-mentioned processing may be implemented by a logic circuit, while another part may be implemented by executing a program. The configurations of the various processing units may be independent of each other, and, for example, part of the above-mentioned processing may be implemented by the logic circuits of some processing units, the above-mentioned processing may be implemented by some other processing units that execute a program, and the above-mentioned processing may be implemented by another other processing unit that executes both the logic circuit and the program.

[0154] <3D Reconstruction Unit>

[0155] Fig.11 It is shown Fig.10 2 is a block diagram of a main configuration example of the 3D reconstruction unit 217 in FIG. Fig.11 As shown, the 3D reconstruction unit 217 has a correction setting unit 251 and a reconstruction unit 252 .

[0156] The correction setting unit 251 performs processing related to control of correction of 3D data. For example, the correction setting unit 251 acquires an occupancy map with N×N precision after decoding provided from the OMap decoding unit 215. In addition, the correction setting unit 251 acquires correction information provided from the auxiliary patch information decoding unit 212.

[0157] The correction information may include, for example, information indicating a set blank removal method (blank correction type). For example, the correction information may include information indicating an optimal mode selected from candidates of "blank removal method modes" prepared in advance.

[0158] Furthermore, the correction information may include, for example, information indicating a correction amount (eg, a removal amount or an addition amount, or both) of the set blank.

[0159] The correction setting unit 251 sets how to perform correction based on the information. The correction setting unit 251 generates control information to implement correction according to the setting, and supplies the control information to the reconstruction unit 252.

[0160] The reconstruction unit 252 performs processing related to reconstruction of 3D data. For example, the reconstruction unit 252 reconstructs 3D data based on position information extracted from the geometry video frame provided by the unpacking unit 216 or attribute information extracted from the color video frame provided by the unpacking unit 216. In addition, the reconstruction unit 252 corrects the 3D data according to the control of the correction setting unit 251 (according to the control information provided from the correction setting unit 251). The reconstruction unit 252 outputs the generated (corrected) 3D data to the outside of the decoding device 200.

[0161] <Decoding Process Flow>

[0162] The processing to be performed by such a decoding device 200 will be described. First, reference will be made to Fig.12 An example of the flow of the decoding process is described with reference to the flowchart of FIG.

[0163] When the decoding process starts, in step S201 , the demultiplexer 211 of the decoding device 200 demultiplexes a bit stream.

[0164] In step S202 , the auxiliary patch information decoding unit 212 decodes the encoded data of the auxiliary patch information extracted from the bit stream obtained by the process of step S201 , and generates auxiliary patch information and correction information.

[0165] In step S203 , the OMap decoding unit 215 decodes the encoded data of the occupancy map extracted from the bit stream obtained by the process of step S201 .

[0166] In step S204 , the video decoding unit 213 decodes the encoded data of the geometry video frame extracted from the bit stream obtained through the process of step S201 , and generates a geometry video frame.

[0167] In step S205 , the video decoding unit 214 decodes the encoded data of the color video frame extracted from the bit stream obtained by the process of step S201 , and generates a color video frame.

[0168] In step S206 , the unpacking unit 216 unpacks each of the geometry video frame and the color video frame based on the control information about the packing and the occupancy map.

[0169] In step S207, the 3D reconstruction unit 217 performs 3D data reconstruction processing, and reconstructs 3D data such as a point cloud based on the auxiliary patch information generated in step S202 and the various information generated in steps S203 to S205, for example. In addition, the 3D reconstruction unit 217 corrects the reconstructed 3D data based on the correction information generated in step S202. When the processing of step S207 is completed, the decoding processing is completed.

[0170] <Flow of 3D data reconstruction processing>

[0171] Next, we will refer to Fig.13 The flowchart describes the Fig.12 An example of the flow of the 3D data reconstruction process performed in step S207 of FIG.

[0172] When the 3D data reconstruction process starts, the correction setting unit 251 sets a correction method related to 3D data reconstruction in step S221 based on the occupancy map with N×N precision after decoding and the correction information, and generates control information.

[0173] In step S222, the reconstruction unit 252 reconstructs 3D data such as a point cloud by using the position information (geometric data), the attribute information (picture data), and the correction method set in step S221. More specifically, the reconstruction unit 252 reconstructs the 3D data based on the position information (geometric data), the attribute information (picture data), the auxiliary patch information, etc. Then, the reconstruction unit 252 corrects the reconstructed 3D data according to the control information provided from the correction setting unit 251.

[0174] By performing each process as described above, the decoding device 200 can correct the 3D data based on the correction information provided from the encoding side. Therefore, the degradation of the 3D data can be suppressed, and the reduction in the subjective image quality of the display image can be suppressed.

[0175] <4. Appendix>

[0176] <Control Information>

[0177] Control information related to the present technology described in each of the above embodiments may be sent from the encoding side to the decoding side. For example, control information (e.g., enabled_flag) for controlling whether to allow (or prohibit) the application of the above technology may be sent. In addition, for example, control information specifying a range (e.g., an upper limit or a lower limit of a block size or both, a slice, a picture, a sequence, a component, a view, a layer, etc.) for allowing (or prohibiting) the application of the above technology may be sent.

[0178] <Computer>

[0179] The above series of processing can be performed by hardware or software. In the case where a series of processing is performed by software, the program constituting the software is installed on a computer. Here, the computer includes, for example, a computer embedded in dedicated hardware, a general-purpose personal computer capable of performing various functions by installing various programs, etc.

[0180] Fig.14 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes in a programmatic manner.

[0181] exist Fig.14 In the illustrated computer 900 , a central processing unit (CPU) 901 , a read only memory (ROM) 902 , and a random access memory (RAM) 903 are connected to one another via a bus 904 .

[0182] The input / output interface 910 is also coupled to the bus 904. An input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected to the input / output interface 910.

[0183] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a nonvolatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0184] In the computer configured as described above, for example, the CPU 901 loads a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executes the program to perform the above-described series of processes. The RAM 903 also appropriately stores data and the like necessary for the CPU 901 to perform various processes.

[0185] The program to be executed by the computer may be recorded and applied to, for example, the removable medium 921 which is a package medium or the like. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by mounting the removable medium 921 in the drive 915 .

[0186] Furthermore, the program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0187] Furthermore, the program may also be installed in the ROM 902 or the storage unit 913 in advance.

[0188] <Target of application of this technology>

[0189] Although the above description illustrates the case where the present technology is applied to encoding / decoding of point cloud data, the present technology is not limited to these examples and can be applied to encoding / decoding of 3D data of any standard. That is, various processes such as encoding / decoding methods and specifications of various data such as 3D data or metadata are arbitrary as long as they do not conflict with the above-mentioned present technology. In addition, part of the above-mentioned processes and specifications can be omitted as long as they do not conflict with the present technology.

[0190] Furthermore, although the above description explains the encoding device 100 and the decoding device 200 as application examples of the present technology, the present technology can be applied to any configuration.

[0191] For example, the present technology can be applied to various electronic devices such as transmitters or receivers (e.g., television receivers or mobile phones) used for satellite broadcasting, cable broadcasting such as cable TV, distribution on the Internet, and distribution to terminals via cellular communications, etc., or the present technology can be applied to devices that record images on media such as optical disks, magnetic disks, and flash memories, or reproduce images from these storage media (e.g., hard disk recorders or camera devices).

[0192] In addition, for example, the present technology can be implemented as part of the configuration of the following devices, such as a processor (e.g., a video processor) of a system large-scale integration (LSI) or the like, a module using multiple processors or the like (e.g., a video module), a unit using multiple modules or the like (e.g., a video unit), or a collection of units to which other functions are further added (e.g., a video collection).

[0193] In addition, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology can be implemented as cloud computing shared and jointly processed by multiple devices via a network. For example, the present technology can be implemented in a cloud service for providing image (moving image) related services to any terminal such as a computer, an audio-visual (AV) device, a portable information processing terminal, and an Internet of Things (IoT) device.

[0194] Note that in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), and it is not important whether all components are placed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network and a device in which multiple modules are housed in one housing are both systems.

[0195] <Fields and applications where this technology can be applied>

[0196] Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, health care, crime prevention, agriculture, animal husbandry, mining, beauty, factories, household appliances, weather or nature monitoring. In addition, its application is also arbitrary.

[0197] <Others>

[0198] Note that in this specification, a "flag" is information for identifying multiple states, and includes not only information for identifying two states of true (1) or false (0), but also information capable of identifying three or more states. Therefore, the "flag" can take, for example, two values ​​1 / 0 or 3 or more values. In other words, the number of bits configuring the "flag" is arbitrary, and can be one or more bits. In addition, since it is assumed that the identification information (including the flag) includes not only the identification information included in the bit stream, but also the difference information of the identification information relative to the specific reference information included in the bit stream, the "flag" and "identification information" in this specification include not only information, but also the difference information relative to the reference information.

[0199] In addition, various information (metadata, etc.) about the coded data (bitstream) can be sent or recorded in any form, as long as the information is associated with the coded data. Here, the term "association" means, for example, making other data available for (linkable to) processing one data. In other words, data associated with each other can be combined into one data or can be separate data. For example, information associated with the coded data (image) can be transmitted on a transmission path different from the transmission path of the coded data (image). In addition, for example, information associated with the coded data (image) can be recorded on a recording medium different from the recording medium of the coded data (image) (or another recording area of ​​the same recording medium). Note that the "association" can be a part of the data, not the entire data. For example, an image and information corresponding to the image can be associated with each other in any unit such as multiple frames, one frame, or a portion of a frame.

[0200] Note that terms such as "synthesize", "multiplex", "add", "integrate", "put in", "store", "bring in", "accept" and "insert" in this specification mean combining multiple objects into one object, for example, combining encoded data and metadata into one data, and mean a method of the above-mentioned "association".

[0201] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made without departing from the gist of the present technology.

[0202] For example, a configuration described as one device (or processing unit) may be divided to be configured as a plurality of devices (or processing units). Conversely, the configuration described above as a plurality of devices (or processing units) may be combined to be configured as one device (or processing unit). In addition, of course, configurations other than the above configurations may also be added to the configuration of each device (or each processing unit). In addition, a portion of the configuration of one device (or processing unit) may be included in the configuration of another device (or another processing unit), as long as the configuration and operation as the entire system are substantially the same.

[0203] In addition, for example, the above-mentioned program can be executed in any device. In this case, the device only needs to have necessary functions (functional blocks, etc.) so that necessary information can be obtained.

[0204] In addition, for example, each step of a flowchart may be performed by one device, or may be shared and performed by multiple devices. In addition, in the case where a step includes multiple processes, the multiple processes may be performed by one device, or may be shared and performed by multiple devices. In other words, the multiple processes included in a step may be performed as processes of multiple steps. Conversely, the processes described as multiple steps may also be performed together as one step.

[0205] In addition, in a program executed by a computer, for example, the processing of the steps describing the program may be performed in chronological order in the order described in this specification, or may be performed in parallel, or may be performed separately at a required timing (for example, when a call is made). That is, as long as there is no contradiction, the processing of the individual steps may be performed in an order different from the above order. In addition, the processing of the steps describing the program may be performed in parallel with the processing of another program, or may be performed in combination with the processing of another program.

[0206] In addition, for example, as long as there is no contradiction, multiple technologies related to the present technology can be implemented separately and independently. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. In addition, part or all of any of the technologies described above can also be implemented in combination with other technologies not described above.

[0207] Note that the present technology can have the following configurations.

[0208] (1) An image processing device comprising:

[0209] a correction information generating unit that generates correction information, the correction information being information on correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image; and

[0210] A correction information encoding unit encodes the correction information generated by the correction information generating unit.

[0211] (2) The image processing device according to (1),

[0212] The correction information generating unit generates the correction information based on an occupancy map with 1×1 precision and an occupancy map with N×N precision.

[0213] (3) The image processing device according to (2),

[0214] The correction information generation unit further generates the correction information based on a margin which is a setting value of a size of a blank which is a portion of the 3D data enlarged by occupying a map.

[0215] (4) The image processing device according to (1),

[0216] The correction information includes information on correction of a blank, which is a portion of the 3D data enlarged by occupying a map.

[0217] (5) The image processing device according to (4),

[0218] The information on the correction of the blank includes information indicating a method of removing the blank.

[0219] (6) The image processing device according to (5),

[0220] The information on the correction of the blank includes information indicating a mode of a method of removing the blank selected from the candidates.

[0221] (7) The image processing device according to (4),

[0222] The information on the correction of the blank includes information indicating the correction amount of the blank.

[0223] (8) The image processing device according to (7),

[0224] The information indicating the amount of blank correction includes information indicating the amount of blank removal.

[0225] (9) The image processing device according to (7),

[0226] The information indicating the correction amount of the blank includes information indicating the increase amount of the blank.

[0227] (10) An image processing method, comprising:

[0228] generating correction information, the correction information being information on correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image; and

[0229] The generated correction information is encoded.

[0230] (11) An image processing device comprising:

[0231] a correction information decoding unit that decodes encoded data of correction information, the correction information being information about correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image; and

[0232] A construction unit that constructs 3D data by using the 2D data and the correction information generated by decoding the encoded data of the correction information by the correction information decoding unit.

[0233] (12) The image processing device according to (11),

[0234] The correction information includes information on correction of a blank, which is a portion of the 3D data enlarged by occupying a map.

[0235] (13) The image processing device according to (12),

[0236] The information on the correction of the blank includes information indicating a method of removing the blank.

[0237] (14) The image processing device according to (13),

[0238] The information on the correction of the blank includes information indicating a mode of a method of removing the blank selected from the candidates.

[0239] (15) The image processing device according to (12),

[0240] The information on the correction of the blank includes information indicating the correction amount of the blank.

[0241] (16) The image processing device according to (15),

[0242] The information indicating the amount of blank correction includes information indicating the amount of blank removal.

[0243] (17) The image processing device according to (15),

[0244] The information indicating the correction amount of the blank includes information indicating the increase amount of the blank.

[0245] (18) The image processing device according to (11), further comprising:

[0246] a video decoding unit that decodes the encoded data of the 2D data;

[0247] The construction unit constructs the 3D data by using the 2D data decoded and generated by the video decoding unit and the correction information generated by decoding the coded data by the correction information decoding unit.

[0248] (19) The image processing device according to (11), further comprising:

[0249] OMap decoding unit, which decodes the encoded data occupying the map;

[0250] The construction unit corrects a blank, which is a portion enlarged by an occupation map decoded and generated by the OMap decoding unit during construction from 2D data to 3D data, by using the correction information generated by the correction information decoding unit decoding coded data of the correction information.

[0251] (20) An image processing method, comprising:

[0252] decoding coded data of correction information, the correction information being information on correction of 3D data representing a three-dimensional structure constructed using 2D data representing a two-dimensional image; and

[0253] 3D data is constructed by using 2D data and the correction information generated by decoding encoded data of the correction information.

[0254] Reference numerals list

[0255] 100 Encoding device

[0256] 111 Patch Decomposition Unit

[0257] 112 Packing Units

[0258] 113 Auxiliary patch information compression unit

[0259] 114 Video Coding Unit

[0260] 115 Video Encoding Unit

[0261] 116 OMap encoding unit

[0262] 117 Multiplexer

[0263] 118 Decoding Unit

[0264] 119 Correction information generation unit

[0265] 151 Type setting unit

[0266] 152 Removal amount setting unit

[0267] 200 Decoding device

[0268] 211 Demultiplexer

[0269] 212 Auxiliary patch information decoding unit

[0270] 213 Video decoding unit

[0271] 214 Video decoding unit

[0272] 215 OMap decoding unit

[0273] 216 Unpacking Unit

[0274] 217 3D Reconstruction Unit

[0275] 251 Calibration setting unit

[0276] 252 Reconstruction Unit

Claims

1. An image processing device, comprising: A processing unit, the processing unit being configured to: Generate an occupancy map consisting of a patch area, where The patch area includes the patch and the blank space at the boundary of the patch, and The face patch is 2D data corresponding to 3D data representing a three-dimensional structure. generating correction information, wherein the correction information is information about correction of the blank; encoding the occupancy map and the correction information; The occupancy map and the correction information are multiplexed, wherein The correction information includes information indicating a method of removing the blank.

2. The image processing device according to claim 1, in, The processing unit generates the correction information based on an occupancy map of the 3D data having 1×1 precision and an occupancy map of the 3D data having N×N precision.

3. The image processing device according to claim 1, in, The processing unit also generates the correction information based on a margin, the margin being a set value of the size of the margin.

4. The image processing device according to claim 1, in, The correction information includes information indicating a mode of the removal method of the blank selected from candidates.

5. The image processing device according to claim 1, in, The correction information includes information indicating an amount of removal of the blank.

6. An image processing method, comprising: Generate an occupancy map consisting of a patch area, where The patch area includes the patch and the blank space at the boundary of the patch, and The face patch is 2D data corresponding to 3D data representing a three-dimensional structure. generating correction information, wherein the correction information is information about correction of the blank; encoding the occupancy map and the correction information; and The occupancy map and the correction information are multiplexed, wherein The correction information includes information indicating a method of removing the blank.

7. An image processing device, comprising: A processing unit, the processing unit being configured to: The bitstream is demultiplexed to extract the encoded data including the occupancy map and correction information of the patch area, wherein The patch area includes the patch and the blank space at the boundary of the patch, and The face patch is 2D data corresponding to 3D data representing a three-dimensional structure. The correction information is information on correction of the blank and includes information indicating a removal method of the blank; and The 3D data is constructed by using the 2D data and the correction information generated by decoding the occupancy map and the encoded data of the correction information.

8. The image processing device according to claim 7, in, The correction information includes information indicating a mode of the removal method of the blank selected from candidates.

9. The image processing device according to claim 7, in, The correction information includes information indicating an amount of removal of the blank.

10. The image processing apparatus according to claim 7, wherein the processing unit is further configured to: Decoding the encoded data of the 2D data; The 3D data is constructed by using the decoded 2D data and the correction information.

11. The image processing apparatus according to claim 7, wherein the processing unit is further configured to: decoding the encoded data of the occupancy map; The decoded blank during construction from 2D data to 3D data is corrected by using the correction information generated by decoding the encoded data of the correction information.

12. An image processing method, comprising: The bitstream is demultiplexed to extract the encoded data including the occupancy map and correction information of the patch area, wherein The patch area includes the patch and the blank space at the boundary of the patch, and The face patch is 2D data corresponding to 3D data representing a three-dimensional structure. The correction information is information on correction of the blank and includes information indicating a removal method of the blank; and The 3D data is constructed by using the 2D data and the correction information generated by decoding the occupancy map and the encoded data of the correction information.

Images