Information processing device and method

Abstract

The present disclosure pertains to an information processing device and a method that enable easier control of the information volume of point cloud data. The invention includes: forming a reference structure of geometric data in point cloud encoding, the reference structure being hierarchized by groups obtained by sorting points of a point cloud expressing a three-dimensionally shaped object as a collection of points; deriving a geometric data prediction value for each of the points on the basis of the formed reference structure; deriving a prediction residual, which is the difference between the geometric data and the prediction value; and encoding the derived geometric data prediction residual of each of the points. The present disclosure can be applied to, for example, an information processing device, an image processing device, an encoding device, a decoding device, an electronic apparatus, an information processing method, a program or the like.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that enable easier control of the amount of information in point cloud data. [Background technology]

[0002] Conventionally, a method for encoding 3D data representing a three-dimensional structure such as a point cloud has been considered (see, for example, Non-Patent Document 1). In addition, when encoding the geometry data of this point cloud, a method for deriving a difference value (prediction residual) from a predicted value and encoding the prediction residual has been considered (see, for example, Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] R. Mekuria, Student Member IEEE, K. Blom, P. Cesar., Member, IEEE, "Design, Implementation and Evaluation of a Point Cloud Codec for Tele-Immersive Video",tcsvt_paper_submitted_february.pdf [Non-Patent Document 2] Zhenzhen Gao, David Flynn, Alexis Tourapis, and Khaled Mammou, "[G-PCC][New proposal] Predictive Geometry Coding", ISO / IEC JTC1 / SC29 / WG11 MPEG2019 / m51012, October 2019, Geneva, CH Summary of the Invention [Problem to be solved by the invention]

[0004] Since point cloud data is composed of geometry data and attribute data of a plurality of points, the amount of information can be easily controlled by controlling the number of points.

[0005] However, in the case of the method described in Non-Patent Document 2, since the geometry data of other points is referenced when deriving a predicted value, once a reference structure is constructed, the constraints imposed by the reference structure are large, and it may be difficult to reduce the number of points. Therefore, it may be difficult to control the amount of information.

[0006] The present disclosure has been made in consideration of these circumstances, and makes it possible to more easily control the amount of information in point cloud data. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present technology is an information processing device including: a reference structure forming unit that forms a reference structure of geometry data in encoding of a point cloud, the point cloud representing a three-dimensional object as a collection of points, hierarchically organized according to groups into which the points of the point cloud are classified; a prediction residual derivation unit that derives a predicted value of the geometry data for each point based on the reference structure formed by the reference structure forming unit, and derives a prediction residual, which is the difference between the geometry data and the predicted value; and an encoding unit that encodes the prediction residual of the geometry data for each point derived by the prediction residual derivation unit.

[0008] An information processing method according to one aspect of the present technology is an information processing method that forms a reference structure of geometry data in encoding of a point cloud, the reference structure being hierarchical according to groups into which points of the point cloud are classified, derives a predicted value of the geometry data for each point based on the formed reference structure, derives a prediction residual, which is the difference between the geometry data and the predicted value, and encodes the prediction residual of the derived geometry data for each point.

[0009] An information processing device according to another aspect of the present technology is an information processing device that includes a decoding unit that decodes encoded data corresponding to a desired group hierarchy among encoded data in which prediction residuals, which are differences between geometry data of each point derived based on the reference structure and its predicted value, are encoded based on layer information indicating a group hierarchy, which is a hierarchy of groups in a reference structure of geometry data in encoding of a point cloud in which points of the point cloud are hierarchically categorized into groups into which the points of the point cloud are classified, the prediction residuals being the differences between the geometry data of each point derived based on the reference structure and the predicted values ​​of the points.

[0010] An information processing method according to another aspect of the present technology is an information processing method for decoding encoded data corresponding to a desired group hierarchy among encoded data in which prediction residuals, which are differences between geometry data of each point derived based on a reference structure and its predicted value, are encoded, based on layer information indicating a group hierarchy, which is a hierarchy of groups in a reference structure of geometry data in encoding of a point cloud in which points of the point cloud are hierarchically categorized into groups into which the points of the point cloud are classified, the reference structure being derived from the geometry data of each point and the predicted value.

[0011] In an information processing device and method according to one aspect of the present technology, a reference structure for geometry data in encoding of a point cloud is formed, the reference structure being hierarchical according to groups into which points of the point cloud, which represents a three-dimensional object as a collection of points, is classified, and based on the reference structure thus formed, a predicted value of the geometry data for each point is derived, a prediction residual, which is the difference between the geometry data and the predicted value, is derived, and the derived prediction residual of the geometry data for each point is encoded.

[0012] In an information processing device and method according to another aspect of the present technology, based on layer information indicating a group hierarchy, which is a hierarchy of groups in a reference structure of geometry data in encoding of a point cloud in which points of the point cloud representing a three-dimensional object as a collection of points are hierarchically categorized into groups, the coded data corresponding to a desired group hierarchy is decoded from coded data in which prediction residuals, which are the difference between the geometry data of each point derived based on the reference structure and its predicted value, are coded. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating Predictive Geometry Coding. [Diagram 2] FIG. 1 is a diagram illustrating Predictive Geometry Coding. [Diagram 3] FIG. 1 is a diagram illustrating Predictive Geometry Coding. [Figure 4] FIG. 1 is a diagram illustrating an extension method of Predictive Geometry Coding. [Diagram 5] 1A to 1C are diagrams illustrating an example of a method for forming a reference structure. [Figure 6] FIG. 11 is a diagram illustrating an example of layer information. [Figure 7] FIG. 2 is a block diagram showing an example of the main configuration of an encoding device. [Figure 8] 11 is a block diagram showing an example of the main configuration of a geometry data encoding unit. FIG. [Figure 9] 4 is a block diagram showing an example of a main configuration of a reference structure forming unit; FIG. [Figure 10] 11 is a flowchart illustrating an example of the flow of an encoding process. [Figure 11] 13 is a flowchart illustrating an example of the flow of a geometry data encoding process. [Figure 12] 13 is a flowchart illustrating an example of the flow of a reference structure forming process. [Figure 13] FIG. 2 is a block diagram showing an example of the main configuration of a decoding device. [Figure 14] 13 is a block diagram showing an example of a main configuration of a geometry data decoding unit. FIG. [Figure 15] 13 is a flowchart illustrating an example of the flow of a decoding process. [Figure 16] 13 is a flowchart illustrating an example of the flow of a geometry data decoding process. [Figure 17] FIG. 2 is a block diagram showing an example of the main configuration of a transcoder. [Figure 18] 11 is a flowchart illustrating an example of the flow of a transcoding process. [Figure 19] FIG. 1 is a diagram illustrating an extension method of Predictive Geometry Coding. [Figure 20] FIG. 2 is a block diagram showing an example of the main configuration of an encoding device. [Figure 21] 11 is a flowchart illustrating an example of the flow of an encoding process. [Figure 22] FIG. 2 is a block diagram showing an example of the main configuration of a decoding device. [Figure 23] 13 is a flowchart illustrating an example of the flow of a decoding process. [Figure 24] FIG. 2 is a block diagram showing an example of the main configuration of a transcoder. [Diagram 25] 11 is a flowchart illustrating an example of the flow of a transcoding process. [Figure 26]11 is a flowchart illustrating an example of the flow of an encoding process. [Figure 27] 13 is a flowchart illustrating an example of the flow of a decoding process. [Figure 28] FIG. 1 is a block diagram showing an example of a main configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1. First embodiment (hierarchical reference structure) 2. Second embodiment (prediction residual coding of attribute data) 3. Third embodiment (Hierarchical reference structure and predictive residual coding of attribute data) 4. Notes

[0015] <1. First embodiment> <References supporting technical content and technical terms> The scope of disclosure of the present technology includes not only the contents described in the embodiments, but also the contents described in the following non-patent documents that were publicly known at the time of filing the application.

[0016] Non-patent document 1: (mentioned above) Non-patent document 2: (mentioned above)

[0017] In other words, the contents described in the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0018] <Point cloud> Previously, there was 3D data such as point clouds, which represent three-dimensional structures using point position information and attribute information, and meshes, which are composed of vertices, edges, and faces and define three-dimensional shapes using polygonal representations.

[0019] For example, in the case of a point cloud, a three-dimensional structure (an object with a three-dimensional shape) is represented as a collection of many points. Point cloud data (also referred to as point cloud data) is composed of position information (also referred to as geometry data) and attribute information (also referred to as attribute data) of each point. The attribute data can include any information. For example, color information, reflectance information, normal information, etc. of each point may be included in the attribute data. In this way, point cloud data has a relatively simple data structure, and by using a sufficiently large number of points, any three-dimensional structure can be represented with sufficient accuracy.

[0020] <Predictive Geometry Coding> Since such point cloud data has a relatively large amount of data, the amount of data is generally reduced by encoding or the like when recording or transmitting the data. Various methods have been proposed as this encoding method. For example, Non-Patent Document 2 describes predictive geometry coding as a method for encoding geometry data.

[0021] In predictive geometry coding, the difference between the geometry data of each point and its predicted value (also called prediction residual) is derived, and the prediction residual is coded. When deriving the predicted value, the geometry data of other points is referenced.

[0022] For example, as shown in FIG. 1, a reference structure (also called a prediction tree) is formed that indicates which points' geometry data are to be referenced when deriving a predicted value of the geometry data of each point. In FIG. 1, circles indicate points, and arrows indicate reference relationships. This reference structure can be formed in any manner. For example, it can be formed so that the geometry data of nearby points is referenced.

[0023] In the example prediction tree of Figure 1, point 11 (Root vertex) that does not reference the geometry data of other points, point 12 (Branch vertex with one child) that is referenced by one other point, point 13 (Branch vertex with 3 children) that is referenced by three other points, point 14 (Branch vertex with 2 children) that is referenced by two other points, and point 15 (Leaf vertex) that is not referenced by other points are formed.

[0024] In FIG. 1, only one point 12 is labeled, but all the points indicated by white circles are point 12. Similarly, only one point 14 is labeled, but all the points indicated by diagonally shaded circles in FIG. 1 are point 14. Similarly, only one point 15 is labeled, but all the points indicated by gray circles in FIG. 1 are point 15. Note that this prediction tree structure is an example and is not limited to the example in FIG. 1. Therefore, the numbers of points 11 to 15 are arbitrary. Furthermore, the pattern of the number of points to be referenced is not limited to the example in FIG. 1. For example, there may be a point referenced from four or more points.

[0025] The predicted value of the geometry data of each point is derived according to such a reference structure (prediction tree). For example, the predicted value is derived in four ways (four modes), and the best predicted value is selected from them.

[0026] For example, in a reference structure such as points 21 to 24 in FIG. 2, point 24 is set as a processing target point (Target point pi) and a predicted value of its geometry data is derived. In the first mode, point 23 (P parent) is the predicted point 31 of the point 24, and its geometry data is the predicted value of the geometry data of the point 24. The geometry data of this predicted point 31 (i.e., the predicted value of the first mode of the geometry data of the point 24) is q (Delta) It is called.

[0027] In the second mode, in such a reference structure, the point 23 is the starting point and the point 22 (P grandparent If the start point of the inverse vector of the reference vector (the arrow between point 23 and point 22) whose end point is point 23 is set as point 23, the end point of the inverse vector is set as predicted point 32, and the geometry data of the inverse vector is set as the predicted value of the geometry data of point 24. The geometry data of this predicted point 32 (i.e., the predicted value of the second mode of the geometry data of point 24) is set as q (Linear) It is called.

[0028] In the third mode, in such a reference structure, the point 22 is the starting point and the point 21 (P great-grandparent If the start point of the inverse vector of the reference vector (the arrow between point 22 and point 21) whose end point is point 23 is set as point 23, the end point of the inverse vector is set as predicted point 33, and its geometry data is set as the predicted value of the geometry data of point 24. The geometry data of this predicted point 33 (i.e., the predicted value of the third mode of the geometry data of point 24) is set as q (Parallelogram) It is called.

[0029] In the fourth mode, point 24 is the root point (Root vertex) and does not refer to the geometry data of other points. In other words, for this point 24, the geometry data of point 24 is coded, not the prediction residual. In the case of the reference structure of the example of FIG. 2, point 24 references point 23, so this mode is excluded.

[0030] For the predicted values ​​of each of the above modes (three modes in the example of FIG. 2), the prediction residual (difference from the geometry data of point 24) is derived, and the smallest prediction residual is selected. In other words, the nearest prediction point to point 24 is selected, and the prediction residual corresponding to that prediction point is selected.

[0031] By performing such processing for each point, the prediction residual for each point is derived. Then, the prediction residual is coded. In this way, it is possible to suppress an increase in the amount of coding.

[0032] Note that the reference structure (prediction tree) is formed based on a predetermined method, but this formation method is arbitrary. For example, assume that each point 41 is captured in the order shown in A of FIG. 3. In FIG. 3, a circle indicates a point, and a number in the circle indicates a capture order. In FIG. 3, only point 41 with a capture order of 0 is labeled, but all circles in FIG. 3 are points 41. That is, five points 41 with capture orders from 0 to 5 are shown in FIG. 3.

[0033] In such a case, a reference structure may be formed so that each point references the nearest point (as a parent node), as shown in B of Fig. 3. Encoding and decoding are performed according to the reference structure, and a search for child nodes (nodes that reference the node being processed) is performed using a stack. Therefore, the decoding order in this case is point 41 with capture order "0", point 41 with capture order "1", point 41 with capture order "3", point 41 with capture order "4", point 41 with capture order "5", and point 41 with capture order "2".

[0034] Also, a reference structure may be formed such that each point refers to (is a parent node of) the previous point in the capture order, as shown in C of Fig. 3. In this case, the decoding order is the same as the capture order.

[0035] <Information volume control for point cloud data> Since point cloud data is composed of geometry data and attribute data of a plurality of points, the amount of information can be easily controlled by controlling the number of points.

[0036] For example, by reducing the number of points when encoding geometry data, it is possible to suppress an increase in the load of the encoding process. Also, it is possible to control the bit rate of the encoded data of the geometry data generated by the encoding. In other words, it is possible to control the storage capacity when storing the encoded data and the transmission rate when transmitting the encoded data.

[0037] Also, the number of points can be reduced when decoding geometry data. For example, by configuring the encoded data of the geometry data of the point cloud to be partially decoded, it is possible to generate geometry data by decoding encoded data for only some points. For example, by controlling the points to be decoded, it is possible to control the resolution (also called spatial resolution) of the generated point cloud. Such a decoding method is also called scalable decoding (or decoding scalability). By realizing such scalable decoding, it is possible to omit the decoding process of unnecessary data, thereby suppressing an increase in the load of the decoding process.

[0038] Furthermore, the number of points can be reduced during transcoding, which is a process of decoding the encoded data of the geometry data and re-encoding it by changing desired parameters. This makes it possible to suppress an increase in the transcoding load. Also, it is possible to control the bit rate of the encoded data of the geometry data generated by transcoding.

[0039] <Information volume control when applying predictive geometry coding> However, in the case of predictive geometry coding described in Non-Patent Document 2, as described above, geometry data of other points is referenced when deriving a predicted value. Therefore, when a reference structure is constructed, the constraints imposed by the reference structure are large, and it may be difficult to reduce the number of points. Therefore, it may be difficult to control the amount of information.

[0040] For example, when a node in the middle of a prediction tree is deleted, it becomes impossible to derive the predicted values ​​of the child nodes and subsequent nodes. In other words, it is not possible to delete only the desired points regardless of the prediction tree (points of child nodes are also deleted). Therefore, when a certain point is deleted, a phenomenon may occur in which many surrounding points are also deleted according to the structure of the prediction tree. In such a case, for example, there is a risk of problems occurring, such as the distribution of points changing significantly locally, and the shape of the object indicated by the point cloud being deformed (i.e., the point cloud cannot correctly represent the shape of the object).

[0041] As described above, in the case of the predictive geometry coding described in Non-Patent Document 2, in reality, it is difficult to achieve the above-mentioned bit rate control and scalability when encoding, decoding, or transcoding geometry data.

[0042] <Hierarchical reference structure> Therefore, as shown in the top row of the table in Fig. 4, predictive geometry coding is extended, and as shown in the second row from the top (row "1") of the table in Fig. 4, groups of points are formed, and a reference structure is formed in which the groups form a hierarchy. This hierarchy is called a group hierarchy, and the hierarchicalization of such a reference structure is called group hierarchization.

[0043] For example, a reference structure for geometry data in encoding of a point cloud is formed, which is hierarchical according to groups into which points of the point cloud, which represents a three-dimensional object as a collection of points, and based on the formed reference structure, a predicted value of the geometry data for each point is derived, a prediction residual, which is the difference between the geometry data and the predicted value, is derived, and the derived prediction residual of the geometry data for each point is encoded.

[0044] Also, for example, an information processing device may be provided with a reference structure forming unit that forms a reference structure of geometry data in encoding of a point cloud hierarchically organized into groups into which points of the point cloud, which represents a three-dimensional object as a collection of points, a prediction residual derivation unit that derives a predicted value of the geometry data for each point based on the reference structure formed by the reference structure forming unit and derives a prediction residual, which is the difference between the geometry data and the predicted value, and an encoding unit that encodes the prediction residual of the geometry data for each point derived by the prediction residual derivation unit.

[0045] By dividing the prediction tree into groups in this way, the selection of points for each group hierarchy is made in accordance with the prediction tree. For example, a point belonging to the lowest hierarchy of this group hierarchy corresponds to a node on the leaf side of the prediction tree, and a point belonging to another group hierarchy corresponds to a node on the root side of that node. Therefore, a point belonging to a lower group hierarchy can be deleted without affecting a point belonging to a higher group hierarchy. For example, it is possible to encode only the geometry data of points belonging to the top layer to intermediate layers of the group hierarchy (and delete the geometry data of points belonging to lower group hierarchies).

[0046] Therefore, it is possible to suppress an increase in the load of the encoding process. Also, it is possible to control the bit rate of the encoded data of the geometry data generated by the encoding. In addition, it is possible to control the storage capacity when storing the encoded data and the transmission rate when transmitting the encoded data.

[0047] As shown in the third row from the top of the table in FIG. 4 (row "1-1"), when forming this reference structure, the captured points may be grouped and the points sorted (rearranged) in the order of the group.

[0048] For example, points may be divided into groups, sorted by group, and the reference destination of the geometry data of each point may be set in the sorted order, thereby forming a hierarchical reference structure by group.

[0049] For example, in an information processing device, a reference structure formation unit may include a grouping processing unit that groups points, a sorting unit that sorts the points by group, and a group hierarchical reference structure formation unit that forms a hierarchical reference structure by group by setting a reference destination for the geometry data of each point in the order sorted by the sorting unit.

[0050] For example, assume that eight points 51 are captured as shown in A of Fig. 5. In A of Fig. 5, the circles indicate the points, and the numbers in the circles indicate the order of capture. Note that, although only one point 51 is labeled in Fig. 5, all the points indicated by the circles in Fig. 5 are points 51.

[0051] These points 51 are grouped in a predetermined manner. This grouping method is arbitrary. In the case of A in FIG. 5, the points 51 are divided into a group of points 51 whose capture order is odd numbers (1, 3, 5, 7) and a group of points 51 whose capture order is even numbers (2, 4, 6, 8).

[0052] Then, for example, as shown in Fig. 5B, these points 51 are sorted into groups. In the example of Fig. 5B, the points 51 are rearranged in the order of a group of points 51 whose capture order is odd numbers (1, 3, 5, 7) and a group of points 51 whose capture order is even numbers (2, 4, 6, 8).

[0053] Then, the reference of each point is obtained in this rearranged order, and a prediction tree is formed. For example, in the case of the prediction tree shown in C of Fig. 5, first, a reference is set for each point 51 whose capture order is odd, and then a reference is set for each point 51 whose capture order is even. Therefore, in this prediction tree, a node corresponding to a point 51 whose capture order is even is formed as a node on the leaf side (child node side) of a node corresponding to a point 51 whose capture order is odd. Therefore, even if a point 51 whose capture order is even is deleted, it does not affect a point 51 whose capture order is odd.

[0054] In other words, even after forming a prediction tree, it is possible to select points to be encoded, so that it is possible to suppress an increase in the load of the encoding process. Also, it is possible to control the bit rate of the encoded data of the geometry data generated by the encoding.

[0055] By forming a prediction tree using this method, after sorting, a prediction tree can be formed using the same method as the method described in Non-Patent Document 2. Therefore, the prediction tree can be more easily organized into a group hierarchy. This makes it possible to suppress an increase in the cost of forming a prediction tree.

[0056] As shown in the fourth row from the top of the table in FIG. 4 (row "1-1-1"), grouping (division of points into groups) may be performed according to the position of the points.

[0057] For example, based on the position of each point, grouping may be performed so that the density of points belonging to each group in three-dimensional space is uniform (so that the points are spaced apart at a predetermined interval). By performing grouping in this manner, the number of points can be reduced so that the density of the points to be encoded is uniform. In other words, it is possible to suppress an increase in the load of the encoding process and control the bit rate of the encoded data so as to reduce changes in the distribution mode of the point cloud (i.e., the shape of the object indicated by the point cloud). In addition, in this case, the resolution (spatial resolution) in the three-dimensional space of the point cloud can be controlled by increasing or decreasing the number of group hierarchies from which points are deleted.

[0058] Furthermore, as shown in the fifth row from the top of the table shown in FIG. 4 (row "1-1-2"), grouping (division of points into groups) may be performed according to the characteristics of the points.

[0059] The characteristics of the points used for this grouping are arbitrary. For example, the points may be grouped by points corresponding to edges or corners of the point cloud, or by points corresponding to flat parts of the point cloud. Of course, the characteristics may be other than these examples. By grouping in this way, for example, it is possible to delete points (having characteristics) that have a relatively small subjective effect during playback, and to encode points (having characteristics) that have a relatively large subjective effect during playback. This makes it possible to suppress an increase in the load of the encoding process and control the bit rate of the encoded data so as to suppress a decrease in the subjective image quality during playback.

[0060] The grouping method is not limited to these examples and may be any method. For example, the grouping may be performed according to both the position and the feature of the points.

[0061] Also, as shown in the sixth row from the top of the table shown in FIG. 4 (row "1-2"), layer information, which is information relating to the group hierarchy, may be signaled.

[0062] For example, layer information indicating a group hierarchy for each point may be generated and encoded. For example, in the information processing device, the reference structure forming unit may further include a layer information generating unit that generates, for each point, layer information indicating a group hierarchy that is a hierarchy based on groups in the reference structure, and the encoding unit may further encode the layer information generated by the layer information generating unit.

[0063] In this layer information, the group hierarchy may be indicated by the difference from the group hierarchy of the parent node (a relative value from the parent node).

[0064] For example, assume that points 60 to 69 are captured as shown in A of Fig. 6. Then, assume that points 60, 63, 66, and 69 are divided into a first group, points 61, 64, and 67 are divided into a second group, and points 62, 65, and 68 are divided into a third group. Then, assume that these points are sorted by group as described above, references are set, and a prediction tree as shown in B of Fig. 6 is formed.

[0065] In this example, point 61 belongs to the second group, and the parent node of the node corresponding to point 61 in the prediction tree is the node corresponding to point 60 which belongs to the first group, so "+1" is generated as layer information for the node corresponding to point 61. This "+1" indicates that the node to be processed belongs to a group hierarchy (second group) one hierarchy lower than the group hierarchy (first group) of its parent node.

[0066] Similarly, point 62 belongs to the third group, and the parent node of the node corresponding to point 62 in the prediction tree is the node corresponding to point 61 which belongs to the second group, so “+1” is generated as layer information for the node corresponding to point 62.

[0067] In contrast, point 63 belongs to the first group, and the parent node of the node corresponding to point 63 in the prediction tree is the node corresponding to point 60 which belongs to the same first group, so "+0" is generated as layer information for the node corresponding to point 63. Similarly, point 64 belongs to the second group, and the parent node of the node corresponding to point 64 in the prediction tree is the node corresponding to point 61 which belongs to the same second group, so "+0" is generated as layer information for the node corresponding to point 64. Similarly, point 65 belongs to the third group, and the parent node of the node corresponding to point 65 in the prediction tree is the node corresponding to point 62 which belongs to the same third group, so "+0" is generated as layer information for the node corresponding to point 65.

[0068] Similarly, point 66 belongs to the first group, and the parent node of the node corresponding to point 66 in the prediction tree is the node corresponding to point 63 which also belongs to the first group, so "+0" is generated as layer information for the node corresponding to point 66. Point 67 belongs to the second group, and the parent node of the node corresponding to point 67 in the prediction tree is the node corresponding to point 66 which also belongs to the first group, so "+1" is generated as layer information for the node corresponding to point 67.

[0069] Point 68 belongs to the third group, and the parent node of the node corresponding to point 68 in the prediction tree is the node corresponding to point 65 which also belongs to the third group, so "+0" is generated as layer information for the node corresponding to point 68. Similarly, point 69 belongs to the first group, and the parent node of the node corresponding to point 69 in the prediction tree is the node corresponding to point 66 which also belongs to the first group, so "+0" is generated as layer information for the node corresponding to point 669.

[0070] Since there is no parent node for the node corresponding to point 60, "+0" is generated as layer information for the node corresponding to point 60.

[0071] In this way, by signaling layer information, the decoding side can easily grasp the group hierarchy of each point based on the signaled layer information. Therefore, when decoding, based on the layer information, only the encoded data of the desired group hierarchy can be decoded. That is, it is possible to easily realize scalable decoding. In other words, since the decoding side can grasp the structure of the group hierarchy based on the layer information, the encoding side can arbitrarily set the group.

[0072] Additionally, by indicating the group hierarchy to which the node to be processed belongs as a relative value from the group hierarchy to which the parent node belongs, as described above, it is possible to suppress an increase in the amount of code due to this layer information.

[0073] When such layer information is signaled, the layer information may be signaled in the parent node as shown in the seventh row from the top (row "1-2-1") of the table shown in Fig. 4. For example, layer information indicating the group hierarchy of each child node belonging to the processing target node in the reference structure as a relative value with respect to the group hierarchy of the processing target node may be generated and encoded.

[0074] For example, in the case of the prediction tree of B in Fig. 6, as layer information of the node (parent node) corresponding to point 60, information (+1) indicating the group hierarchy to which the node (child node) corresponding to point 61 belongs and information (+0) indicating the group hierarchy to which the node (child node) corresponding to point 63 belongs are signaled. Also, as layer information of the node (parent node) corresponding to point 61, information (+1) indicating the group hierarchy to which the node (child node) corresponding to point 62 belongs and information (+0) indicating the group hierarchy to which the node (child node) corresponding to point 64 belongs are signaled. Similarly, as layer information of each node, information indicating the group hierarchy of its child node is signaled.

[0075] In this way, when a parent node is decoded, the group hierarchy of its child nodes can be ascertained.

[0076] Also, layer information may be signaled in a child node, as shown in the eighth row from the top (row "1-2-2") of the table shown in Fig. 4. For example, layer information indicating the group hierarchy of a processing target node in a reference structure as a relative value to the group hierarchy of a parent node to which the processing target node belongs may be generated and encoded.

[0077] For example, in the case of the prediction tree in B of Fig. 6, information (+1) indicating the group hierarchy to which the node corresponding to point 61 belongs is signaled as layer information of the node corresponding to point 61. Also, information (+0) indicating the group hierarchy to which the node corresponding to point 63 belongs is signaled as layer information of the node corresponding to point 63. Similarly, information indicating the group hierarchy of each node is signaled as layer information of that node.

[0078] In this way, the group hierarchy of the node to be processed can be grasped.

[0079] In addition, when encoding various information such as geometry data (prediction residuals), quantization may be performed. In the quantization, the quantization step may be controlled for each layer, as shown in the ninth row from the top (row "1-3") of the table shown in Fig. 4. In other words, the prediction residuals, etc. may be quantized and encoded with a quantization step set for each group layer, which is a layer by groups in the reference structure. For example, the quantization step may be changed for each layer.

[0080] Also, as shown in the tenth row from the top (row "1-3-1") of the table shown in Fig. 4, the layer information may be signaled in the child node. The quantization step may be signaled. That is, information indicating the quantization step may be coded.

[0081] Also, as shown in the eleventh row (row "1-4") from the top of the table in FIG. 4, when encoding various information such as geometry data (prediction residuals), arithmetic coding may be performed independently for each layer (group hierarchy) as shown in the twelfth row (row "1-4-1") from the top of the table in FIG. 4. That is, the prediction residuals and the like may be arithmetically coded for each group hierarchy, which is a hierarchy by groups in the reference structure. In this way, the encoded data can be decoded for each group hierarchy. For example, when some group hierarchies are deleted and transcoded, transcoding can be performed by simply selecting (without decoding) the encoded data of the group hierarchy that is not deleted. This makes it possible to suppress an increase in the load of transcoding. In this case, the layer information may be signaled at the parent node.

[0082] It should be noted that arithmetic coding may be performed independently for units smaller than the group hierarchy, for example, for each branch or each node of a prediction tree.

[0083] Also, as shown in the bottom row (row "1-4-2") of the table shown in FIG. 4, arithmetic coding may be performed without separating for each layer (group hierarchy). In other words, prediction residuals and the like may be arithmetically coded without separating for each group hierarchy, which is a hierarchy based on groups in a reference structure. For example, prediction residuals and the like of a plurality of group hierarchies may be arithmetically coded without separating them. Also, prediction residuals and the like of all group hierarchies may be arithmetically coded without separating them.

[0084] Also, whether or not to encode may be selected for each group hierarchical layer, and the prediction residuals and the like of the group hierarchical layer selected to be encoded may be encoded. In other words, control of whether or not to encode (control of whether or not to delete) may be performed for each group hierarchical layer.

[0085] For example, by grouping points so that their density in three-dimensional space is uniform, and controlling the encoding for each group hierarchy in this manner, the spatial resolution of the point cloud data to be encoded can be controlled.

[0086] Of course, coding control may be performed in units smaller than the group hierarchy. For example, whether or not to code may be selected for each branch of the reference structure, and the prediction residuals of the nodes belonging to the branch selected for coding may be coded. In other words, control of whether or not to code (control of whether or not to delete) may be performed for each branch. In this way, information on some branches in the group hierarchy can be deleted, and more detailed coding control can be achieved.

[0087] The information unit for this coding control is arbitrary, and coding control may be performed for each information unit other than the above-mentioned example. For example, coding control may be performed for each of a plurality of information units. For example, coding control may be performed for each group hierarchy or each branch.

[0088] <Encoding device> Next, a device to which the present technology is applied will be described. Fig. 7 is a block diagram showing an example of a configuration of an encoding device, which is one aspect of an information processing device to which the present technology is applied. The encoding device 100 shown in Fig. 7 is a device that encodes a point cloud (3D data). The encoding device 100 encodes the point cloud by applying the present technology described above with reference to Fig. 4, for example.

[0089] Note that Fig. 7 shows the main processing units, data flows, etc., and is not limited to all that is shown in Fig. 7. In other words, in encoding device 100, there may be processing units that are not shown as blocks in Fig. 7, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 7.

[0090] As shown in FIG. 7, the encoding device 100 includes a geometry data encoding unit 111 and an attribute data encoding unit 112 .

[0091] The geometry data encoding unit 111 acquires the point cloud (3D data) input to the encoding device 100, encodes the geometry data (position information) to generate encoded data of the geometry data, and supplies the generated encoded data of the geometry data and attribute data (attribute information) to the attribute data encoding unit 112.

[0092] The attribute data encoding unit 112 acquires the encoded data of the geometry data and the attribute data supplied from the geometry data encoding unit 111, uses them to encode the attribute data, generates encoded data of the attribute data, and outputs the encoded data of the geometry data and the generated encoded data of the attribute data to the outside of the encoding device 100 (e.g., the decoding side) as encoded data of point cloud data.

[0093] These processing units (the geometry data encoding unit 111 and the attribute data encoding unit 112) may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-mentioned processing. Also, each processing unit may have, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and may realize the above-mentioned processing by executing a program using them. Of course, each processing unit may have both configurations, and may realize part of the above-mentioned processing by a logic circuit and the other by executing a program. The configurations of each processing unit may be independent of each other, and for example, some processing units may realize part of the above-mentioned processing by a logic circuit, other processing units may realize the above-mentioned processing by executing a program, and still other processing units may realize the above-mentioned processing by both a logic circuit and executing a program.

[0094] <Geometry data encoding part> Fig. 8 is a block diagram showing a main configuration example of the geometry data encoding unit 111. Note that Fig. 8 shows main processing units, data flows, etc., and is not limited to all that are shown in Fig. 8. In other words, in the geometry data encoding unit 111, there may be processing units that are not shown as blocks in Fig. 8, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 8.

[0095] As shown in FIG. 8, the geometry data encoding unit 111 includes a reference structure forming unit 131, a stack 132, a prediction mode determining unit 133, an encoding unit 134, and a prediction point generating unit 135.

[0096] The geometry data of the point cloud data supplied to the geometry data encoding unit 111 is supplied to a reference structure forming unit 131. Note that the attribute data is not processed in the geometry data encoding unit 111, and is supplied to an attribute data encoding unit 112.

[0097] The reference structure forming unit 131 generates a reference structure (prediction tree) for encoding the point cloud for the supplied geometry data. At this time, the reference structure forming unit 131 may form a group-hierarchical reference structure by applying various methods such as those described above with reference to the table in FIG. 4. In addition, the reference structure forming unit 131 generates layer information indicating the formed reference structure for each point. The reference structure forming unit 131 supplies the geometry data and layer information of the processing target point to the stack 132 according to the formed reference structure. At this time, the reference structure forming unit 131 may apply various methods such as those described above with reference to the table in FIG. 4.

[0098] For example, the reference structure forming unit 131 may determine whether or not to encode a child node in a reference structure (prediction tree) formed by the reference structure forming unit 131 according to encoding control by a user or the like, and when it is determined to encode, may supply geometry data, layer information, and the like of the child node of the processing target node to the stack 132. For example, the reference structure forming unit 131 may select whether or not to encode a prediction residual or the like for each group hierarchy, and supply the prediction residual or the like of a node belonging to the group hierarchy to the stack 132. In addition, the reference structure forming unit 131 may select whether or not to encode a prediction residual or the like for each branch of the reference structure, and supply the prediction residual or the like of a node belonging to the branch to the stack 132. In this way, it is possible to encode geometry data only for some points.

[0099] The stack 132 holds information in a last-in, first-out manner. For example, the stack 132 holds geometry data and layer information of each point supplied from the reference structure forming unit 131. In addition, the stack 132 supplies the prediction mode determining unit 133 with the last-held information in response to a request from the prediction mode determining unit 133.

[0100] The prediction mode determination unit 133 performs processing related to the determination of a prediction mode (prediction point). For example, the prediction mode determination unit 133 acquires geometry data and layer information of the point last held in the stack 132. The prediction mode determination unit 133 also acquires geometry data of the prediction point of the point from the prediction point generation unit 135. When there are multiple prediction points corresponding to the processing target point as in the example of FIG. 2, all of the prediction points are acquired. Then, the prediction mode determination unit 133 determines a prediction point (i.e., a prediction mode) to be applied. At that time, the prediction mode determination unit 133 may apply various methods as described above with reference to the table of FIG. 4.

[0101] For example, the prediction mode determination unit 133 derives a prediction residual, which is a difference between the geometry data (prediction value) of each prediction point and the geometry data of the processing target point, and compares the values. By such comparison, a prediction mode (prediction method) to be applied is selected. For example, a prediction point closest to the processing target point is selected. The prediction mode determination unit 133 supplies information about each point (for example, a prediction residual of the selected prediction mode, layer information, etc.) to the encoding unit 134.

[0102] The encoding unit 134 acquires information supplied by the prediction mode determination unit 133 (for example, prediction residuals and layer information of the selected prediction mode), encodes the information, and generates encoded data.

[0103] In this case, the encoding unit 134 may apply various methods as described above with reference to the table in Fig. 4. For example, the encoding unit 134 may quantize and encode the data using a quantization step set for each group layer. The encoding unit 134 may also encode and signal information indicating the quantization step. Furthermore, the encoding unit 134 may perform arithmetic encoding of the prediction residuals and the like by dividing them into group layers, or may perform arithmetic encoding of the prediction residuals and the like without dividing them into group layers.

[0104] The encoding unit 134 supplies the generated encoded data as encoded data of geometry data to the attribute data encoding unit 112. In addition, the encoding unit 134 supplies information such as the geometry data of the processing target point to the predicted point generating unit 135.

[0105] The prediction point generating unit 135 performs processing related to the generation of a prediction point, that is, the derivation of a prediction value. For example, the prediction point generating unit 135 acquires information such as geometry data of a processing target point supplied from the encoding unit 134. In addition, the prediction point generating unit 135 derives geometry data of a prediction point that can be generated using the geometry data of the processing target point, etc. (for example, a predicted value of the geometry data of a child node of a processing target node). In this case, the prediction point generating unit 135 may apply various methods such as those described above with reference to the table of FIG. 4. The prediction point generating unit 135 supplies the derived prediction value to the prediction mode determining unit 133 as necessary.

[0106] <Reference structure forming part> Fig. 9 is a block diagram showing a main configuration example of the reference structure forming unit 131. Note that Fig. 9 shows main processing units, data flows, etc., and is not limited to everything shown in Fig. 9. In other words, in the reference structure forming unit 131, there may be processing units not shown as blocks in Fig. 9, and there may be processing and data flows not shown as arrows, etc. in Fig. 9.

[0107] As shown in FIG. 9, the reference structure forming section 131 has a grouping processing section 151, a sorting section 152, a group hierarchical reference structure forming section 153, and a layer information generating section 154.

[0108] The grouping processing unit 151 performs processing related to grouping. For example, the grouping processing unit 151 performs grouping of points for the geometry data supplied to the reference structure forming unit 131. At that time, the grouping processing unit 151 may apply various methods as described above with reference to the table of FIG. 4. For example, the grouping processing unit 151 may perform the grouping according to the position of the point. Also, the grouping processing unit 151 may perform the grouping according to the feature of the point. The grouping processing unit 151 supplies the geometry data of each grouped point to the sorting unit 152.

[0109] The sorting unit 152 performs processing related to rearrangement of points. For example, the sorting unit 152 acquires geometry data of each grouped point supplied from the grouping processing unit 151. Then, the sorting unit 152 rearranges the geometry data of each point. In this case, the sorting unit 152 may apply various methods as described above with reference to the table in FIG. 4. For example, the sorting unit 152 rearranges the geometry data of each point grouped by the grouping processing unit 151 so as to consolidate the data into each group. The sorting unit 152 supplies the sorted geometry data of each point to the group hierarchical reference structure forming unit 153.

[0110] The group hierarchical reference structure forming unit 153 performs processing related to the formation of a reference structure. For example, the group hierarchical reference structure forming unit 153 acquires the sorted geometry data of each point supplied from the sorting unit 152. The group hierarchical reference structure forming unit 153 forms a reference structure. In this case, the group hierarchical reference structure forming unit 153 may apply various methods as described above with reference to the table in FIG. 4. For example, the group hierarchical reference structure forming unit 153 forms a group hierarchical reference structure by setting the reference destination of each point according to the sorted order supplied from the sorting unit 152. The method of forming this reference structure is arbitrary. The group hierarchical reference structure forming unit 153 supplies the reference structure thus formed to the layer information generating unit 154.

[0111] The layer information generating unit 154 acquires the reference structure supplied from the group hierarchical reference structure forming unit 153. The layer information generating unit 154 generates layer information indicating the reference structure. At this time, the layer information generating unit 154 may apply various methods as described above with reference to the table in FIG. 4. For example, the layer information generating unit 154 may generate and signal information indicating the group hierarchy of each child node belonging to the processing target node in the reference structure (for example, a relative value with respect to the group hierarchy of the processing target node) as the layer information of the processing target node. The layer information generating unit 154 may also generate and signal information indicating the group hierarchy of the processing target node in the reference structure (for example, a relative value with respect to the group hierarchy of the parent node to which the processing target node belongs) as the layer information of the processing target node. The layer information generating unit 154 supplies the generated layer information to the stack 132 (FIG. 8).

[0112] With the above configuration, the encoding device 100 can organize the reference structure of the geometry data into a group hierarchy. Therefore, as described above, the encoding device 100 can suppress an increase in the load of the encoding process. In addition, the encoding device 100 can control the bit rate of the encoded data of the geometry data generated by the encoding. In addition, the encoding device 100 can control the storage capacity when storing the encoded data and the transmission rate when transmitting the encoded data.

[0113] <Encoding process flow> Next, a description will be given of the processing executed by the encoding device 100. The encoding device 100 encodes point cloud data by executing an encoding process. An example of the flow of the encoding process will be described with reference to the flowchart in FIG.

[0114] When the encoding process is started, in step S101, the geometry data encoding unit 111 of the encoding device 100 executes a geometry data encoding process to encode the geometry data of the input point cloud and generate encoded data of the geometry data.

[0115] In step S102, the attribute data encoding unit 112 encodes the attribute data of the input point cloud to generate encoded data of the attribute data.

[0116] When the process of step S102 ends, the encoding process ends.

[0117] <Geometry data encoding process flow> Next, an example of the flow of the geometry data encoding process executed in step S101 of FIG. 10 will be described with reference to the flowchart of FIG.

[0118] When the geometry data encoding process is started, in step S131, the reference structure forming unit 131 executes a reference structure forming process to form a reference structure (prediction tree) of the geometry data. The reference structure forming unit 131 also generates layer information corresponding to the formed reference structure.

[0119] In step S132, the reference structure formation unit 131 stores in the stack 132 the geometry data and the like of the top node of the reference structure formed in step S131.

[0120] In step S133, the prediction mode determination unit 133 obtains, from the stack 132, the geometry data of the last stored point (node) and the like.

[0121] In step S134, the prediction mode determination unit 133 sets the point about which information was acquired in step S133 as a processing target, derives a prediction residual of the geometry data for the processing target point, and determines a prediction mode.

[0122] In step S135, the encoding unit 134 encodes the prediction mode determined in step S134. In addition, in step S136, the encoding unit 134 encodes the prediction residual of the prediction mode determined in step S134 of the geometry data. Furthermore, in step S137, the encoding unit 134 encodes child node information indicating whether a child node of the processing target node is a node of the degree. In addition, in step S138, the encoding unit 134 encodes the layer information generated in step S131. The encoding unit 134 supplies the encoded data of these pieces of information to the attribute data encoding unit 112 as encoded data of the geometry data.

[0123] In step S139, the reference structure formation unit 131 determines whether or not to encode the child node of the processing target node based on encoding control by a user, etc. If it is determined that the child node is to be encoded, the process proceeds to step S140.

[0124] In step S140, the reference structure forming unit 131 stores the geometry data of the child node, etc., in the stack 132. When the process of step S140 ends, the process proceeds to step S141. Also, if it is determined in step S139 that the child node is not to be encoded, the process of step S140 is skipped and the process proceeds to step S141.

[0125] In step S141, the predicted point generating unit 135 generates geometry data of a predicted point that can be generated using the geometry data of the processing target point.

[0126] In step S142, the prediction mode determination unit 133 determines whether the stack 132 is empty. If it is determined that the stack 132 is not empty (that is, information of at least one or more points is stored), the process returns to step S133. That is, the process of steps S133 to S142 is executed with the point last stored in the stack 132 as the processing target.

[0127] Such processing is repeated, and if it is determined in step S142 that the stack is empty, the geometry data encoding processing ends and the process returns to FIG.

[0128] <Reference structure creation process flow> Next, an example of the flow of the reference structure formation process executed in step S131 of FIG. 11 will be described with reference to the flowchart of FIG.

[0129] When the reference structure formation process is started, the grouping processing unit 151 divides the points of the point cloud into groups in step S161.

[0130] In step S162, the sorting unit 152 rearranges the processing order of each point in the point cloud so that the points are arranged according to the groups set in step S161.

[0131] In step S163, group hierarchical reference structure formation section 153 forms a group hierarchical reference structure by forming the reference destination of each point in the order sorted in step S163.

[0132] In step S164, the layer information generating unit 154 forms layer information for each point.

[0133] When the process of step S164 ends, the reference structure forming process ends, and the process returns to FIG.

[0134] By executing various processes as described above, the encoding device 100 can organize the reference structure of the geometry data into a group hierarchy. Therefore, as described above, the encoding device 100 can suppress an increase in the load of the encoding process. In addition, the encoding device 100 can control the bit rate of the encoded data of the geometry data generated by the encoding. In addition, the encoding device 100 can control the storage capacity when storing the encoded data and the transmission rate when transmitting the encoded data.

[0135] <Decryption device> Fig. 13 is a block diagram showing an example of a configuration of a decoding device, which is one aspect of an information processing device to which the present technology is applied. The decoding device 200 shown in Fig. 13 is a device that decodes encoded data of a point cloud (3D data). The decoding device 200 decodes the encoded data of the point cloud generated in the encoding device 100, for example.

[0136] Note that Fig. 13 shows the main processing units, data flows, and the like, and is not limited to all that is shown in Fig. 13. In other words, in the decoding device 200, there may be processing units that are not shown as blocks in Fig. 13, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 13.

[0137] As shown in FIG. 13, the decoding device 200 includes a geometry data decoding unit 211 and an attribute data decoding unit 212 .

[0138] The geometry data decoding unit 211 acquires the encoded data of the point cloud (3D data) input to the encoding device 100, decodes the encoded data of the geometry data to generate geometry data, and supplies the generated geometry data and the encoded data of the attribute data to the attribute data decoding unit 212.

[0139] The attribute data decoding unit 212 acquires the geometry data and the encoded data of the attribute data supplied from the geometry data decoding unit 211. Furthermore, the attribute data decoding unit 212 uses the geometry data to decode the encoded data of the attribute data to generate attribute data, and outputs the geometry data and the generated attribute data to the outside of the decoding device 200 as point cloud data.

[0140] These processing units (the geometry data decoding unit 211 and the attribute data decoding unit 212) may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-mentioned processing. Also, each processing unit may have, for example, a CPU, a ROM, a RAM, etc., and may realize the above-mentioned processing by executing a program using them. Of course, each processing unit may have both configurations, and may realize part of the above-mentioned processing by a logic circuit and the other by executing a program. The configurations of each processing unit may be independent of each other, and for example, some processing units may realize part of the above-mentioned processing by a logic circuit, other processing units may realize the above-mentioned processing by executing a program, and still other processing units may realize the above-mentioned processing by both a logic circuit and by executing a program.

[0141] <Geometry data decoding part> Fig. 14 is a block diagram showing a main configuration example of the geometry data decoding unit 211. Note that Fig. 14 shows main processing units, data flows, etc., and is not limited to all that are shown in Fig. 14. In other words, in the geometry data decoding unit 211, there may be processing units that are not shown as blocks in Fig. 14, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 14.

[0142] As shown in FIG. 14, the geometry data decoding unit 211 includes a storage unit 231, a stack 232, a decoding unit 233, a geometry data generation unit 234, and a predicted point generation unit 235.

[0143] The encoded data of the geometry data supplied to the geometry data decoding unit 211 is supplied to a storage unit 231. It should be noted that the encoded data of the attribute data is not processed in the geometry data decoding unit 211, but is supplied to an attribute data decoding unit 212.

[0144] The storage unit 231 stores the encoded data of the geometry data supplied to the geometry data decoding unit 211. Furthermore, the storage unit 231 supplies the encoded data of the geometry data of the point to be decoded to the stack 232 under the control of the decoding unit 233. At that time, the storage unit 231 can apply various methods such as those described above with reference to the table in FIG.

[0145] The stack 232 holds information in a last-in, first-out manner. For example, the stack 232 holds the encoded data of each point supplied from the storage unit 231. In addition, the stack 232 supplies the last-held information to the decoding unit 233 in response to a request from the decoding unit 233.

[0146] The decoding unit 233 performs processing related to decoding of encoded data of geometry data. For example, the decoding unit 233 obtains encoded data of a point last held in the stack 232. The decoding unit 233 also decodes the obtained encoded data to generate geometry data (prediction residuals, etc.). At that time, the decoding unit 233 may apply various methods as described above with reference to the table of FIG. 4. The decoding unit 233 supplies the generated geometry data (prediction residuals, etc.) to the geometry data generation unit 234.

[0147] Furthermore, the decoding unit 233 can perform decoding control so as to decode only a portion of the encoded data requested by a user or the like. For example, the decoding unit 233 can control whether or not to decode for each group hierarchy. Furthermore, the decoding unit 233 can control whether or not to decode for each branch of the reference structure. Then, the decoding unit 233 controls the storage unit 231 to store in the stack 232 only the encoded data of the point to be decoded. By such decoding control, the decoding unit 233 can realize scalable decoding of the geometry data.

[0148] In other words, the decoding unit 233 may decode coded data corresponding to a desired group hierarchy from among coded data in which prediction residuals, which are the differences between the geometry data of each point derived based on the reference structure and its predicted value, are coded, based on layer information indicating a group hierarchy, which is a hierarchy of groups in a reference structure of geometry data in coding of a point cloud in which points of the point cloud, which represents a three-dimensional object as a collection of points, are hierarchically categorized by groups.

[0149] The geometry data generating unit 234 performs processing related to the generation of geometry data. For example, the geometry data generating unit 234 acquires information such as a prediction residual supplied from the decoding unit 233. The geometry data generating unit 234 also acquires a prediction point corresponding to the processing target point (i.e., a predicted value of the geometry data of the processing target point) from the prediction point generating unit 235. Then, the geometry data generating unit 234 uses the acquired prediction residual and predicted value (for example, by adding them together) to generate geometry data of the processing target point. The geometry data generating unit 234 supplies the generated geometry data to the attribute data decoding unit 212.

[0150] The prediction point generating unit 235 performs processing related to the generation of a prediction point, that is, the derivation of a prediction value. For example, the prediction point generating unit 235 acquires information such as geometry data of a processing target point generated in the geometry data generating unit 234. In addition, the prediction point generating unit 235 derives geometry data of a prediction point that can be generated using the geometry data of the processing target point, etc. (for example, a predicted value of the geometry data of a child node of a processing target node). In this case, the prediction point generating unit 235 may apply various methods such as those described above with reference to the table of FIG. 4. For example, the prediction point generating unit 235 may generate a prediction point in the same way as the prediction point generating unit 135 of the encoding device 100. The prediction point generating unit 235 supplies the derived prediction value to the geometry data generating unit 234 as necessary.

[0151] With the above configuration, the decoding device 200 can decode the encoded data using the grouped reference structure of the geometry data. Therefore, as described above, the decoding device 200 can realize scalable decoding and suppress an increase in the load of the decoding process.

[0152] <Decryption process flow> Next, a description will be given of the processing executed by the decoding device 200. The decoding device 200 decodes the encoded data of the point cloud by executing a decoding process. An example of the flow of the decoding process will be described with reference to the flowchart in FIG.

[0153] When the decoding process is started, in step S201, the geometry data decoding unit 211 of the decoding device 200 executes a geometry data decoding process to decode the coded data of the input geometry data of the point cloud and generate geometry data.

[0154] In step S202, the attribute data decoding unit 212 encodes the encoded data of the attribute data of the input point cloud to generate attribute data.

[0155] When the process of step S202 ends, the decoding process ends.

[0156] <Flow of geometry data decoding process> Next, an example of the flow of the geometry data decoding process executed in step S201 in FIG. 15 will be described with reference to the flowchart in FIG.

[0157] When the geometry data decoding process is started, the storage unit 231 stores the encoded data of the supplied geometry data, and in step S231 stores the encoded data of the top node of the reference structure (prediction tree) of the geometry data in the stack 232.

[0158] In step S232, the decoding unit 233 obtains the encoded data of the last stored point (node) from the stack 232.

[0159] In step S233, the decoding unit 233 decodes the encoded data acquired in step S232 to generate layer information. In addition, in step S234, the decoding unit 233 decodes the encoded data acquired in step S232 to generate a prediction mode and a prediction residual of the geometry data.

[0160] In step S235, the geometry data generation unit 234 generates geometry data of the node to be processed by using the prediction residual generated in step S234 and the predicted value of the node to be processed (for example, by adding them together).

[0161] In step S236, the prediction point generating unit 235 generates geometry data (that is, predicted values) of prediction points that can be generated using the geometry data of the node to be processed.

[0162] In step S237, the decoding unit 233 decodes the encoded data acquired in step S232, and generates child node information.

[0163] In step S238, the decoding unit 233 determines whether or not to also decode the child node based on the child node information, layer information, etc., in accordance with decoding control by the user, etc. If it is determined that the child node is also decoded, the process proceeds to step S239.

[0164] In step S239, the decoding unit 233 controls the storage unit 231 to store the encoded data of the child node in the stack 232. When the process of step S239 ends, the process proceeds to step S240. Also, if it is determined in step S238 that the child node is not to be decoded, the process of step S239 is skipped and the process proceeds to step S240.

[0165] In step S240, the decryption unit 233 judges whether the stack 232 is empty. If it is judged that the stack 232 is not empty (that is, information of at least one or more points is stored), the process returns to step S232. That is, the process of steps S232 to S240 is executed for the point last stored in the stack 232.

[0166] Such processing is repeated, and if it is determined in step S240 that the stack is empty, the geometry data decoding processing ends and the process returns to FIG.

[0167] By performing various processes as described above, the decoding device 200 can decode the encoded data using the grouped reference structure of the geometry data. Therefore, as described above, the decoding device 200 can realize scalable decoding and suppress an increase in the load of the decoding process.

[0168] <Transcoder> Fig. 17 is a block diagram showing an example of a configuration of a transcoder, which is one aspect of an information processing device to which the present technology is applied. The transcoder 300 shown in Fig. 17 is a device that decodes encoded data of a point cloud (3D data) and re-encodes the data by performing, for example, parameter conversion. The transcoder 300 transcodes (decodes and encodes) the encoded data of the point cloud generated in the encoding device 100, for example.

[0169] Note that Fig. 17 shows the main processing units, data flows, etc., and is not limited to all that are shown in Fig. 17. In other words, in transcoder 300, there may be processing units that are not shown as blocks in Fig. 17, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 17.

[0170] As shown in FIG. 17, the transcoder 300 includes a geometry data decoding unit 311, a geometry data encoding unit 312, and an attribute data transcode processing unit 313.

[0171] The geometry data decoding unit 311 obtains the encoded data of the point cloud data input to the transcoder 300. The geometry data decoding unit 311 decodes the encoded data to generate geometry data. In this case, the geometry data decoding unit 311 may apply various methods as described above with reference to the table in FIG. 4. For example, the geometry data decoding unit 311 may have a similar configuration to the geometry data decoding unit 211 of the decoding device 200 and perform the same processing. In other words, the geometry data decoding unit 311 may realize scalable decoding. The geometry data decoding unit 311 supplies the encoded data of the attribute data and the generated geometry data to the geometry data encoding unit 312.

[0172] The geometry data encoding unit 312 acquires the coded data of the attribute data and the geometry data supplied from the geometry data decoding unit 311. The geometry data encoding unit 312 re-encodes the geometry data to generate coded data of the geometry data. In this case, the geometry data encoding unit 312 may apply various methods as described above with reference to the table in FIG. 4. For example, the geometry data encoding unit 312 may have a configuration similar to that of the geometry data encoding unit 111 of the encoding device 100 and perform similar processing. That is, the geometry data encoding unit 312 may group the reference structure of the geometry data into a hierarchical structure. After determining the reference structure, the geometry data encoding unit 312 may also reduce the number of points and perform coding. That is, the geometry data encoding unit 312 may be configured to control the bit rate of the coded data to be generated. The geometry data encoding unit 312 supplies the coded data of the attribute data and the coded data of the generated geometry data to the attribute data transcode processing unit 313.

[0173] Changes to parameters of the geometry data, such as reducing the number of points, may be performed in the geometry data decoding unit 311 (by scalable decoding), in the geometry data encoding unit 312, or in both.

[0174] The attribute data transcoding processing unit 313 performs processing related to transcoding of attribute data. For example, the attribute data transcoding processing unit 313 acquires encoded data of geometry data and encoded data of attribute data supplied from the geometry data encoding unit 312. The attribute data transcoding processing unit 313 also decodes and re-encodes (transcodes) the acquired encoded data of attribute data using a predetermined method. The attribute data transcoding processing unit 313 outputs the encoded data of geometry data and the generated encoded data of attribute data to the outside of the transcoder 300 as transcoding results.

[0175] With the above configuration, the transcoder 300 can reduce the number of points during transcoding. In other words, the transcoder 300 can suppress an increase in the transcoding load. In addition, the transcoder 300 can control the bit rate of the encoded data of the geometry data generated by transcoding.

[0176] These processing units (the geometry data decoding unit 311 to the attribute data transcoding unit 313) may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-mentioned processing. Also, each processing unit may have, for example, a CPU, a ROM, a RAM, etc., and may realize the above-mentioned processing by executing a program using them. Of course, each processing unit may have both configurations, and may realize part of the above-mentioned processing by a logic circuit and the other by executing a program. The configurations of each processing unit may be independent of each other, and for example, some processing units may realize part of the above-mentioned processing by a logic circuit, other processing units may realize the above-mentioned processing by executing a program, and still other processing units may realize the above-mentioned processing by both a logic circuit and by executing a program.

[0177] <Transcoding process flow> Next, a description will be given of the processing executed by the transcoder 300. The transcoder 300 transcodes the encoded data of the point cloud by executing a transcoding process. An example of the flow of the transcoding process will be described with reference to the flowchart in FIG.

[0178] When the transcoding process is started, in step S301, the geometry data decoding unit 311 of the transcoder 300 executes a geometry data decoding process to decode the encoded data and generate geometry data. For example, the geometry data decoding unit 311 can perform this geometry data decoding process in a similar flow to the geometry data decoding process described with reference to the flowchart of FIG.

[0179] In step S302, the geometry data encoding unit 312 executes a geometry data encoding process to encode the geometry data and generate the encoded data. For example, the geometry data encoding unit 312 can execute this geometry data encoding process in a similar flow to the geometry data encoding process described with reference to the flowchart of FIG.

[0180] In step S303, the attribute data transcoder 313 transcodes the attribute data. When the process of step S303 ends, the transcode process ends.

[0181] By executing each process as described above, the transcoder 300 can reduce the number of points during transcoding. In other words, the transcoder 300 can suppress an increase in the transcoding load. In addition, the transcoder 300 can control the bit rate of the encoded data of the geometry data generated by transcoding.

[0182] <2. Second embodiment> <Prediction residual coding of attribute data> However, in the case of predictive geometry coding described in Non-Patent Document 2, the processing target is geometry data, and attribute data is coded by a different method. Therefore, it is necessary to apply different coding / decoding methods to the geometry data and the attribute data, respectively, which may increase costs.

[0183] Therefore, as shown in the top row of the table in Figure 19, predictive geometry coding is extended, and as shown in the second row from the top (row "2") of the table in Figure 19, predictive geometry coding is applied to encoding of attribute data.

[0184] For example, a reference structure for attribute data is formed in encoding a point cloud that represents a three-dimensional object as a collection of points, and based on the formed reference structure, a predicted value of the attribute data for each point is derived, a prediction residual, which is the difference between the attribute data and the predicted value, is derived, and the derived prediction residual of the attribute data for each point is encoded.

[0185] For example, an information processing device may include a reference structure forming unit that forms a reference structure of attribute data in encoding a point cloud that represents a three-dimensional object as a collection of points, a prediction residual derivation unit that derives a predicted value of the attribute data for each point based on the reference structure formed by the reference structure forming unit and derives a prediction residual that is the difference between the attribute data and the predicted value, and an encoding unit that encodes the prediction residual of the attribute data for each point derived by the prediction residual derivation unit.

[0186] In this way, by applying a method similar to predictive geometry coding to the coding of attribute data, the same effect as in the case of geometry data can be obtained in coding attribute data. For example, since the prediction residual is coded, it is possible to suppress an increase in the amount of code of the coded data of the attribute data. In addition, it is possible to control the storage capacity when storing the coded data and the transmission rate when transmitting the coded data. In other words, it is possible to control the bit rate of the coded data of the attribute data.

[0187] In addition, as shown in the third row from the top of the table in Figure 19 (row "2-1"), in this case, the reference structure of the attribute data may be made common between the geometry data and the attribute data, as shown in the fourth row from the top of the table in Figure 19 (row "2-1-1").

[0188] For example, the reference structure applied in predictive geometry coding (i.e., encoding of geometry data) may also be applied to encoding of attribute data. In this way, it is not necessary to form a reference structure for attribute data, so that an increase in the load of the encoding process can be suppressed. In addition, a common reference structure is used for geometry data and attribute data, so that scalable decoding of point cloud data (geometry data and attribute data) is possible. Therefore, an increase in the load of the decoding process can be suppressed. Also, decoding with lower delay is possible.

[0189] In this manner, when a reference structure is shared between geometry data and attribute data, the reference structure may be formed based on the geometry data, the reference structure may be formed based on the attribute data, or the reference structure may be formed based on both the geometry data and the attribute data.

[0190] Also, as shown in the fifth row from the top (row "2-1-2") of the table shown in Fig. 19, the reference structure of all or part of the attribute data may be formed independently of the reference structure of the geometry data. For example, parameters related to the color (RGB) of the attribute data may be a common reference structure with the geometry data, and parameters such as reflectance of the attribute data may be formed independently of the reference structure of the geometry data.

[0191] In addition, as shown in the sixth row from the top of the table in FIG. 19 (row "2-2"), the attribute data of the prediction point of the node to be processed may be the same as the attribute data of the parent node of the node to be processed, as shown in the seventh row from the top of the table in FIG. 19 (row "2-2-1").

[0192] Also, as shown in the eighth row from the top (row "2-2-2") of the table shown in FIG. 19, the attribute data of the prediction point of the node to be processed may be the average of the attribute data of the node to be processed with the parent node and the attribute data of the parent node. Furthermore, as shown in the ninth row from the top (row "2-2-3") of the table shown in FIG. 19, the attribute data of the prediction point of the node to be processed may be the weighted average of the attribute data of the node to be processed with the parent node and the attribute data of the parent node, the weight being the inverse of the distance between the points corresponding to these nodes. Also, as shown in the tenth row from the top (row "2-2-4") of the table shown in FIG. 19, the attribute data of the prediction point of the node to be processed may be the average of the attribute data of k neighboring nodes of the decoded node.

[0193] As shown in the 11th row (row "2-3") from the top of the table shown in Fig. 19, a prediction point that minimizes the prediction residual of the geometry data may be selected as shown in the 12th row (row "2-3-1") from the top of the table shown in Fig. 19. As shown in the 13th row (row "2-3-2") from the top of the table shown in Fig. 19, a prediction point that minimizes the prediction residual of the attribute data may be selected.

[0194] Furthermore, as shown in the bottom row (row "2-3-3") of the table shown in FIG. 19, a prediction point that minimizes the prediction residual of the geometry data and attribute data may be selected.

[0195] For example, as shown in the following equation (1), the prediction residuals of each variable of the geometry data (x coordinate, y coordinate, z coordinate) and each variable of the attribute data (color, reflectance, etc.) may be used as variables of the evaluation function f(), and a prediction point may be selected based on the sum of the prediction residuals of those variables.

[0196] f(diffX,diffY,diffZ,diffAttr1,...) = sum(diffX + diffY+ diffZ + diffAttr1 +...) (1)

[0197] In addition, in formula (1), diff[variable name] indicates the prediction residual of each variable. In this way, a prediction mode can be selected taking into consideration not only the position but also each variable of the attribute data, so that encoding and decoding can be performed in accordance with the characteristics of not only the geometry data but also the attribute data. For example, when there are many variables of the attribute data (when there are many dimensions) or when the range of the variables of the attribute data is larger than that of the variables of the geometry data, a prediction mode can be selected so as to reduce the dependency on the variables of the geometry data (in other words, to improve the dependency on the variables of the attribute data).

[0198] <Encoding device> Next, a device to which the present technology is applied will be described. Fig. 20 is a block diagram showing an example of the configuration of an encoding device, which is one aspect of an information processing device to which the present technology is applied. The encoding device 400 shown in Fig. 20 is a device that encodes a point cloud (3D data) similarly to the encoding device 100. However, the encoding device 400 encodes the point cloud by applying the present technology described above with reference to Fig. 19, for example.

[0199] Note that Fig. 20 shows the main processing units, data flows, etc., and is not limited to all that is shown in Fig. 20. In other words, in encoding device 400, there may be processing units that are not shown as blocks in Fig. 20, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 20.

[0200] As shown in FIG. 20, the encoding device 400 includes a reference structure forming unit 411, a stack 412, a prediction mode determining unit 413, an encoding unit 414, and a prediction point generating unit 415.

[0201] The point cloud data (geometry data and attribute data) supplied to the encoding device 400 is supplied to a reference structure forming unit 411.

[0202] The reference structure forming unit 411 generates a reference structure (prediction tree) for encoding the point cloud for both the supplied geometry data and attribute data. In this case, the reference structure forming unit 411 can form the reference structure by applying various methods as described above with reference to the table in FIG. 19. For example, the reference structure forming unit 411 can form a common reference structure for the geometry data and the attribute data. In addition, the reference structure forming unit 411 can form the reference structure for the geometry data and the reference structure for the attribute data independently of each other. The reference structure forming unit 411 supplies the geometry data and attribute data of the processing target point to the stack 412 according to the formed reference structure.

[0203] The stack 412 holds information in a last-in, first-out manner. For example, the stack 412 holds information of each point supplied from the reference structure forming unit 131. In addition, the stack 412 supplies the prediction mode determination unit 413 with the last-held information in response to a request from the prediction mode determination unit 413. The stack 412 can perform these processes on both geometry data and attribute data.

[0204] The prediction mode determination unit 413 performs processing related to determining a prediction mode (prediction point). For example, the prediction mode determination unit 413 obtains information on the point last held in the stack 412. Furthermore, the prediction mode determination unit 413 obtains information on the prediction point of that point (i.e., the predicted value of the processing target point) and the like from the prediction point generation unit 415. When there are multiple prediction points corresponding to the processing target point as in the example of FIG. 2, all of them are obtained. Then, the prediction mode determination unit 413 determines the prediction point (i.e., the prediction mode) to be applied.

[0205] The prediction mode determination unit 413 can perform such processing on both the geometry data and the attribute data. In addition, the prediction mode determination unit 413 can apply various methods such as those described above with reference to the table in FIG. 19 during the processing.

[0206] For example, the prediction mode determination unit 413 can select a prediction point (prediction mode) that minimizes the prediction residual of the geometry data. Also, the prediction mode determination unit 413 can select a prediction point (prediction mode) that minimizes the prediction residual of the attribute data. Furthermore, the prediction mode determination unit 413 can select a prediction point (prediction mode) that minimizes the prediction residual of the geometry data and the attribute data.

[0207] The prediction mode determination unit 413 supplies information about each point (for example, prediction residuals of the geometry data and attribute data in the selected prediction mode) to the encoding unit 414.

[0208] The encoding unit 414 acquires information (e.g., a prediction residual of a selected prediction mode, etc.) provided by the prediction mode determination unit 413, encodes the information, and generates encoded data. The encoding unit 414 can perform such processing on both geometry data and attribute data.

[0209] The encoding unit 414 supplies the generated encoded data as encoded data of geometry data and attribute data to an external device (e.g., a decoding side) of the encoding device 400. In addition, the encoding unit 414 supplies the geometry data and attribute data of the processing target point to the prediction point generating unit 415.

[0210] The prediction point generating unit 415 performs processing related to the generation of a prediction point, that is, the derivation of a prediction value. For example, the prediction point generating unit 415 acquires information such as geometry data and attribute data of a processing target point supplied from the encoding unit 414. In addition, the prediction point generating unit 415 derives geometry data and attribute data of a prediction point that can be generated using the geometry data and attribute data of the processing target point (for example, a predicted value of the geometry data and attribute data of a child node of a processing target node). At that time, the prediction point generating unit 415 may apply various methods such as those described above with reference to the table of FIG. 19. The prediction point generating unit 415 supplies the derived prediction value to the prediction mode determining unit 413 as necessary.

[0211] With the above configuration, the encoding device 400 can encode attribute data by a method similar to predictive geometry coding. Therefore, the encoding device 400 can obtain the same effect in encoding attribute data as in the case of geometry data. For example, since the encoding device 400 encodes the prediction residual, the encoding device 400 can suppress an increase in the amount of encoded data of the attribute data. In addition, the encoding device 400 can control the storage capacity when storing the encoded data and the transmission rate when transmitting the encoded data. In other words, the encoding device 400 can control the encoded data bit rate of the attribute data.

[0212] These processing units (reference structure forming unit 411 to predicted point generating unit 415) may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-mentioned processing. Also, each processing unit may have, for example, a CPU, a ROM, a RAM, etc., and may realize the above-mentioned processing by executing a program using them. Of course, each processing unit may have both configurations, and may realize part of the above-mentioned processing by a logic circuit and the other by executing a program. The configurations of each processing unit may be independent of each other, and for example, some processing units may realize part of the above-mentioned processing by a logic circuit, other processing units may realize the above-mentioned processing by executing a program, and still other processing units may realize the above-mentioned processing by both a logic circuit and executing a program.

[0213] <Encoding process flow> Next, a description will be given of the processing executed by the encoding device 400. The encoding device 400 encodes point cloud data by executing an encoding process. An example of the flow of the encoding process will be described with reference to the flowchart in FIG.

[0214] When the encoding process is started, in step S401, the reference structure forming unit 411 executes a reference structure forming process to form a reference structure (prediction tree) of geometry data and attribute data. The reference structure forming unit 411 may form the reference structure by a method similar to the method described in Non-Patent Document 2, for example.

[0215] In step S402, the reference structure formation unit 411 stores in the stack 412 the geometry data, attribute data, and the like of the top node of the reference structure formed in step S401.

[0216] In step S403, the prediction mode determination unit 413 obtains, from the stack 412, the geometry data, attribute data, and the like, of the point (node) that was last stored.

[0217] In step S404, the prediction mode determination unit 413 sets the point about which information was acquired in step S403 as a processing target, derives a prediction residual of the geometry data for the processing target point, and determines a prediction mode.

[0218] In step S405, the encoding unit 414 encodes the prediction mode determined in step S404. In addition, in step S406, the encoding unit 414 encodes the prediction residual of the geometry data in the prediction mode determined in step S404.

[0219] In step S407, the prediction mode determination unit 413 performs recolor processing to derive a prediction residual of the attribute data. In step S408, the encoding unit 414 encodes the prediction residual of the attribute data.

[0220] In step S409, the encoding unit 414 encodes child node information indicating which node is a child node of the processing target node.

[0221] In step S410, the reference structure formation unit 411 determines whether or not to encode the child node of the processing target node based on encoding control by a user, etc. If it is determined that the child node is to be encoded, the process proceeds to step S411.

[0222] In step S411, the reference structure forming unit 411 stores the geometry data of the child node, etc., in the stack 412. When the process of step S411 ends, the process proceeds to step S412. Also, if it is determined in step S410 that the child node is not to be encoded, the process of step S411 is skipped and the process proceeds to step S412.

[0223] In step S412, the prediction point generating unit 415 generates geometry data and attribute data of a prediction point that can be generated using information (geometry data, attribute data, etc.) of the processing target point.

[0224] In step S413, the prediction mode determination unit 413 determines whether the stack 412 is empty. If it is determined that the stack 412 is not empty (that is, information of at least one or more points is stored), the process returns to step S403. That is, the processes of steps S403 to S413 are executed for the point last stored in the stack 412 as the processing target.

[0225] Such processing is repeated, and when it is determined in step S413 that the stack is empty, the geometry data encoding processing ends.

[0226] By performing various processes as described above, the encoding device 400 can encode attribute data by a method similar to predictive geometry coding. Therefore, the encoding device 400 can obtain the same effect in encoding attribute data as in the case of geometry data. For example, since the encoding device 400 encodes the prediction residual, the encoding device 400 can suppress an increase in the amount of encoded data of the attribute data. In addition, the encoding device 400 can control the storage capacity when storing the encoded data and the transmission rate when transmitting the encoded data. In other words, the encoding device 400 can control the encoded data bit rate of the attribute data.

[0227] <Decryption device> Fig. 22 is a block diagram showing an example of a configuration of a decoding device, which is one aspect of an information processing device to which the present technology is applied. The decoding device 500 shown in Fig. 22 is a device that decodes encoded data of a point cloud (3D data). The decoding device 500 decodes the encoded data of the point cloud generated in the encoding device 400, for example.

[0228] Note that Fig. 22 shows the main processing units, data flows, and the like, and is not limited to all that is shown in Fig. 22. In other words, in the decoding device 500, there may be processing units that are not shown as blocks in Fig. 22, and there may be processing and data flows that are not shown as arrows or the like in Fig. 22.

[0229] As shown in FIG. 22, the decoding device 50 includes a storage unit 511, a stack 512, a decoding unit 513, a point data generating unit 514, and a predicted point generating unit 515.

[0230] The encoded data of the geometry data and attribute data supplied to the decoding device 500 is supplied to a storage unit 511 .

[0231] The storage unit 511 stores the encoded data of the geometry data and attribute data supplied to the decoding device 500. In addition, the storage unit 511 supplies the encoded data of the geometry data and attribute data of the point to be decoded to the stack 512 under the control of the decoding unit 513.

[0232] The stack 512 holds information in a last-in, first-out manner. For example, the stack 512 holds encoded data of geometry data and attribute data of each point supplied from the storage unit 511. In addition, in response to a request from the decoding unit 513, the stack 512 supplies the decoding unit 513 with the last information held therein (e.g., encoded data of geometry data and attribute data).

[0233] The decoding unit 513 performs processing related to decoding of encoded data for both geometry data and attribute data. For example, the decoding unit 513 obtains encoded data of a point last held in the stack 512. The decoding unit 513 also decodes the obtained encoded data to generate geometry data (prediction residuals, etc.) and attribute data (prediction residuals, etc.). At this time, the decoding unit 513 may apply various methods as described above with reference to the table of FIG. 19. The decoding unit 513 supplies the generated geometry data (prediction residuals, etc.) and attribute data (prediction residuals, etc.) to the point data generation unit 514.

[0234] The point data generating unit 514 performs processing related to the generation of point data (geometry data and attribute data). For example, the point data generating unit 514 acquires information such as a prediction residual supplied from the decoding unit 513. The point data generating unit 514 also acquires a predicted point corresponding to the processing target point (i.e., a predicted value of the geometry data and a predicted value of the attribute data of the processing target point) from the predicted point generating unit 515. Then, the point data generating unit 514 uses the acquired prediction residual and predicted value (for example, by adding them together) to generate geometry data and attribute data of the processing target point, respectively. The point data generating unit 514 supplies the generated geometry data and attribute data to the outside of the decoding device 500.

[0235] The prediction point generating unit 515 performs processing related to the generation of a prediction point, that is, the derivation of a prediction value. For example, the prediction point generating unit 515 acquires information such as geometry data and attribute data of a processing target point generated in the point data generating unit 514. In addition, the prediction point generating unit 515 derives geometry data of a prediction point that can be generated using the geometry data and attribute data of the processing target point (for example, a predicted value of geometry data of a child node of a processing target node) and attribute data of the prediction point. In this case, the prediction point generating unit 515 may apply various methods such as those described above with reference to the table of FIG. 19. For example, the prediction point generating unit 515 may generate a prediction point in the same way as the prediction point generating unit 415 of the encoding device 400. The prediction point generating unit 515 supplies the derived prediction value to the point data generating unit 514 as necessary.

[0236] With the above-described configuration, the decoding device 500 can decode not only the encoded data of the geometry data but also the encoded data of the attribute data. Therefore, the decoding device 500 can suppress an increase in the load of the decoding process.

[0237] <Decryption process flow> Next, a description will be given of the processing executed by the decoding device 500. The decoding device 500 decodes the encoded data of the point cloud by executing a decoding process. An example of the flow of the decoding process will be described with reference to the flowchart in FIG.

[0238] When the decoding process starts, the memory unit 511 stores the encoded data of the supplied geometry data and attribute data, and in step S501, stores the encoded data of the top node of the reference structure (prediction tree) of the geometry data and attribute data in the stack 232.

[0239] In step S502, the decoding unit 513 obtains the encoded data of the last stored point (node) from the stack 512.

[0240] In step S503, the decoding unit 513 decodes the encoded data acquired in step S502, and generates a prediction mode and a prediction residual of the geometry data.

[0241] In step S504, the point data generating unit 514 generates geometry data of the node to be processed by using the prediction residual generated in step S503 and the predicted value of the node to be processed (for example, by adding them together).

[0242] In step S505, the decoding unit 513 decodes the encoded data acquired in step S502 to generate a prediction residual of the attribute data. In step S506, the point data generating unit 514 generates attribute data of the node to be processed by using the prediction residual generated in step S503 and the predicted value of the node to be processed (for example, by adding them together).

[0243] In step S507, the prediction point generating unit 515 generates geometry data and attribute data (i.e., predicted value) of a prediction point that can be generated using the geometry data of the node to be processed. The prediction residual generated in step S503 and the predicted value of the node to be processed (for example, by adding them together) are used to generate the geometry data and attribute data of the prediction point.

[0244] In step S508, the decoding unit 513 decodes the encoded data acquired in step S502, and generates child node information.

[0245] In step S509, the decoding unit 513 determines whether or not to also decode the child node based on the child node information, layer information, etc., in accordance with decoding control by a user, etc. If it is determined that the child node is also decoded, the process proceeds to step S510.

[0246] In step S510, the decoding unit 513 controls the storage unit 511 to store the encoded data of the child node in the stack 512. When the process of step S510 ends, the process proceeds to step S511. Also, if it is determined in step S509 that the child node is not to be decoded, the process of step S510 is skipped and the process proceeds to step S511.

[0247] In step S511, the decryption unit 513 determines whether the stack 512 is empty. If it is determined that the stack 512 is not empty (that is, information on at least one or more points is stored), the process returns to step S502. That is, the process of steps S502 to S511 is executed for the point last stored in the stack 502.

[0248] Such processing is repeated, and if it is determined in step S511 that the stack is empty, the geometry data decoding processing ends and the process returns to FIG.

[0249] By performing various processes as described above, the decoding device 500 can decode the encoded data of not only the geometry data but also the attribute data. Therefore, as described above, the decoding device 500 can suppress an increase in the load of the decoding process.

[0250] <Transcoder> Fig. 24 is a block diagram showing an example of the configuration of a transcoder, which is one aspect of an information processing device to which the present technology is applied. The transcoder 600 shown in Fig. 24 is a device that decodes encoded data of a point cloud (3D data) and re-encodes the data by performing, for example, parameter conversion. The transcoder 600 transcodes (decodes and encodes) the encoded data of the point cloud generated in the encoding device 400, for example.

[0251] Note that Fig. 24 shows the main processing units, data flows, etc., and is not limited to all that are shown in Fig. 24. In other words, in the transcoder 600, there may be processing units that are not shown as blocks in Fig. 24, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 24.

[0252] As shown in FIG. 24, the transcoder 600 includes a decoding unit 611 and an encoding unit 612 .

[0253] The decoding unit 611 obtains the encoded data of the point cloud data input to the transcoder 600. The decoding unit 611 decodes the encoded data to generate geometry data and attribute data. In this case, the decoding unit 611 may apply various methods as described above with reference to the table in FIG. 19. For example, the decoding unit 611 may have a similar configuration to the decoding device 500 and perform similar processing. The decoding unit 611 supplies the generated geometry data and attribute data to the encoding unit 612.

[0254] The encoding unit 612 acquires the geometry data and attribute data supplied from the decoding unit 611. The encoding unit 612 re-encodes the geometry data to generate encoded data of the geometry data. The encoding unit 612 also re-encodes the attribute data to generate encoded data of the attribute data. In this case, the encoding unit 612 may apply various methods as described above with reference to the table in FIG. 19. For example, the encoding unit 612 may have a similar configuration to the encoding device 400 and perform similar processing.

[0255] In addition, changes to parameters of geometry data and attribute data during transcoding, such as reducing the number of points, may be performed in the decoding unit 611 (by scalable decoding), in the encoding unit 612, or in both.

[0256] The encoding unit 612 outputs the generated encoded data of the geometry data and encoded data of the attribute data to the outside of the transcoder 600 as a transcoding result.

[0257] With the above configuration, the transcoder 600 can reduce the number of points during transcoding. In other words, the transcoder 600 can suppress an increase in the transcoding load. In addition, the transcoder 600 can control the bit rate of the encoded data of the geometry data and attribute data generated by transcoding.

[0258] In addition, these processing units (the decoding unit 611 and the encoding unit 612) have an arbitrary configuration. For example, each processing unit may be configured by a logic circuit that realizes the above-mentioned processing. Also, each processing unit may have, for example, a CPU, a ROM, a RAM, etc., and may realize the above-mentioned processing by executing a program using them. Of course, each processing unit may have both configurations, and may realize part of the above-mentioned processing by a logic circuit and the other by executing a program. The configurations of each processing unit may be independent of each other, and for example, some processing units may realize part of the above-mentioned processing by a logic circuit, other processing units may realize the above-mentioned processing by executing a program, and still other processing units may realize the above-mentioned processing by both a logic circuit and executing a program.

[0259] <Transcoding process flow> Next, a description will be given of the processing executed by the transcoder 600. The transcoder 600 transcodes the encoded data of the point cloud by executing a transcoding process. An example of the flow of the transcoding process will be described with reference to the flowchart in FIG.

[0260] When the transcoding process is started, the decoding unit 611 of the transcoder 600 executes a decoding process in step S601 to decode the encoded data and generate geometry data and attribute data. For example, the decoding unit 611 can perform this decoding process in a similar flow to the decoding process described with reference to the flowchart of FIG.

[0261] In step S602, the encoding unit 612 executes an encoding process to encode the geometry data and the attribute data, and generates the encoded data. For example, the encoding unit 612 can perform this encoding process in a similar flow to the encoding process described with reference to the flowchart of Fig. 21. When the process of step S602 ends, the transcoding process ends.

[0262] By executing each process as described above, the transcoder 600 can reduce the number of points during transcoding. In other words, the transcoder 600 can suppress an increase in the transcoding load. In addition, the transcoder 600 can control the bit rate of the encoded data of the geometry data generated by transcoding.

[0263] <3. Third embodiment> <Hierarchical reference structure and prediction residual coding of attribute data> Note that the present technology described in the first embodiment and the present technology described in the second embodiment may be combined.

[0264] For example, as shown in the second row from the top (row "1") of the table shown in Fig. 4, when forming groups of points and forming a reference structure with the groups as a hierarchy, predictive geometry coding may be applied to encoding attribute data, as shown in the second row from the top (row "2") of the table shown in Fig. 19. For example, as shown in the fourth row from the top (row "2-1-1") of the table shown in Fig. 19, a common reference structure may be used for geometry data and attribute data.

[0265] For example, in the case of forming a reference structure for geometry data in encoding a point cloud hierarchically organized into groups into which points of the point cloud, which represents a three-dimensional object as a collection of points, deriving a predicted value of the geometry data for each point based on the formed reference structure, deriving a prediction residual which is the difference between the geometry data and the predicted value, and encoding the derived prediction residual of the geometry data for each point, it is also possible to further derive a predicted value of attribute data for each point based on the formed reference structure, further derive a prediction residual which is the difference between the attribute data and the predicted value, and further encode the derived prediction residual of the attribute data for each point.

[0266] Furthermore, as shown in the seventh to tenth rows from the top of the table in Figure 19, the attribute data of the predicted point of the node to be processed may be the same as the attribute data of the parent node of the node to be processed, or the average of the attribute data of the parent node of the node to be processed and the attribute data of that parent node, the attribute data of the predicted point of the node to be processed may be a weighted average of the attribute data of the parent node of the node to be processed and the attribute data of that parent node, where the weight is the inverse of the distance between the points corresponding to those nodes, or the average of the attribute data of k nodes in the vicinity of the decoded node.

[0267] For example, the attribute data of the parent node to which the node to be processed in the reference structure belongs, the average of the attribute data of the parent node and the attribute data of its parent node's parent node, the weighted average of the attribute data of the parent node and the attribute data of its parent node's parent node, or the average of the attribute data of nodes in the vicinity of the node to be processed may be used as a predicted value of the attribute data of the node to be processed, and a prediction residual of the attribute data may be derived.

[0268] Also, as shown in the 12th to 15th rows from the top of the table in FIG. 19, a prediction point that results in the smallest prediction residual for the geometry data, a prediction point that results in the smallest prediction residual for the attribute data, or a prediction point that results in the smallest prediction residual for both the geometry data and the attribute data may be selected.

[0269] Also, as shown in the fifth row from the top of the table in Fig. 19, the reference structure of all or a part of the attribute data may be formed independently of the reference structure of the geometry data. For example, parameters related to the color (RGB) of the attribute data may be a common reference structure with the geometry data, and parameters of the attribute data such as reflectance may be formed independently of the reference structure of the geometry data.

[0270] Conversely, when predictive geometry coding is applied to encoding attribute data, as shown in the second row from the top (row "1") of the table shown in Figure 19, groups of points may be formed and a reference structure may be formed in which the groups form a hierarchy, as shown in the second row from the top (row "1") of the table shown in Figure 4.

[0271] For example, a reference structure for geometry data in encoding a point cloud may be formed in which the points of the point cloud, which represents a three-dimensional object as a collection of points, are hierarchically organized according to groups into which the points of the point cloud are classified. Based on the reference structure thus formed, a predicted value of the geometry data for each point may be derived, a prediction residual, which is the difference between the geometry data and the predicted value, and the derived prediction residual of the geometry data for each point may be encoded. In this case, a reference structure for attribute data may be formed in which the points of the point cloud are hierarchically organized according to groups into which the points of the point cloud are classified.

[0272] Also, a hierarchical reference structure may be formed by grouping the points, sorting the points by group, and setting the reference destination of the attribute data of each point in the sorted order.

[0273] The points may also be grouped according to their location, or their characteristics in the point cloud, or both.

[0274] Also, for each point, layer information indicating a group hierarchy, which is a hierarchy according to groups in the reference structure, may be generated, and the generated layer information may be further encoded.

[0275] Furthermore, layer information may be generated that indicates the group hierarchy of each child node belonging to the processing target node in the reference structure as a relative value to the group hierarchy of the processing target node.

[0276] Furthermore, the layer information may be generated to indicate the group hierarchy of the node to be processed in the reference structure as a relative value to the group hierarchy of the parent node to which the node to be processed belongs.

[0277] Furthermore, whether to encode the prediction residual may be selected for each group hierarchy, which is a hierarchy based on groups in the reference structure, or for each branch of the reference structure, or for both, and the prediction residual of the group hierarchy or branch selected for encoding may be encoded.

[0278] <Encoding device> In this case, the encoding device has, for example, the configuration of the encoding device 400 shown in FIG. 20, and the reference structure forming unit 411 has the configuration shown in FIG. 9. With such a configuration, the encoding device can obtain the effects described in the first and second embodiments. That is, the reference structure can be hierarchically organized into groups, and the prediction residual of the attribute data can be encoded. Therefore, an increase in the load of the encoding process can be suppressed. Also, the bit rate of the encoded data of the geometry data generated by the encoding can be controlled.

[0279] <Encoding process flow> Moreover, an example of the flow of the encoding process in this case will be described with reference to the flowchart in Fig. 26. In this case, as shown in the flowchart in Fig. 26, in the encoding process, in step S701, the reference structure formation unit 411 executes the reference structure formation process in the same manner as in step S131 in Fig. 11, according to the flow of the flowchart in Fig. 12.

[0280] The processes in steps S702 to S709 are executed in the same manner as the processes in steps S402 to S409 in FIG.

[0281] In step S710, the encoding unit 414 encodes the layer information generated in step S701, similarly to step S138 in FIG.

[0282] The processes in steps S711 to S714 are executed in the same manner as the processes in steps S410 to S413 in FIG.

[0283] By executing various processes as described above, it is possible to obtain the effects described in the first and second embodiments. That is, the reference structure can be hierarchically organized into groups, and the prediction residual of the attribute data can be encoded. Therefore, it is possible to suppress an increase in the load of the encoding process. In addition, it is possible to control the bit rate of the encoded data of the geometry data generated by the encoding.

[0284] <Decryption device> The decoding device in this case has a configuration similar to that of a decoding device 500 shown in FIG. 22, for example. <Decryption process flow> An example of the flow of the decoding process in this case will be described with reference to the flowchart in Fig. 27. In this case, when the decoding process is started, the processes in steps S801 and S802 are executed in the same manner as the processes in steps S501 and S502 in Fig. 23.

[0285] In step S803, the decoding unit decodes the encoded data and generates layer information in the same manner as in step S233 of Fig. 16. Thereafter, the processes of steps S804 to S812 are executed in the same manner as the processes of steps S503 to S511 of Fig. 23.

[0286] By executing various processes as described above, it is possible to obtain the effects described in the first and second embodiments. That is, the reference structure can be hierarchically organized into groups, and the prediction residual of the attribute data can be encoded. Therefore, it is possible to suppress an increase in the load of the decoding process. In addition, it is possible to realize scalability of the decoding, and it is possible to control the bit rate of the encoded data of the geometry data generated by the encoding.

[0287] <Transcoder> The transcoder in this case has a configuration similar to that of the transcoder 600 shown in FIG.

[0288] <Transcoding process flow> The transcoding process in this case is performed in the same manner as in the flowchart shown in Fig. 25. However, in step S601, the encoding process is performed in the same manner as in the flowchart shown in Fig. 26. In addition, in step S602, the encoding process is performed in the same manner as in the flowchart shown in Fig. 27.

[0289] By performing such processing, the transcoder in this case can reduce the number of points during transcoding. In other words, it is possible to suppress an increase in the transcoding load. In addition, it is possible to achieve scalability in decoding and to control the bit rate of the encoded data of geometry data and attribute data generated by transcoding.

[0290] <4. Notes> <Control information> Control information related to the present technology described in each of the above embodiments may be transmitted from the encoding side to the decoding side. For example, control information (e.g., enabled_flag) that controls whether or not to permit (or prohibit) application of the present technology described above may be transmitted. In addition, for example, control information that specifies a range (e.g., upper or lower limit of block size, or both, slice, picture, sequence, component, view, layer, etc.) in which application of the present technology described above is permitted (or prohibited) may be transmitted.

[0291] <Surroundings / nearby> In this specification, positional relationships such as "nearby" and "around" may include not only spatial positional relationships but also temporal positional relationships.

[0292] <Computer> The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, etc., capable of executing various functions by installing various programs.

[0293] FIG. 28 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0294] In a computer 900 shown in FIG. 28, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0295] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0296] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0297] In the computer configured as above, the series of processes described above are performed by the CPU 901, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data and the like necessary for the CPU 901 to execute various processes as appropriate.

[0298] The program executed by the computer can be applied by recording it on a removable medium 921 such as a package medium, for example. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by mounting the removable medium 921 on the drive 915.

[0299] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0300] In addition, this program can also be installed in advance in the ROM 902 or the storage unit 913.

[0301] <Target of application of this technology> Although the above describes the application of the present technology to the encoding and decoding of point cloud data, the present technology is not limited to these examples and can be applied to the encoding and decoding of 3D data of any standard. For example, in encoding and decoding of mesh data, mesh data may be converted into point cloud data and encoded and decoded by applying the present technology. In other words, various processes such as encoding and decoding methods and specifications of various data such as 3D data and metadata are arbitrary as long as they do not conflict with the present technology described above. In addition, some of the processes and specifications described above may be omitted as long as they do not conflict with the present technology.

[0302] This technology can be applied to any configuration. For example, this technology can be applied to various electronic devices such as transmitters and receivers (e.g., television receivers and mobile phones) in satellite broadcasting, wired broadcasting such as cable TV, distribution over the Internet, and distribution to terminals by cellular communication, or devices (e.g., hard disk recorders and cameras) that record images on media such as optical disks, magnetic disks, and flash memories, and play images from these storage media.

[0303] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set in which other functions are added to a unit (e.g., a video set).

[0304] In addition, for example, the present technology can also be applied to a network system composed of multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and cooperate with each other to perform processing via a network. For example, the present technology may be implemented in a cloud service that provides services related to images (video images) to any terminal such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0305] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0306] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. In addition, the applications are also arbitrary.

[0307] <Other> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Thus, the value that this "flag" can take may be, for example, a binary value of 1 / 0, or may be three or more values. In other words, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, it is assumed that identification information (including flags) is not only included in the bit stream, but also includes difference information of the identification information with respect to certain reference information in the bit stream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information with respect to the reference information.

[0308] Also, various information (metadata, etc.) related to the coded data (bit stream) may be transmitted or recorded in any form as long as it is associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, data associated with each other may be combined into one piece of data, or each piece of data may be made into separate data. For example, information associated with coded data (image) may be transmitted on a transmission path different from that of the coded data (image). Also, for example, information associated with coded data (image) may be recorded on a recording medium different from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may be a part of the data, not the entire data. For example, an image and information corresponding to the image may be associated with each other in any unit, such as multiple frames, one frame, or a part of a frame.

[0309] In this specification, the terms "combine," "multiplex," "append," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0310] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0311] For example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0312] Also, for example, the above-mentioned program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0313] Also, for example, each step of one flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0314] Also, for example, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel, or individually at a required timing such as when a call is made. In other words, the processing of each step may be executed in an order different from the order described above, so long as no contradiction occurs. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0315] Also, for example, a plurality of techniques related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any of a plurality of the present technologies can be implemented in combination. For example, a part or all of the present technology described in any of the embodiments can be implemented in combination with a part or all of the present technology described in other embodiments. Also, a part or all of any of the above-mentioned present technologies can be implemented in combination with other technologies not described above.

[0316] The present technology can also be configured as follows. (1) a reference structure forming unit that forms a reference structure of geometry data in encoding of a point cloud, the reference structure being hierarchical based on groups into which points of the point cloud are classified, the point cloud representing a three-dimensional object as a set of points; a prediction residual deriving unit that derives a predicted value of the geometry data for each point based on the reference structure formed by the reference structure forming unit, and derives a prediction residual that is a difference between the geometry data and the predicted value; an encoding unit that encodes the prediction residual of the geometry data of each point derived by the prediction residual derivation unit; An information processing device comprising: (2) The reference structure forming portion is A grouping processing unit that groups the points; A sorting unit that sorts the points by group; a group hierarchical reference structure forming unit which forms the reference structure hierarchical by the groups by setting a reference destination of the geometry data of each point in the order sorted by the sorting unit; The information processing device according to (1) is provided. (3) The grouping processing unit performs grouping of the points according to the positions of the points. An information processing device according to (2). (4) The grouping processing unit groups the points according to characteristics of the points in the point cloud. An information processing device according to (2) or (3). (5) The reference structure forming portion is a layer information generating unit that generates, for each point, layer information indicating a group hierarchy that is a hierarchy of the groups in the reference structure; The encoding unit further encodes the layer information generated by the layer information generating unit. An information processing device according to any one of (2) to (4). (6) The layer information generating unit generates the layer information indicating the group hierarchy of each child node belonging to the processing target node in the reference structure as a relative value with respect to the group hierarchy of the processing target node. (5) An information processing device according to the present invention. (7) The layer information generating unit generates the layer information indicating the group hierarchy of the processing target node in the reference structure by a relative value with respect to the group hierarchy of a parent node to which the processing target node belongs. (5) An information processing device according to the present invention. (8) The encoding unit quantizes and encodes the prediction residual with a quantization step set for each group layer, which is a layer by the group in the reference structure. An information processing device according to any one of (1) to (7). (9) The encoding unit encodes information indicating the quantization step. (8) An information processing device. (10) The encoding unit performs arithmetic encoding on the prediction residuals by dividing the prediction residuals by group hierarchies, which are hierarchies according to the groups in the reference structure. An information processing device according to any one of (1) to (9). (11) The encoding unit arithmetically encodes the prediction residual without dividing the prediction residual into group hierarchies, which are hierarchies according to the groups in the reference structure. An information processing device according to any one of (1) to (9). (12) The encoding unit selects whether to encode the prediction residual for each group hierarchy, which is a hierarchy based on the group in the reference structure, and encodes the prediction residual of the group hierarchy selected for encoding. An information processing device according to any one of (1) to (11). (13) The encoding unit selects whether to encode the prediction residual for each branch of the reference structure, and encodes the prediction residual for the branch selected for encoding. An information processing device according to any one of (1) to (12). (14) The prediction residual derivation unit further derives a prediction value of attribute data for each point based on the reference structure formed by the reference structure formation unit, and further derives a prediction residual which is a difference between the attribute data and the prediction value; The encoding unit further encodes the prediction residual of the attribute data of each point derived by the prediction residual derivation unit. An information processing device according to any one of (1) to (13). (15) The prediction residual derivation unit determines, as the predicted value of the attribute data of the node to be processed, the attribute data of a parent node to which the node to be processed in the reference structure belongs, an average of the attribute data of the parent node and the attribute data of the parent node's parent node, a weighted average of the attribute data of the parent node and the attribute data of the parent node's parent node, or an average of the attribute data of nodes in the vicinity of the node to be processed, and derives the prediction residual of the attribute data. An information processing device according to (14). (16) The prediction residual derivation unit applies a derivation method that minimizes the prediction residual of the geometry data or the attribute data, or both, and derives the prediction residual of the geometry data and the attribute data by the derivation method. An information processing device according to (14) or (15). (17) The reference structure forming unit further forms a reference structure of attribute data in encoding the point cloud independently of the reference structure of the geometry data; the prediction residual derivation unit further derives a predicted value of the attribute data for each point based on the reference structure of the attribute data formed by the reference structure formation unit, and further derives a prediction residual which is a difference between the attribute data and the predicted value; The encoding unit further encodes the prediction residual of the attribute data of each point derived by the prediction residual derivation unit. An information processing device according to any one of (1) to (16). (18) forming a reference structure of geometry data in encoding of a point cloud, the reference structure being hierarchical according to groups into which points of the point cloud representing a three-dimensional object are classified; deriving a predicted value of the geometry data for each point based on the formed reference structure, and deriving a prediction residual which is a difference between the geometry data and the predicted value; encoding the prediction residual of the geometry data for each derived point Information processing methods.

[0317] (19) A decoding unit that decodes coded data corresponding to a desired group hierarchy among coded data in which prediction residuals, which are differences between geometry data of each point derived based on the reference structure and its predicted value, are coded based on layer information indicating a group hierarchy, which is a hierarchy of groups in a reference structure of geometry data in coding of a point cloud in which points of the point cloud are hierarchically classified into groups, the point cloud representing an object of three-dimensional shape being a set of points, are coded. An information processing device comprising: (20) In encoding a point cloud in which a point cloud expressing a three-dimensional object as a set of points is hierarchically categorized into groups into which the points of the point cloud are classified, based on layer information indicating a group hierarchy, which is a hierarchy of the groups in a reference structure of geometry data, decoding encoded data corresponding to a desired group hierarchy among encoded data in which prediction residuals, which are differences between geometry data of each point derived based on the reference structure and its predicted value, are encoded. Information processing methods.

[0318] (21) A reference structure forming unit that forms a reference structure of attribute data in encoding a point cloud that represents a three-dimensional object as a set of points; a prediction residual derivation unit that derives a prediction value of the attribute data for each point based on the reference structure formed by the reference structure formation unit, and derives a prediction residual that is a difference between the attribute data and the prediction value; an encoding unit that encodes the prediction residual of the attribute data of each point derived by the prediction residual derivation unit; An information processing device comprising: (22) The prediction residual derivation unit determines the attribute data of a parent node to which the processing target node belongs in the reference structure as the prediction value of the attribute data of the processing target node, and derives the prediction residual of the attribute data. An information processing device according to (21). (23) The prediction residual derivation unit determines an average of the attribute data of a parent node to which the processing target node belongs in the reference structure and the attribute data of a parent node of the parent node as the prediction value of the attribute data of the processing target node, and derives the prediction residual of the attribute data. The information processing device according to (21) or (22). (24) The prediction residual derivation unit determines a weighted average of the attribute data of a parent node to which the processing target node belongs in the reference structure and the attribute data of a parent node of the parent node as the prediction value of the attribute data of the processing target node, and derives the prediction residual of the attribute data. An information processing device according to any one of (21) to (23). (25) The prediction residual deriving unit determines an average of the attribute data of nodes in the reference structure adjacent to the processing target node as the prediction value of the attribute data of the processing target node, and derives the prediction residual of the attribute data. An information processing device according to any one of (21) to (24). (26) The prediction residual derivation unit further derives a predicted value of geometry data for each point based on the reference structure formed by the reference structure formation unit, and further derives a prediction residual which is a difference between the geometry data and the predicted value; The encoding unit further encodes the prediction residual of the geometry data of each point derived by the prediction residual derivation unit. An information processing device according to any one of (21) to (25). (27) The prediction residual derivation unit applies a derivation method that minimizes the prediction residual of the geometry data, and derives the prediction residual of the geometry data and the attribute data by the derivation method. (26) An information processing device according to (26). (28) The prediction residual derivation unit applies a derivation method that minimizes the prediction residual of the attribute data, and derives the prediction residual of the geometry data and the attribute data by the derivation method. (26) An information processing device according to (26). (29) The prediction residual derivation unit applies a derivation method that minimizes the prediction residual of the geometry data and the attribute data, and derives the prediction residual of the geometry data and the attribute data by the derivation method. (26) An information processing device according to (26). (30) The reference structure forming unit further forms a reference structure of geometry data in encoding the point cloud independently of the reference structure of the attribute data, the prediction residual deriving unit further derives a predicted value of the geometry data for each point based on the reference structure of the geometry data formed by the reference structure forming unit, and further derives a prediction residual which is a difference between the geometry data and the predicted value; The encoding unit further encodes the prediction residual of the geometry data of each point derived by the prediction residual derivation unit. An information processing device according to any one of (21) to (29). (31) The reference structure forming unit forms the reference structure of the attribute data so as to hierarchically classify the points of the point cloud into groups. An information processing device according to any one of (21) to (30). (32) The reference structure forming portion is A grouping processing unit that groups the points; A sorting unit that sorts the points by group; a group hierarchical reference structure forming unit that forms the reference structure hierarchical by the group by setting a reference destination of the attribute data of each point in the order sorted by the sorting unit; The information processing device according to (31) , (33) The grouping processing unit groups the points according to positions of the points, or characteristics of the points in the point cloud, or both. An information processing device according to (32). (34) The reference structure forming portion is a layer information generating unit that generates, for each point, layer information indicating a group hierarchy that is a hierarchy of the groups in the reference structure; The encoding unit further encodes the layer information generated by the layer information generating unit. The information processing device according to (32) or (33). (35) The layer information generating unit generates the layer information indicating the group hierarchy of each child node belonging to the processing target node in the reference structure by a relative value with respect to the group hierarchy of the processing target node. (34) An information processing device according to the above. (36) The layer information generating unit generates the layer information indicating the group hierarchy of the processing target node in the reference structure as a relative value with respect to the group hierarchy of a parent node to which the processing target node belongs. (34) An information processing device according to the above. (37) The encoding unit selects whether to encode the prediction residual for each group hierarchical layer, which is a hierarchical layer based on the group in the reference structure, or for each branch of the reference structure, or for both, and encodes the prediction residual for the group hierarchical layer or the branch selected for encoding. An information processing device according to any one of (31) to (36). (38) Forming a reference structure for attribute data in encoding a point cloud that represents a three-dimensional object as a set of points; deriving a predicted value of the attribute data for each point based on the formed reference structure, and deriving a prediction residual which is a difference between the attribute data and the predicted value; encoding the prediction residual of the attribute data for each derived point; Information processing methods.

[0319] (39) A decoding unit that decodes encoded data in which a prediction residual, which is a difference between attribute data of each point derived based on a reference structure of attribute data in encoding of a point cloud that represents an object of a three-dimensional shape as a set of points, and a prediction residual of the attribute data is encoded, and generates the prediction residual of the attribute data; a generating unit that generates the attribute data by using the prediction residual of the attribute data generated by the decoding unit and a predicted value of the attribute data; An information processing device comprising: (40) Decoding encoded data in which a prediction residual, which is a difference between attribute data of each point derived based on a reference structure of attribute data in encoding a point cloud that represents a three-dimensional object as a set of points, and a prediction residual of the attribute data is encoded, and generating the prediction residual of the attribute data; generating the attribute data using the prediction residual of the attribute data generated by decoding the encoded data and a predicted value of the attribute data; Information processing methods. [Explanation of symbols]

[0320] 100 encoding device, 111 geometry data encoding unit, 112 attribute data encoding unit, 131 reference structure forming unit, 132 stack, 133 prediction mode determination unit, 134 encoding unit, 135 prediction point generating unit, 151 grouping processing unit, 152 sorting unit, 153 group hierarchical reference structure forming unit, 154 layer information generating unit, 200 decoding device, 211 geometry data decoding unit, 212 attribute data decoding unit, 231 storage unit, 232 stack, 233 decoding unit, 234 geometry data generating unit, 235 prediction point generating unit, 300 transcoder, 311 geometry data decoding unit, 312 geometry data encoding unit, 313 attribute data transcode processing unit, 400 encoding device, 411 reference structure forming unit, 412 stack, 413 prediction mode determination unit, 414 encoding unit, 415 prediction point generation unit, 500 decoding device, 511 storage unit, 512 stack, 513 decoding unit, 514 point data generation unit, 515 prediction point generation unit, 600 transcoder, 611 decoding unit, 612 encoding unit, 900 computer

Claims

1. a reference structure forming unit that forms a reference structure of geometry data in encoding of a point cloud, the reference structure being hierarchical based on groups into which points of the point cloud are classified, the point cloud representing a three-dimensional object as a set of points; a prediction residual deriving unit that derives a predicted value of the geometry data for each point based on the reference structure formed by the reference structure forming unit, and derives a prediction residual that is a difference between the geometry data and the predicted value; an encoding unit that encodes the prediction residual of the geometry data of each point derived by the prediction residual derivation unit; An information processing device comprising:

2. The reference structure forming unit includes: A grouping processing unit that groups the points; A sorting unit that sorts the points by group; a group hierarchical reference structure forming unit that forms the reference structure hierarchically organized by the groups by setting a reference destination of the geometry data of each point in the order sorted by the sorting unit; The information processing device according to claim 1 .

3. The grouping processing unit performs grouping of the points according to the positions of the points. The information processing device according to claim 2 .

4. The grouping processing unit groups the points according to characteristics of the points in the point cloud. The information processing device according to claim 2 .

5. The reference structure forming unit includes: a layer information generating unit that generates, for each point, layer information indicating a group hierarchy that is a hierarchy of the groups in the reference structure; The encoding unit further encodes the layer information generated by the layer information generating unit. The information processing device according to claim 2 .

6. The layer information generating unit generates the layer information indicating the group hierarchy of each child node belonging to the processing target node in the reference structure as a relative value with respect to the group hierarchy of the processing target node. The information processing device according to claim 5 .

7. The layer information generating unit generates the layer information indicating the group hierarchy of the processing target node in the reference structure by a relative value with respect to the group hierarchy of the parent node to which the processing target node belongs. The information processing device according to claim 5 .

8. The encoding unit quantizes and encodes the prediction residual with a quantization step set for each group layer, which is a layer based on the group in the reference structure. The information processing device according to claim 1 .

9. The encoding unit encodes information indicating the quantization step. The information processing device according to claim 8.

10. The encoding unit performs arithmetic encoding on the prediction residuals by dividing the prediction residuals into group hierarchies, which are hierarchies according to the groups in the reference structure. The information processing device according to claim 1 .

11. The encoding unit arithmetically encodes the prediction residual without dividing the prediction residual into group hierarchies that are hierarchies according to the groups in the reference structure. The information processing device according to claim 1 .

12. The encoding unit selects whether to encode the prediction residual for each group hierarchical layer, which is a hierarchical layer based on the group in the reference structure, and encodes the prediction residual for the group hierarchical layer selected for encoding. The information processing device according to claim 1 .

13. The encoding unit selects whether to encode the prediction residual for each branch of the reference structure, and encodes the prediction residual for the branch for which encoding has been selected. The information processing device according to claim 1 .

14. the prediction residual derivation unit further derives a predicted value of attribute data for each point based on the reference structure formed by the reference structure formation unit, and further derives a prediction residual which is a difference between the attribute data and the predicted value; The encoding unit further encodes the prediction residual of the attribute data of each point derived by the prediction residual derivation unit. The information processing device according to claim 1 .

15. The prediction residual derivation unit determines, as the predicted value of the attribute data of the node to be processed, the attribute data of a parent node to which the node to be processed in the reference structure belongs, an average of the attribute data of the parent node and the attribute data of the parent node's parent node, a weighted average of the attribute data of the parent node and the attribute data of the parent node's parent node, or an average of the attribute data of nodes in the vicinity of the node to be processed, and derives the prediction residual of the attribute data. The information processing device according to claim 14.

16. The prediction residual derivation unit applies a derivation method that minimizes the prediction residual of the geometry data or the attribute data, or both, and derives the prediction residual of the geometry data and the attribute data by the derivation method. The information processing device according to claim 14.

17. The reference structure forming unit further forms a reference structure of attribute data in encoding the point cloud independently of the reference structure of the geometry data; the prediction residual derivation unit further derives a predicted value of the attribute data for each point based on the reference structure of the attribute data formed by the reference structure formation unit, and further derives a prediction residual which is a difference between the attribute data and the predicted value; The encoding unit further encodes the prediction residual of the attribute data of each point derived by the prediction residual derivation unit. The information processing device according to claim 1 .

18. forming a reference structure of geometry data in encoding of a point cloud, the reference structure being hierarchical according to groups into which points of the point cloud are classified, the point cloud representing a three-dimensional object as a set of points; deriving a predicted value of the geometry data for each point based on the formed reference structure, and deriving a prediction residual which is a difference between the geometry data and the predicted value; encoding the prediction residual of the geometry data for each derived point Information processing methods.

19. a decoding unit that decodes coded data corresponding to a desired group hierarchy among coded data in which prediction residuals, which are differences between geometry data of each point derived based on the reference structure and its predicted value, are coded based on layer information indicating a group hierarchy, which is a hierarchy according to the groups in a reference structure of geometry data in coding of a point cloud in which points of the point cloud are hierarchically classified according to groups into which the points of the point cloud are expressed as a set of points, An information processing device comprising:

20. A point cloud expressing a three-dimensional object as a collection of points is coded based on layer information indicating a group hierarchy, which is a hierarchy of groups in a reference structure of geometry data in coding of the point cloud, into which the points of the point cloud are hierarchically classified, and a prediction residual, which is a difference between the geometry data of each point derived based on the reference structure and its predicted value, is coded, and the coded data corresponding to a desired group hierarchy is decoded. Information processing methods.

Images