Information processing device and method

Abstract

The present disclosure relates to an information processing device and method which make it possible to suppress a reduction in coding efficiency. The method includes: using a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature quantity set, which is a set of feature quantities of the coordinates, and a guide surface, which is a reference surface formed in a three-dimensional space, to derive a guide surface feature quantity, which is a feature quantity based on a positional relationship between the guide surface and the points; performing a predetermined calculation using the first coordinate set and the first feature quantity set reflecting the guide surface feature quantity to derive a second coordinate set and a second feature quantity set; encoding the derived second coordinate set; and encoding the derived second feature quantity set. The present disclosure is applicable, for example, to information processing devices, electronic instruments, information processing methods and programs.

Description

Information processing device and method

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that can suppress a decrease in coding efficiency.

[0002] Recently, AI (Artificial Intelligence)-based point cloud compression methods (hereinafter also referred to as AI-PCC (Artificial Intelligence - Point Cloud Compression)) have been developed, which are point cloud compression methods that have neural networks in the encoder or decoder. For example, the method described in Non-Patent Document 1 is known as one such AI-PCC method. This method uses a sparse 3D-CNN (3-dimensional convolutional neural network) (a type of neural network) in the encoder and decoder.

[0003] Jianqiang Wang, Dandan Ding, Zhu Li, and Zhan Ma, "Multiscale Point Cloud Geometry Compression", 2021 Data Compression Conference (DCC), 2021 Data Compression Conference (DCC), https: / / ieeexplore.ieee.org / document / 9418789 / figures#figures

[0004] However, the method described in Non-Patent Document 1 is not always the best method, and other methods have been sought. In other words, conventional methods may result in reduced coding efficiency.

[0005] The present disclosure has been made in light of such circumstances, and makes it possible to suppress a decrease in coding efficiency.

[0006] An information processing device according to one aspect of the present technology includes a guide surface feature derivation unit that derives guide surface features, which are features based on the positional relationship between the guide surface and the points, using a first coordinate set, which is a set of coordinates of points that form a point cloud, a first feature set, which is a set of features of the coordinates, and a guide surface, which is a reference surface formed in three-dimensional space; a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature set that reflects the guide surface features, to derive a second coordinate set and a second feature set; a coordinate encoding unit that encodes the derived second coordinate set; and a feature encoding unit that encodes the derived second feature set, and the information processing device has a code size that is smaller than the code size of the encoded data of the first coordinate set and the first feature set.

[0007] An information processing method according to one aspect of the present technology is an information processing method that uses a first coordinate set, which is a set of coordinates of points that form a point cloud, a first feature set, which is a set of features of the coordinates, and a guide surface, which is a reference surface formed in three-dimensional space, to derive guide surface features, which are features based on the positional relationship between the guide surface and the points, performs a predetermined calculation using the first coordinate set and the first feature set that reflects the guide surface features, derives a second coordinate set and a second feature set, encodes the derived second coordinate set, encodes the derived second feature set, and the code size of the encoded data of the second coordinate set and the second feature set is smaller than the code size of the encoded data of the first coordinate set and the first feature set.

[0008] An information processing device according to another aspect of the present technology is an information processing device including: a coordinate decoding unit that decodes encoded data to generate a first coordinate set, which is a set of coordinates of points that form a point cloud; a feature decoding unit that decodes encoded data to generate a first feature set, which is a set of features of the coordinates; a guide surface feature derivation unit that uses the generated first coordinate set, the generated first feature set, and a guide surface that is a reference surface formed in three-dimensional space to derive guide surface features, which are features based on the positional relationship between the guide surface and the points; and a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature set that reflects the guide surface features, to derive a second coordinate set and a second feature set.

[0009] Another aspect of the information processing method of the present technology is an information processing method that decodes encoded data to generate a first coordinate set which is a set of coordinates of points that form a point cloud, decodes the encoded data to generate a first feature set which is a set of features of the coordinates, derives guide surface features which are features based on the positional relationship between the guide surface and the points using the generated first coordinate set, the generated first feature set, and a guide surface which is a reference surface formed in three-dimensional space, and performs an operation using the first coordinate set and the first feature set that reflects the guide surface features, to derive a second coordinate set and a second feature set.

[0010] In an information processing device and method according to one aspect of the present technology, a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature set, which is a set of features of those coordinates, and a guide surface, which is a reference surface formed in three-dimensional space, are used to derive guide surface features, which are features based on the positional relationship between the guide surface and the points, and an operation is performed using the first coordinate set and the first feature set that reflects the guide surface features, to derive a second coordinate set and a second feature set, the derived second coordinate set is encoded, and the derived second feature set is encoded.

[0011] In an information processing device and method according to another aspect of the present technology, encoded data is decoded to generate a first coordinate set, which is a set of coordinates of points forming a point cloud; the encoded data is decoded to generate a first feature set, which is a set of features of the coordinates; the generated first coordinate set, the generated first feature set, and a guide surface, which is a reference surface formed in three-dimensional space, are used to derive guide surface features, which are features based on the positional relationship between the guide surface and the points; a predetermined calculation is performed using the first coordinate set and the first feature set that reflects the guide surface features, and a second coordinate set and a second feature set are derived.

[0012] 12 is a diagram showing an example of the main configuration of a conventional calculation unit. FIG. 13 is a diagram showing an example of the main configuration of a conventional encoding device. FIG. 14 is a diagram showing an example of the main configuration of a conventional encoding device. FIG. 15 is a diagram showing an example of an encoding / decoding method to which the present technology is applied. FIG. 16 is a diagram showing an example of the main configuration of a calculation unit. FIG. 17 is a diagram showing an example of a guide surface. FIG. 18 is a block diagram showing an example of the main configuration of an encoding device. FIG. 19 is a flowchart illustrating an example of the flow of an encoding process. FIG. 19 is a flowchart illustrating an example of the flow of a guide surface feature set derivation process. FIG. 19 is a block diagram showing an example of the main configuration of a decoding device. FIG. 20 is a flowchart illustrating an example of the flow of a decoding process. FIG. 21 is a flowchart illustrating an example of an example of a flow of a decoding process. FIG. 22 is a flowchart illustrating an example of an example of a flow of a decoding process. FIG. 23 is a flowchart illustrating an example of an example of a flow of a decoding process. 29 is a flowchart illustrating an example of the flow of a guide surface feature amount set derivation process. FIG. 30 is a block diagram illustrating an example of the main configuration of a decoding device. FIG. 31 is a flowchart illustrating an example of the flow of a decoding process. FIG. 32 is a flowchart continuing from FIG. 29, illustrating an example of the flow of a decoding process. FIG. 33 is a diagram illustrating an example of a simulation result. FIG. 34 is a block diagram illustrating an example of the main configuration of a computer.

[0013] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature supporting technical content and technical terms 2. Encoding and decoding of point clouds 3. Encoding and decoding of point clouds using guide surfaces 4. Supplementary notes

[0014] <1. Literature, etc. supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the content described in the embodiments, but also the content described in the following non-patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following non-patent documents.

[0015] Non-patent document 1: (mentioned above) Non-patent document 2: Michael Kazhdan, Matthew Bolitho, Hugues Hoppe, "Poisson Surface Reconstruction", Eurographics Symposium on Geometry Processing (2006), Konrad Polthier, Alla Sheffer (Editors) Non-patent document 3: Stoll Carsten, Karni Zachi, Rossl Christian, Yamauchi Hitoshi, Seidel Hans-Peter, "Template "Deformation for Point Cloud Fitting", Non-Patent Document 4: Danhang Tang, Philip A. Chou, Christian Hane, Mingsong Dou, Sean Fanello, Jonathan Taylor, Philip Davidson, Onur G. Guleryuz, Yinda Zhang, Shahram Izadi, Andrea Tagliasacchi, Sofien Bouaziz, Cem Keskin, "Deep Implicit Volume Compression", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1293-1303

[0016] In other words, the contents of the above-mentioned non-patent documents and the contents of other documents referenced in the above-mentioned non-patent documents are also used as the basis for determining the support requirements.

[0017] <2. Encoding and Decoding of Point Clouds> <Point Clouds> Conventionally, point clouds (also called point groups) have been used as 3D data representing the three-dimensional structure of a three-dimensional structure (a three-dimensional object), representing the object as a collection of numerous points. Point cloud data (also called point cloud data) consists of the geometry (position information) and attributes (attribute information) of each point that makes up the point cloud. The geometry indicates the position of the point in three-dimensional space. The attributes indicate the attributes of the point. These attributes can include any information. For example, the attributes may include color information, reflectance information, normal information, etc. for each point. As such, point clouds have a relatively simple data structure, and by using a sufficient number of points, they can represent any three-dimensional structure with sufficient accuracy.

[0018] <AI-PCC> Recently, AI-based point cloud compression methods (hereinafter also referred to as AI-PCC (Artificial Intelligence - Point Cloud Compression)) have been developed, which are point cloud compression methods that have a neural network in the encoder or decoder. For example, the method described in Non-Patent Document 1 is known as one such AI-PCC method. In this specification, this method is also referred to as the "conventional AI-PCC method." This conventional AI-PCC method uses a sparse 3D-CNN (3-dimensional convolutional neural network) (a type of neural network) in the encoder and decoder. For example, as shown in FIG. 1, this conventional AI-PCC method uses a CNN (Convolution Neural Network) 11 that converts an input sparse tensor into an output sparse tensor. Other common AI-PCC methods use some kind of calculation unit 21, such as a 3D-CNN, a multilayer perceptron, or another neural network, that converts a set of input coordinates and input features into a set of output coordinates and output features.

[0019] Figure 2 shows the main components of an encoder using this conventional AI-PCC method. As shown in Figure 2, in this encoder 50, a sparse tensor construction unit 51 constructs an input sparse tensor X = {C_X, F_X} from an input point cloud C_X. The input feature set F_X uses dummy features consisting of a series of 1s. CNNs 52 (CNN_E1), 53 (CNN_E2), 54 (CNN_E3), 55 (CNN_E4), 56 (CNN_E5), and 57 (CNN_E6) receive the sparse tensor and return the transformed sparse tensor. The point encoding unit 58 encodes the points of C_X, the points of the coordinate set obtained by downscaling C_X by 1 / 2, and the points of the coordinate set obtained by downscaling C_X by 1 / 4, respectively, and outputs a point bitstream. The coordinate encoding unit 59 encodes (octree-encodes) the coordinate set C_Y obtained by downscaling the input point cloud C_X by 1 / 8, and outputs a coordinate bitstream. The feature encoding unit 60 encodes the feature set F_Y associated with each coordinate in the coordinate set C_Y, and outputs a feature bit stream.

[0020] Figure 3 shows an example of the main configuration of a decoder using this conventional AI-PCC technique. In this decoder 70, the point decoding unit 71 decodes the point bitstream. The coordinate decoding unit 72 is a G-PCC decoder that losslessly decodes the coordinate set bitstream to generate (restore) the coordinate set C_Y. The feature decoding unit 73 entropy decodes the feature bitstream to generate (restore) the feature set F_Y. However, the feature bitstream is lossy compressed, and strictly speaking, the restored feature set F_Y may differ from the F_Y before encoding. CNN74 (CNN_D1), CNN75 (CNN_D2), CNN77 (CNN_D3), CNN78 (CNN_D4), CNN80 (CNN_D5), and CNN81 (CNN_D6) are sparse 3D-CNNs that receive sparse tensors and return transformed sparse tensors. These are primarily composed of sparse 3D convolutional layers and nonlinear transformation layers. The occupancy state classification units 76, 79, and 82 each consist of a single sparse 3D convolutional layer. This sparse 3D convolutional layer inputs a sparse tensor and outputs a sparse tensor as the result of the calculation. The features of the sparse tensor resulting from the calculation of this sparse 3D convolutional layer represent the occupancy probability values ​​of the coordinates. The occupancy state classification units 76, 79, and 82 retain only the top k coordinates with the highest occupancy probability values ​​and delete the remaining coordinates. In other words, the occupancy state classification units 76, 79, and 82 output a coordinate set from the input sparse tensor, with all but the top k coordinates deleted, and a feature set corresponding to that coordinate set (i.e., the feature set of the input sparse tensor from which the features corresponding to the deleted coordinates have been deleted). k is the corresponding scale among N_X, N_X', and N_X''. The features corresponding to the deleted coordinates in F^out are also deleted.

[0021] However, the method described in Non-Patent Document 1 is not always the best method, and other methods have been sought. In other words, conventional methods may result in reduced coding efficiency.

[0022] <3. Encoding and decoding of point cloud using guide surface> <Method 1> Therefore, as shown in the top row of the table in Fig. 4, a coordinate set and a feature set derived using guide surface features are transmitted (Method 1). That is, on the transmitting side, the coordinate set and feature set derived using the guide surface features are encoded and a bit stream is transmitted, and on the receiving side, the bit stream is received and decoded.

[0023] For example, in the case of a computing unit 100 using a CNN like the conventional AI-PCC method described above, as shown in Fig. 5, a guide surface feature derivation unit 101 reflects a set of guide surface features g(C^in) in an input sparse tensor {C^in, F^in}, and inputs the input sparse tensor {C^in, [F^in, g(C^in)]} to a CNN 102. Similarly, in the case of a computing unit 120 other than the conventional AI-PCC method, a guide surface feature derivation unit 121 reflects a set of guide surface features g(C^in) in an input coordinate set and an input feature set {C^in, F^in}, and inputs the input coordinate set and input feature set {C^in, [F^in, g(C^in)]} to a computing unit 122. This computing unit 122 may have any configuration, such as a 3D-CNN, a multilayer perceptron, or another neural network.

[0024] For example, when points 141 exist in a three-dimensional space 140 as shown in FIG. 6 , in conventional methods, information about each point 141 is encoded individually. In contrast, in Method 1, a guide surface 142 is provided. This guide surface 142 is a virtual surface in the three-dimensional space set in the present technology, which is used to convert information about each point 141. This guide surface 142 may be any surface, such as a flat surface or a curved surface. Furthermore, this guide surface 142 may represent the three-dimensional shape (outer shape) of the same 3D object as the point cloud with a predetermined degree of resolution, or may be a surface unrelated to the outer shape of the 3D object. Then, in Method 1, information about each point 141 is converted into information (guide surface feature) based on this guide surface 142 and encoded. Therefore, in 2D encoding, the amount of information to be encoded can be reduced, as in the case of encoding prediction residuals. Therefore, a decrease in encoding efficiency can be suppressed.

[0025] For example, a first information processing device is provided with a guide surface feature derivation unit that uses a first coordinate set, which is a set of coordinates of points that form a point cloud, a first feature set, which is a set of features of those coordinates, and a guide surface, which is a reference surface formed in three-dimensional space, to derive guide surface features, which are features based on the positional relationship between the guide surface and the points; a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature set that reflects the guide surface features, to derive a second coordinate set and a second feature set; a coordinate encoding unit that encodes the derived second coordinate set; and a feature encoding unit that encodes the derived second feature set.

[0026] In addition, in a first information processing method executed by a first information processing device, a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature set, which is a set of features of those coordinates, and a guide surface, which is a reference surface formed in three-dimensional space, are used to derive guide surface features, which are features based on the positional relationship between the guide surface and the points, and a predetermined calculation is performed using the first coordinate set and the first feature set that reflects the guide surface features to derive a second coordinate set and a second feature set, and the derived second coordinate set is encoded and the derived second feature set is encoded.

[0027] By doing so, as described above, it is possible to reduce the amount of information to be coded, and to prevent a decrease in coding efficiency.

[0028] The scale of the second coordinate set and the second feature set may be smaller than the scale of the first coordinate set and the first feature set. Furthermore, in the first information processing device, the calculation unit may derive the second coordinate set and the second feature set by recursively repeating a predetermined calculation multiple times. Furthermore, the calculation unit may downscale the input coordinate set and feature set in all or part of the predetermined calculation multiple times. Furthermore, the guide surface feature may be derived for each scale. For example, the guide surface feature derivation unit may derive guide surface feature for each of some or all of the scales of the coordinate set and the feature set.

[0029] The guide surface may also be encoded. The bit stream may also be decoded to generate (restore) the guide surface. The guide surface may also be generated. For example, the first information processing device may further include a guide surface generation unit that generates a guide surface using a point cloud, a guide surface encoding unit that encodes the guide surface and generates encoded data of the guide surface, and a guide surface decoding unit that decodes the encoded data of the guide surface and generates the guide surface.

[0030] The first information processing device may further include a score encoding unit that encodes the score. The score encoding unit may encode the score for each scale.

[0031] In addition, in the first information processing device, the calculation unit may be configured by a convolution neural network (CNN). In this case, the first information processing device may further include a sparse tensor construction unit that constructs a sparse tensor from the point cloud. In this case, the CNN may input the constructed sparse tensor.

[0032] The second information processing device also includes a coordinate decoding unit that decodes encoded data to generate a first coordinate set, which is a set of coordinates of points that form the point cloud; a feature decoding unit that decodes encoded data to generate a first feature set, which is a set of features of the coordinates; a guide surface feature derivation unit that derives guide surface features, which are features based on the positional relationship between the guide surface and the points, using the generated first coordinate set, the generated first feature set, and a guide surface, which is a reference surface formed in three-dimensional space; and a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature set that reflects the guide surface features, to derive a second coordinate set and a second feature set, wherein the code size of the encoded data of the first coordinate set and the first feature set is smaller than the code size of the encoded data of the second coordinate set and the second feature set.

[0033] In addition, in a second information processing method executed by a second information processing device, encoded data is decoded to generate a first coordinate set, which is a set of coordinates of points that form a point cloud, and encoded data is decoded to generate a first feature set, which is a set of coordinate features. The generated first coordinate set, the generated first feature set, and a guide surface, which is a reference surface formed in three-dimensional space, are used to derive guide surface features, which are features based on the positional relationship between the guide surface and the points. A predetermined calculation is performed using the first coordinate set and the first feature set that reflects the guide surface features, to derive a new coordinate set and feature set, and the amount of code for the encoded data of the first coordinate set and first feature set is made smaller than the amount of code for the encoded data of the second coordinate set and second feature set.

[0034] By doing so, as described above, it is possible to reduce the amount of information to be coded, and to prevent a decrease in coding efficiency.

[0035] The scale of the second coordinate set and the second feature set may be larger than the scale of the first coordinate set and the first feature set. Furthermore, in the second information processing device, the calculation unit may derive the second coordinate set and the second feature set by recursively repeating a predetermined calculation multiple times. Furthermore, the calculation unit may upscale the input coordinate set and feature set in all or part of the multiple predetermined calculations. Furthermore, the guide surface feature may be derived for each scale. For example, the guide surface feature derivation unit may derive guide surface feature for each of some or all of the scales of the coordinate set and the feature set.

[0036] Furthermore, the second information processing device may generate (restore) the guide surface by decoding. For example, the second information processing device may further include a guide surface decoding unit that decodes the coded data of the guide surface and generates the guide surface.

[0037] The second information processing device may also include an occupancy state classification unit that predicts an occupancy probability value for each coordinate and extracts some coordinates and feature quantities with high occupancy probability values. The second information processing device may also include a score decoding unit that decodes to generate (restore) a score. The score decoding unit may also decode to generate (restore) a score for each scale.

[0038] In the second information processing device, the calculation unit may be configured by a convolution neural network (CNN).

[0039] <Encoding Device> The first information processing device may be any device. For example, it may be an encoding device that encodes 3D data (e.g., a point cloud). FIG. 7 is a block diagram showing an example of the configuration of an encoding device, which is one aspect of an information processing device to which the present technology is applied. The encoding device 300 shown in FIG. 7 is a device that encodes the geometry of a point cloud (3D data). The encoding device 300 encodes the geometry by applying the above-described method 1.

[0040] Note that Fig. 7 shows the main processing units, data flows, etc., and is not necessarily all that is shown in Fig. 7. In other words, in the encoding device 300, there may be processing units that are not shown as blocks in Fig. 7, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 7.

[0041] As shown in Figure 7, the encoding device 300 has a guide surface generation unit 311, a guide surface encoding unit 312, a guide surface decoding unit 313, a sparse tensor construction unit 314, a guide surface feature derivation unit 315, a CNN 316 (CNN_E1), a guide surface feature derivation unit 317, a CNN 318 (CNN_E2), a CNN 319 (CNN_E3), a guide surface feature derivation unit 320, a CNN 321 (CNN_E4), a CNN 322 (CNN_E5), a CNN 323 (CNN_E6), a coordinate encoding unit 324, a feature encoding unit 325, and a score encoding unit 326.

[0042] The guide surface generation unit 311 generates a guide surface from the input point cloud C_X. The guide surface is a reference surface formed in three-dimensional space and is used for encoding the point cloud. For example, the guide surface generation unit 311 may generate a mesh from the point cloud and use the mesh as the guide surface. Any method may be used to generate a mesh from this point cloud, and an existing method such as Poisson surface reconstruction described in Non-Patent Document 2 may be applied. However, this method requires normal data, so it is necessary to calculate the normals of the point cloud. The guide surface generation unit 311 supplies data indicating the generated guide surface to the guide surface encoding unit 312.

[0043] The guide surface encoding unit 312 encodes the supplied guide surface to generate encoded data of the guide surface (also referred to as a guide surface bitstream). Any encoding method may be used. For example, if the guide surface is configured by a mesh, the guide surface encoding unit 312 may encode the guide surface by applying an encoding method for meshes. For example, an existing method such as Draco encoding after reducing the number of triangles by mesh decimation may be applied. The guide surface encoding unit 312 may output the generated guide surface bitstream to the outside of the encoding device 300. Furthermore, the guide surface encoding unit 312 may supply the generated guide surface bitstream to the guide surface decoding unit 313.

[0044] The guide surface decoding unit 313 decodes the supplied guide surface bitstream and generates (restores) a guide surface. This decoding method may be any method that corresponds to the encoding method applied by the guide surface encoding unit 312. For example, if the guide surface is configured as a mesh, the guide surface decoding unit 313 may decode the guide surface bitstream using a decoding method for meshes. This decoding method may be an existing method. For example, if the guide surface is a mesh, Draco decoding may be applied. Note that the encoding and decoding of this guide surface may be a lossless method or a lossy method. In other words, the guide surface generated (restored) by the guide surface decoding unit 313 (also referred to as a decoded guide surface) may not completely match the guide surface before encoding (e.g., the guide surface generated by the guide surface generation unit 311). The guide surface decoding unit 313 may supply the generated decoded guide surface to the guide surface feature derivation unit 315, the guide surface feature derivation unit 317, and the guide surface feature derivation unit 320.

[0045] The sparse tensor construction unit 314 constructs an input sparse tensor X = {C_X, F_X} from the input point group C_X. A sparse tensor is a data structure for representing a set of feature vectors arranged in space. Each feature vector is associated with a single coordinate in space. Specifically, a sparse tensor in three-dimensional space is represented by a set {C, F} of coordinate set C = {(x_i, y_i, z_i)}_i and feature set F = {f_i}_i. Here, feature f_i is arranged at coordinates (x_i, y_i, z_i). Note that a dummy feature consisting of a series of "1"s is applied to the input feature set F_X. The scale of the input sparse tensor X = {C_X, F_X} constructed by the sparse tensor construction unit 314 is 1 (x1). The sparse tensor constructing unit 314 may supply the constructed input sparse tensor X = {C_X, F_X} to the guide surface feature derivation unit 315 .

[0046] The guide surface feature amount derivation unit 315 performs processing related to the derivation of guide surface feature amounts. For example, the guide surface feature amount derivation unit 315 receives a sparse tensor {C^in, F^in} to be processed, and derives a guide surface feature amount g(c_i) for each coordinate c_i ∈ C^in of the coordinate set C^in to be processed. Then, the guide surface feature amount derivation unit 315 concatenates the obtained guide surface feature amount set g(C^in) = {g(c_i)}_i to the feature amount set F^in to be processed, and returns the sparse tensor {C^in, [F^in, g(C^in)]} of the calculation result. Here, [F^in, g(C^in)] is a feature amount set obtained by concatenating the guide surface feature amount g(c_i) calculated from the coordinate value corresponding to each feature amount f_i ∈ F^in. It is defined as [F^in, g(C^in)] = {[f_i, g(c_i)]}_i. As a method of calculating the guide surface feature amount g(c_i), for example, there is a method of searching for the nearest point p_i on the guide surface as viewed from the coordinate value c_i, and setting g(c_i) = p_i - c_i ∈ R^3. In other words, in this case, the dimension of the feature amount set [F^in, g(C^in)] of the calculation result increases by 3 compared to the dimension of the feature amount set F^in to be processed.

[0047] The guide surface feature amount derivation unit 315 may acquire the input sparse tensor X = {C_X, F_X} supplied from the sparse tensor construction unit 314 as a sparse tensor to be processed. Also, the guide surface feature amount derivation unit 315 may acquire the decoded guide surface supplied from the guide surface decoding unit 313. Then, the guide surface feature amount derivation unit 315 may use the decoded guide surface to perform the above-mentioned processing on the input sparse tensor to generate a sparse tensor as a calculation result. Also, the guide surface feature amount derivation unit 315 may supply the generated sparse tensor as a calculation result to the CNN 316 (CNN_E1).

[0048] CNN316 (CNN_E1) ​​is a sparse 3D-CNN. A sparse 3D-CNN is a 3D-CNN with a sparse 3D convolutional layer. It inputs the sparse tensor {C^in, F^in} to be processed and outputs the resulting sparse tensor {C^out, F^out}. A 3D-CNN is a neural network primarily composed of 3D convolutional layers and nonlinear transformation layers. The 3D convolutional layer performs a kernel convolution operation on a set of features arranged in three-dimensional space and outputs the resulting set of features. Roughly speaking, the 3D convolutional layer performs an operation to aggregate neighboring features in three-dimensional space by weighting each feature with the corresponding kernel value. This achieves feature transformation that takes neighboring features into account. By stacking 3D convolutional layers and nonlinear transformation layers, 3D-CNN achieves more nonlinear and complex feature transformation. A sparse 3D convolutional layer is a 3D convolutional layer whose input and output are sparse tensors. A sparse 3D convolutional layer has the advantage of being computationally more efficient (because it ignores empty coordinates that have no corresponding features) than a regular 3D convolutional layer (also called a dense 3D convolutional layer for comparison). For example, the computation of a sparse 3D convolutional layer can be expressed as the following equation (1):

[0049] ...(1)

[0050] Note that C^in is the input coordinate set (also referred to as the coordinate set of the target object), and C^out is the output coordinate set (also referred to as the coordinate set of the calculation result). f_u^in is the input feature vector at coordinate u (also referred to as the feature vector of the target object), and f_u^out is the output feature vector at coordinate u (also referred to as the feature vector of the calculation result). N^3(u, C^in) = {i|u + i ∈ C^in, i ∈ N^3} defines the 3D convolution kernel and is the set of offset values ​​i to neighboring coordinates in C^in centered at coordinate u. W_i represents the kernel value at offset value i. Also, the coordinate set of the target object C^in and the coordinate set of the calculation result C^out are not always equal. For example, when downscaling or upscaling, C^out will differ from C^in.

[0051] The CNN 316 (CNN_E1) ​​may acquire the sparse tensor resulting from the calculation performed by the guide surface feature derivation unit 315 as the sparse tensor to be processed. In other words, the dimension of the feature increases by the amount of the guide surface feature, and the input / output dimensions of each sparse 3D-CNN are changed accordingly. This also applies to the following CNNs (CNN 318, CNN 319, CNN 321, CNN 322, and CNN 323). The CNN 316 (CNN_E1) ​​is configured with two sparse 3D convolutional layers and two rectified linear unit (ReLU) layers, which are a type of nonlinear transformation layer, and may perform 1 / 2 downscaling on the sparse tensor to be processed. The CNN 316 (CNN_E1) ​​may supply the sparse tensor resulting from the calculation obtained to the guide surface feature derivation unit 317. The CNN 316 (CNN_E1) ​​may supply N_X = |C_X|, which corresponds to the number of points in the coordinate set C_X, to the point encoding unit 326.

[0052] The guide surface feature amount derivation unit 317 is a processing unit similar to the guide surface feature amount derivation unit 315, and performs processing related to derivation of guide surface feature amounts. That is, as described above, the guide surface feature amount derivation unit 317 acquires the sparse tensor {C^in, F^in} to be processed, derives the guide surface feature amount g(c_i) for each coordinate c_i ∈ C^in of the coordinate set C^in to be processed, connects the obtained guide surface feature amount set g(C^in) = {g(c_i)}_i to the feature amount set F^in to be processed, and returns the sparse tensor {C^in, [F^in, g(C^in)]} of the calculation result.

[0053] The guide surface feature amount derivation unit 317 may acquire a sparse tensor resulting from the calculation by the CNN 316 (CNN_E1) ​​as a sparse tensor to be processed. The guide surface feature amount derivation unit 317 may also acquire a decoded guide surface supplied from the guide surface decoding unit 313. Then, the guide surface feature amount derivation unit 317 may use the decoded guide surface to perform the above-described processing on the acquired sparse tensor to generate a sparse tensor resulting from the calculation. The guide surface feature amount derivation unit 317 may supply the generated sparse tensor resulting from the calculation to the CNN 318 (CNN_E2).

[0054] Like CNN 316 (CNN_E1), CNN 318 (CNN_E2) is a sparse 3D-CNN. CNN 318 (CNN_E2) may acquire the sparse tensor resulting from the calculation by the guide surface feature derivation unit 317 as the sparse tensor to be processed. CNN 318 (CNN_E2) may be configured with a three-layer Inception-Residual Network (IRN). The IRN is also a sparse 3D-CNN and is configured with a sparse 3D convolution layer and a nonlinear transformation layer. CNN 318 (CNN_E2) may perform calculations on the sparse tensor to be processed using the three-layer IRN. CNN 318 (CNN_E2) may supply the sparse tensor X' = [C_X', F_X'] obtained as the calculation result to CNN 319 (CNN_E3). Note that the scale of this sparse tensor X' = [C_X', F_X'] is 1 / 2 (x1 / 2).

[0055] The CNN 319 (CNN_E3) is a sparse 3D-CNN, similar to the CNN 316 (CNN_E1). The CNN 319 (CNN_E3) may obtain the sparse tensor resulting from the calculations of the CNN 318 (CNN_E2) as the sparse tensor to be processed. The CNN 319 (CNN_E3) may be configured with two sparse 3D convolutional layers and two ReLU layers, and may perform 1 / 2 downscaling on the sparse tensor to be processed. The CNN 319 (CNN_E3) may supply the sparse tensor resulting from the calculations to the guide surface feature derivation unit 320. The CNN 319 (CNN_E3) may supply N_X' = |C_X'|, which corresponds to the points of the coordinate set C_X' obtained by 1 / 2 downscaling the coordinate set C_X, to the point encoding unit 326.

[0056] The guide surface feature amount derivation unit 320 is a processing unit similar to the guide surface feature amount derivation unit 315, and performs processing related to the derivation of guide surface feature amounts. That is, as described above, the guide surface feature amount derivation unit 320 acquires the sparse tensor {C^in, F^in} to be processed, derives the guide surface feature amount g(c_i) for each coordinate c_i ∈ C^in of the coordinate set C^in to be processed, connects the obtained guide surface feature amount set g(C^in) = {g(c_i)}_i to the feature amount set F^in to be processed, and returns the sparse tensor {C^in, [F^in, g(C^in)]} of the calculation result.

[0057] The guide surface feature amount derivation unit 320 may acquire the sparse tensor of the calculation result of the CNN 319 (CNN_E3) as the sparse tensor to be processed. Also, the guide surface feature amount derivation unit 320 may acquire the decoded guide surface supplied from the guide surface decoding unit 313. Then, the guide surface feature amount derivation unit 320 may use the decoded guide surface to perform the above-mentioned processing on the sparse tensor to be processed, thereby generating a sparse tensor of the calculation result. The guide surface feature amount derivation unit 320 may supply the generated sparse tensor of the calculation result to the CNN 321 (CNN_E4).

[0058] The CNN321 (CNN_E4) is a sparse 3D-CNN, similar to the CNN318 (CNN_E2). The CNN321 (CNN_E4) may acquire the sparse tensor resulting from the calculation by the guide surface feature derivation unit 320 as the sparse tensor to be processed. The CNN321 (CNN_E4) may be configured with a three-layer IRN and perform calculations on the sparse tensor to be processed using the three-layer IRN. The CNN321 (CNN_E4) may supply the sparse tensor X'' = [C_X'', F_X''] resulting from the calculation to the CNN322 (CNN_E5). Note that the scale of this sparse tensor X'' = [C_X'', F_X''] is 1 / 4 (x1 / 4).

[0059] Like CNN316 (CNN_E1), CNN322 (CNN_E5) is a sparse 3D-CNN. CNN322 (CNN_E5) may obtain the sparse tensor resulting from the computation performed by CNN321 (CNN_E4) as the sparse tensor to be processed. CNN322 (CNN_E5) may be configured with two sparse 3D convolutional layers and two ReLU layers and may perform 1 / 2 downscaling on the sparse tensor to be processed. CNN322 (CNN_E5) may supply the sparse tensor resulting from the computation to CNN323 (CNN_E6). CNN322 (CNN_E5) may supply N_X'' = |C_X''|, corresponding to the points in the coordinate set C_X'' obtained by downscaling the coordinate set C_X by 1 / 4, to the point encoding unit 326.

[0060] Like CNN318 (CNN_E2), CNN323 (CNN_E6) is a sparse 3D-CNN. CNN323 (CNN_E6) may obtain the sparse tensor resulting from the computation performed by CNN322 (CNN_E5) as the sparse tensor to be processed. CNN323 (CNN_E6) may be configured with three IRN layers and one sparse 3D convolutional layer, and may perform computations on the sparse tensor to be processed using the three IRN layers and one sparse 3D convolutional layer. CNN323 (CNN_E6) may supply the resulting sparse tensor Y = {C_Y, F_Y} to the coordinate encoding unit 324 and the feature encoding unit 325. CNN323 (CNN_E6) may also supply the resulting coordinate set C_Y to the coordinate encoding unit 324 and the resulting feature set F_Y to the feature encoding unit 325. Note that the scale of this sparse tensor Y = {C_Y, F_Y} is 1 / 8 times (x1 / 8).

[0061] The coordinate encoding unit 324 is a G-PCC (Geometry-based Point Cloud Compression) encoder and performs processing related to encoding of coordinates (coordinate sets). The coordinate encoding unit 324 may acquire a sparse tensor Y = {C_Y, F_Y} (or its coordinate set C_Y) supplied from the CNN 323 (CNN_E6). The coordinate encoding unit 324 may encode the coordinate set C_Y and generate encoded data (also referred to as a coordinate bitstream) of the coordinate set. In this case, the coordinate encoding unit 324 may encode the coordinate set C_Y using a lossless method. For example, the coordinate encoding unit 324 may encode the coordinate set C_Y using an octree. The coordinate encoding unit 324 may output the generated coordinate bitstream to the outside of the encoding device 300.

[0062] The feature encoding unit 325 performs processing related to encoding of features (feature set). The feature encoding unit 325 may acquire a sparse tensor Y = {C_Y, F_Y} (or the feature set F_Y) supplied from the CNN 323 (CNN_E6). The feature encoding unit 325 may quantize and entropy encode the feature set F_Y to generate encoded data of the feature set (also referred to as a feature bitstream). In other words, the feature encoding unit 325 encodes the feature set F_Y using a lossy encoding method. The feature encoding unit 325 may output the generated feature bitstream to the outside of the encoding device 300.

[0063] The point encoding unit 326 performs processing related to encoding the points of the coordinate sets at each scale. The point encoding unit 326 may obtain N_X = |C_X| corresponding to the points of the coordinate set C_X supplied from the CNN 316 (CNN_E1). The point encoding unit 326 may obtain N_X' = |C_X'| corresponding to the points of the coordinate set C_X' obtained by downscaling the coordinate set C_X by 1 / 2 supplied from the CNN 319 (CNN_E3). The point encoding unit 326 may obtain N_X'' = |C_X''| corresponding to the points of the coordinate set C_X'' obtained by downscaling the coordinate set C_X by 1 / 4 supplied from the CNN 322 (CNN_E5). The point encoding unit 326 may encode N_X, N_X', and N_X'' to generate encoded data of the points (also referred to as a point bitstream). The score encoding unit 326 may output the generated score bitstream to the outside of the encoding device 300 .

[0064] That is, in the encoding device 300, the guide surface feature derivation unit 315, the guide surface feature derivation unit 317, and the guide surface feature derivation unit 320 derive guide surface feature values, which are features based on the positional relationship between the guide surface and the points, using a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature set, which is a set of features of those coordinates, and a guide surface, which is a reference surface formed in three-dimensional space. Furthermore, the CNNs 316, 318, 319, 321, 322, and 323 perform predetermined calculations using the first coordinate set and the first feature set reflecting the guide surface feature values ​​(using features including the guide surface feature values) to derive a second coordinate set and a second feature set. Therefore, the CNNs 316, 318, 319, CNN 321, CNN 322, and CNN 323 can also be considered calculation units. The coordinate encoding unit 324 encodes the derived second coordinate set. The feature encoding unit 325 encodes the derived second feature set. The amount of code for the encoded data of the second coordinate set and the second feature set is smaller than the amount of code for the encoded data of the first coordinate set and the first feature set.

[0065] With this configuration, the encoding device 300 can reduce the amount of information to be encoded, and can suppress a decrease in encoding efficiency.

[0066] The configuration of the CNN and guide surface feature derivation unit may be any configuration and is not limited to the example of FIG. 7. For example, any number of CNNs and guide surface feature derivation units may be used. Furthermore, any number of downscaling operations and any number of conversion rates may be used. Furthermore, while FIG. 7 illustrates an example in which the calculation unit is a 3D-CNN, the calculation unit may have any configuration, such as a 3D-CNN, a multilayer perceptron, or another neural network, as in the calculation unit 122 of FIG. 5.

[0067] The guide surface feature derivation unit only needs to be located at the beginning of the CNN sequence. Alternatively, the guide surface feature derivation unit may be located anywhere in the CNN sequence. For example, as described above, the guide surface feature may be derived for each scale. For example, as shown in FIG. 7 , the guide surface feature derivation unit 315 may derive the guide surface feature at 1 / 1 scale, the guide surface feature derivation unit 317 may derive the guide surface feature at 1 / 2 scale, and the guide surface feature derivation unit 320 may derive the guide surface feature at 1 / 4 scale. Although omitted in the example of FIG. 7 , guide surface feature may also be derived at 1 / 8 scale. In other words, the scale of the second coordinate set and the second feature set may be smaller than the scale of the first coordinate set and the first feature set. The calculation unit may derive the second coordinate set and the second feature set by recursively repeating a predetermined calculation multiple times. The calculation unit may downscale the input coordinate set and feature set in all or some of the multiple predetermined calculations. Then, the guide surface feature amount derivation section may derive the guide surface feature amounts for some or all of the scales of the coordinate set and the feature amount set.

[0068] The guide surface may also be coded. The bit stream may also be decoded to generate (restore) the guide surface. The guide surface may also be generated. For example, as shown in FIG. 7 , the guide surface generation unit 311 may generate a guide surface using a point cloud, the guide surface coding unit 312 may code the guide surface to generate coded data of the guide surface, and the guide surface decoding unit 313 may decode the coded data of the guide surface to generate the guide surface.

[0069] <Flow of Encoding Process> The encoding device 300 performs encoding process to encode the geometry as described above. An example of the flow of this encoding process will be described with reference to the flowcharts of FIGS. 8 and 9.

[0070] When the encoding process starts, the guide surface generation unit 311 and the sparse tensor construction unit 314 acquire an input point group C_X, which is a point cloud input to the encoding device 300, in step S301 of FIG.

[0071] In step S302, the guide surface generation unit 311 generates a guide surface from the input point group C_X.

[0072] In step S303, the guide surface coding unit 312 codes the generated guide surface to generate a guide surface bit stream, and outputs the guide surface bit stream to the outside of the coding device 300.

[0073] In step S304, the guide plane decoding unit 313 decodes the guide plane bitstream and generates (restores) a decoded guide plane.

[0074] In step S305, the sparse tensor constructor 314 generates an input sparse tensor X={C_X, F_X} (scaled 1) from the input point group C_X.

[0075] In step S306, the guide surface feature amount derivation unit 315 sets {C^in, F^in} ← (C_X, F_X). That is, the guide surface feature amount derivation unit 315 sets the input sparse tensor X={C_X, F_X} as the sparse tensor (C^in, F^in) to be processed.

[0076] In step S307, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 determines whether or not to input the guide surface feature amount to the next 3D-CNN. If it is determined that the guide surface feature amount is to be input to the next 3D-CNN, the process proceeds to step S308.

[0077] In step S308, the guide surface feature derivation unit 315, the guide surface feature derivation unit 317, or the guide surface feature derivation unit 320 executes a guide surface feature set derivation process and derives a guide surface feature set g(C^in) using the sparse tensor to be processed and the decoded guide surface.

[0078] In step S309, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 sets F^in ← [F^in, g(C^in)]. That is, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 concatenates the feature amount set to be processed and the guide surface feature amount set g(C^in) as the feature amount set to be processed by the next 3D-CNN.

[0079] When the processing of step S309 is completed, the processing proceeds to Fig. 9. Also, if it is determined in step S307 that the guide surface feature amount will not be input to the next 3D-CNN, the processing proceeds to Fig. 9. In other words, if there is no guide surface feature amount derivation unit immediately before the 3D-CNN, the processing of steps S308 and S309 is omitted. For example, if the encoding device 300 has the configuration example shown in Fig. 7, when the next 3D-CNN is CNN316 (CNN_E1), CNN318 (CNN_E2), or CNN321 (CNN_E4), the processing of steps S308 and S309 is executed, and when the next 3D-CNN is CNN319 (CNN_E3), CNN322 (CNN_E5), or CNN323 (CNN_E6), the processing of steps S308 and S309 is omitted.

[0080] In step S321 of Figure 9, CNN316 (CNN_E1), CNN318 (CNN_E2), CNN319 (CNN_E3), CNN321 (CNN_E4), CNN322 (CNN_E5), or CNN323 (CNN_E6) performs 3D-CNN operations on the sparse tensor {C^in, F^in} to be processed, and derives the sparse tensor {C^out, F^out} as the operation result.

[0081] In step S322, the CNN 323 (CNN_E6) determines whether processing has been performed for all 3D-CNNs. If it is determined that processing has not been performed for all 3D-CNNs (i.e., that there are unprocessed 3D-CNNs (at least the CNN 323 (CNN_E6) is unprocessed)), the process proceeds to step S323.

[0082] In step S323, CNN316 (CNN_E1), CNN318 (CNN_E2), CNN319 (CNN_E3), CNN321 (CNN_E4), or CNN322 (CNN_E5) sets {C^in, F^in} ← {C^out, F^out}. In other words, CNN316 (CNN_E1), CNN318 (CNN_E2), CNN319 (CNN_E3), CNN321 (CNN_E4), or CNN322 (CNN_E5) sets the sparse tensor {C^out, F^out} of the calculation result derived by itself as the sparse tensor {C^in, F^in} to be processed by the next processing unit (guide surface feature derivation unit or 3D-CNN).

[0083] When the process of step S323 is completed, the process returns to step S307 in Fig. 8, and the subsequent processes are executed. That is, the processes of steps S307 to S309 in Fig. 8 and the processes of steps S321 to S323 in Fig. 9 are executed for each 3D-CNN (including the guide surface feature amount derivation unit if one exists immediately before the 3D-CNN).

[0084] Then, in step S322 of FIG. 9, if it is determined that the processing has been performed for all 3D-CNNs (i.e., the processing has been performed for CNN323 (CNN_E6)), the processing proceeds to step S324.

[0085] In step S324, the CNN 323 (CNN_E6) sets {C_Y, F_Y} ← {C^out, F^out}. That is, the last CNN 323 (CNN_E6) sets the sparse tensor {C^out, F^out}, which is the calculation result derived by itself, as the output sparse tensor {C_Y, F_Y}.

[0086] In step S325, the coordinate encoding unit 324 encodes the coordinate set C_Y of the output sparse tensor {C_Y, F_Y} to generate a coordinate bitstream, and outputs the coordinate bitstream to the outside of the encoding device 300.

[0087] In step S326, the feature encoding unit 325 encodes the feature set F_Y of the output sparse tensor {C_Y, F_Y} to generate a feature bitstream, and outputs the feature bitstream to the outside of the encoding device 300.

[0088] In step S327, the score for each scale is coded to generate a score bit stream, which is then output to the outside of the coding device 300.

[0089] When the process of step S327 is completed, the encoding process ends.

[0090] <Flow of Guide Surface Feature Amount Set Deriving Process> Next, an example of the flow of the guide surface feature amount set deriving process executed in step S308 of FIG. 8 will be described with reference to the flowchart of FIG.

[0091] When the guide surface feature amount set derivation process is started, in step S331, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 extracts coordinates c_i ∈ C^in. That is, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 selects one coordinate c_i from the coordinate set C^in to be processed, and sets it as the processing target.

[0092] In step S332, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 searches for the nearest point p_i on the decoded guide surface as viewed from the selected coordinate c_i.

[0093] In step S333, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 returns g(c_i) = p_i - c_i. That is, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 derives the guide surface feature amount g(c_i) based on the positional relationship between the guide surface and the point (in this example, p_i - c_i).

[0094] In step S334, the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, or the guide surface feature amount derivation unit 320 determines whether or not processing has been performed for all coordinates c_i ∈ C^in. If it is determined that an unprocessed coordinate c_i exists, the processing returns to step S331, and the subsequent processing is executed. That is, the processing of steps S331 to S334 is executed for each coordinate value c_i of the coordinate set C^in to be processed.

[0095] Then, in step S334, if it is determined that the process has been performed for all coordinate values ​​c_i, the guide surface feature amount set derivation process ends, and the process returns to FIG.

[0096] By performing each process in this manner, the encoding device 300 can reduce the amount of information to be encoded, and can suppress a decrease in encoding efficiency.

[0097] <Decoding device> Fig. 11 is a block diagram showing an example of the configuration of a decoding device, which is one aspect of an information processing device to which the present technology is applied. The decoding device 350 shown in Fig. 11 is a device that decodes a bit stream in which the geometry of a point cloud (3D data) is encoded. For example, the decoding device 350 decodes bit streams (a guide surface bit stream, a coordinate bit stream, a feature bit stream, and a point number bit stream) generated by encoding the point cloud (the geometry) by the encoding device 300, and generates (restores) the point cloud (the geometry).

[0098] Note that Fig. 11 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in the decoding device 350, there may be processing units that are not shown as blocks in Fig. 11, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 11.

[0099] As shown in FIG. 11, the decoding device 350 has a score decoding unit 361, a coordinate decoding unit 362, a feature decoding unit 363, a guide surface decoding unit 364, a CNN 365 (CNN_D1), a guide surface feature derivation unit 366, a CNN 367 (CNN_D2), an occupancy state classification unit 368, a CNN 369 (CNN_D3), a guide surface feature derivation unit 370, a CNN 371 (CNN_D4), an occupancy state classification unit 372, a CNN 373 (CNN_D5), a guide surface feature derivation unit 374, a CNN 375 (CNN_D6), and an occupancy state classification unit 376.

[0100] The point decoding unit 361 performs processing related to decoding of a point bitstream. For example, the point decoding unit 361 may acquire a point bitstream input to the decoding device 350. The point decoding unit 361 may decode the acquired point bitstream to generate (restore) information indicating the points of coordinate sets at each scale. For example, the point decoding unit 361 may decode the point bitstream to generate (restore) N_X = |C_X| corresponding to the points of the coordinate set C_X before encoding, N_X' = |C_X'| corresponding to the points of the coordinate set C_X' obtained by downscaling the coordinate set C_X before encoding by half, and N_X'' = |C_X''| corresponding to the points of the coordinate set C_X'' obtained by downscaling the coordinate set C_X before encoding by a quarter, all of which are encoded by the point encoding unit 326 ( FIG. 7 ). The point decoding unit 361 may supply the generated N_X to the occupancy state classification unit 376. The score decoding unit 361 may supply the generated N_X′ to the occupancy state classification unit 372. The score decoding unit 361 may supply the generated N_X″ to the occupancy state classification unit 368.

[0101] The coordinate decoding unit 362 is a G-PCC decoder and performs processing related to decoding of a coordinate bitstream. For example, the coordinate decoding unit 362 may acquire a coordinate bitstream input to the decoding device 350. The coordinate decoding unit 362 may decode the acquired coordinate bitstream and generate (restore) a coordinate set C_Y. In this case, the coordinate decoding unit 362 may decode the coordinate bitstream using a lossless method. For example, the coordinate decoding unit 362 may decode the coordinate bitstream using an octree. The coordinate decoding unit 362 may supply the generated coordinate set C_Y to the CNN 365 (CNN_D1).

[0102] The feature decoding unit 363 executes processing related to the decoding of the feature bitstream. For example, the feature decoding unit 363 may acquire the feature bitstream input to the decoding device 350. The feature decoding unit 363 may entropy decode the acquired feature bitstream to generate (restore) a feature set F_Y. In this case, the feature decoding unit 363 decodes the feature bitstream using a lossy method. Therefore, the feature set F_Y generated by the feature decoding unit 363 does not need to be completely identical to the feature set F_Y before being encoded by the feature encoding unit 325 ( FIG. 7 ). The feature decoding unit 363 may supply the generated feature set F_Y to the CNN 365 (CNN_D1).

[0103] The guide surface decoding unit 364 is a processing unit similar to the guide surface decoding unit 313 ( FIG. 7 ) and performs processing related to the decoding of the guide surface bitstream. For example, the guide surface decoding unit 364 may acquire the guide surface bitstream input to the decoding device 350. The guide surface decoding unit 364 may decode the acquired guide surface bitstream to generate a decoded guide surface. This decoding method may be any method that corresponds to the encoding method applied by the guide surface encoding unit 312 ( FIG. 7 ). For example, if the guide surface is configured by a mesh, the guide surface decoding unit 364 may decode the guide surface bitstream by applying a decoding method for meshes. This decoding method may be an existing method. For example, if the guide surface is a mesh, Draco decoding may be applied. Note that the encoding and decoding of this guide surface may be a lossless method or a lossy method. In other words, the guide surface generated (restored) by the guide surface decoding unit 364 (also referred to as a decoded guide surface) does not have to be completely identical to the guide surface before encoding (for example, the guide surface generated by the guide surface generation unit 311). The guide surface decoding section 364 may supply the generated decoded guide surface to the guide surface feature amount derivation section 366 , the guide surface feature amount derivation section 370 , and the guide surface feature amount derivation section 374 .

[0104] The CNN 365 (CNN_D1) is a sparse 3D-CNN, primarily composed of sparse 3D convolutional layers and nonlinear transformation layers. It acquires a sparse tensor as a processing target, performs a predetermined operation on the sparse tensor, and outputs the resulting sparse tensor. For example, the CNN 365 (CNN_D1) may acquire a coordinate set C_Y supplied from the coordinate decoding unit 362 and a feature set F_Y supplied from the feature decoding unit 363 as a sparse tensor Y = {C_Y, F_Y} to be processed. The scale of this sparse tensor Y = {C_Y, F_Y} is 1 / 8 times (x1 / 8) the scale before encoding. The CNN 365 (CNN_D1) is composed of two sparse 3D convolutional layers and two ReLU layers, and may perform a 2x upscale on the sparse tensor to be processed. The CNN 365 (CNN_D1) may supply the sparse tensor of the obtained calculation result to the guide surface feature derivation unit 366.

[0105] The guide surface feature amount derivation unit 366 is a processing unit similar to the guide surface feature amount derivation unit of the encoding device 300, such as the guide surface feature amount derivation unit 315, and performs processing related to the derivation of guide surface feature amounts. The guide surface feature amount derivation unit 366 acquires a sparse tensor {C^in, F^in} to be processed, derives a guide surface feature amount g(c_i) for each coordinate c_i ∈ C^in of the coordinate set C^in to be processed, concatenates the obtained guide surface feature amount set g(C^in) = {g(c_i)}_i to the feature amount set F^in to be processed, and returns the sparse tensor {C^in, [F^in, g(C^in)]} of the calculation result. As a method of calculating the guide surface feature amount g(c_i), as in the case of the encoding device 300, there is a method of searching for the nearest point p_i on the guide surface as viewed from the coordinate value c_i and setting g(c_i) = p_i - c_i ∈ R^3. That is, in this case, the dimension of the feature set [F^in, g(C^in)] resulting from the calculation is three more than the dimension of the feature set F^in to be processed.

[0106] The guide surface feature amount derivation unit 366 may acquire a sparse tensor resulting from the calculation by the CNN 365 (CNN_D1) as a sparse tensor to be processed. The guide surface feature amount derivation unit 366 may also acquire a decoded guide surface supplied from the guide surface decoding unit 364. Then, the guide surface feature amount derivation unit 366 may use the decoded guide surface to perform the above-described processing on the sparse tensor to be processed, thereby generating a sparse tensor as a calculation result. The guide surface feature amount derivation unit 366 may also supply the generated sparse tensor as a calculation result to the CNN 367 (CNN_D2).

[0107] Like CNN365 (CNN_D1), CNN367 (CNN_D2) is a sparse 3D-CNN, primarily composed of sparse 3D convolutional layers and nonlinear transformation layers. That is, CNN367 (CNN_D2) acquires a sparse tensor as a processing target, performs a predetermined calculation on the sparse tensor, and outputs the sparse tensor resulting from the calculation. For example, CNN367 (CNN_D2) may acquire the sparse tensor resulting from the calculation by the guide surface feature derivation unit 366 as the sparse tensor to be processed. However, since the sparse tensor to be processed has an increased dimension by the amount of the guide surface feature, the input / output dimensions of CNN367 (CNN_D2) are also changed accordingly. CNN367 (CNN_D2) may be composed of a three-layer IRN, and may perform calculations on the sparse tensor to be processed using the three-layer IRN. The CNN 367 (CNN_D2) may supply the sparse tensor {Up(C_Y), F_X''} obtained as the calculation result to the occupancy state classification unit 368. Note that Up() represents a process of upsampling by a factor of 2 in the x, y, and z directions. For example, the score of Up(C_Y) is 8 times the score of C_Y.

[0108] The occupancy state classification unit 368 is composed of a single sparse 3D convolutional layer. This sparse 3D convolutional layer inputs the sparse tensor to be processed and outputs the sparse tensor resulting from the calculation. Note that the features of the sparse tensor resulting from the calculation of this sparse 3D convolutional layer represent the occupancy probability values ​​of the coordinates. The occupancy state classification unit 368 extracts a predetermined number of coordinates (the top k coordinates) with the highest occupancy probability values ​​and deletes the remaining coordinates. In other words, the occupancy state classification unit 368 outputs a coordinate set in which all but the top k coordinates of the input sparse tensor have been deleted, and a feature set corresponding to that coordinate set (i.e., the feature set of the input sparse tensor in which the features corresponding to the deleted coordinates have been deleted). Note that k is the scale value of N_X, N_X', or N_X'' that corresponds to the sparse tensor to be processed. The occupancy state classification unit 368 also deletes the features corresponding to the deleted coordinates in F^out.

[0109] The occupancy state classification unit 368 may acquire the sparse tensor {Up(C_Y), F_X''} resulting from the calculation by the CNN 367 (CNN_D2) as the sparse tensor to be processed. The occupancy state classification unit 368 may acquire N_X'' = |C_X''| corresponding to the points of a coordinate set C_X'' obtained by downscaling the coordinate set C_X before encoding by 1 / 4, supplied from the score decoding unit 361. The occupancy state classification unit 368 may predict an occupancy probability value using the sparse tensor to be processed, extract the top N_X'' coordinate values ​​with the occupancy probability values, and delete the other coordinate values ​​to generate the sparse tensor X'' = {C_X'', F_X''} resulting from the calculation. The occupancy state classification unit 368 may supply the sparse tensor X'' = {C_X'', F_X''} resulting from the calculation to the CNN 369 (CNN_D3).

[0110] Like CNN365 (CNN_D1), CNN369 (CNN_D3) is a sparse 3D-CNN, primarily composed of sparse 3D convolutional layers and nonlinear transformation layers. That is, CNN369 (CNN_D3) acquires a sparse tensor as a processing target, performs a predetermined operation on the target sparse tensor, and outputs the resulting sparse tensor. For example, CNN369 (CNN_D3) may acquire the sparse tensor X'' = {C_X'', F_X''} resulting from the operation of the occupancy state classification unit 368 as the processing target sparse tensor. The scale of this sparse tensor X'' = {C_X'', F_X''} is 1 / 4 times (x1 / 4) the scale before encoding. CNN369 (CNN_D3) is composed of two sparse 3D convolutional layers and two ReLU layers, and may perform upscaling of the processing target sparse tensor by a factor of 2. The CNN 369 (CNN_D3) may supply the sparse tensor of the obtained calculation result to the guide surface feature derivation unit 370.

[0111] The guide surface feature amount derivation unit 370 is a processing unit similar to the guide surface feature amount derivation unit 366, and performs processing related to the derivation of guide surface feature amounts. As in the case of the guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370 acquires the sparse tensor {C^in, F^in} to be processed, derives the guide surface feature amount g(c_i) for each coordinate c_i ∈ C^in of the coordinate set C^in to be processed, concatenates the obtained guide surface feature amount set g(C^in) = {g(c_i)}_i to the feature amount set F^in to be processed, and returns the sparse tensor {C^in, [F^in, g(C^in)]} of the calculation result.

[0112] The guide surface feature amount derivation unit 370 may acquire the sparse tensor of the calculation result of the CNN 369 (CNN_D3) as the sparse tensor to be processed. Also, the guide surface feature amount derivation unit 370 may acquire the decoded guide surface supplied from the guide surface decoding unit 364. Then, the guide surface feature amount derivation unit 370 may use the decoded guide surface to perform the above-mentioned processing on the sparse tensor to be processed, thereby generating a sparse tensor of the calculation result. Also, the guide surface feature amount derivation unit 370 may supply the generated sparse tensor of the calculation result to the CNN 371 (CNN_D4).

[0113] Like CNN367 (CNN_D2), CNN371 (CNN_D4) is a sparse 3D-CNN, primarily composed of sparse 3D convolutional layers and nonlinear transformation layers. That is, CNN371 (CNN_D4) acquires a sparse tensor as a processing target, performs a predetermined calculation on the sparse tensor, and outputs the sparse tensor resulting from the calculation. For example, CNN371 (CNN_D4) may acquire the sparse tensor resulting from the calculation by the guide surface feature derivation unit 370 as the sparse tensor to be processed. However, since the sparse tensor to be processed has an increased dimension by the amount of the guide surface feature, the input / output dimensions of CNN371 (CNN_D4) are also changed accordingly. CNN371 (CNN_D4) may be composed of a three-layer IRN, and may perform calculations on the sparse tensor to be processed using the three-layer IRN. The CNN 371 (CNN_D4) may supply the sparse tensor {Up(C_X″), F_X′} obtained as the calculation result to the occupancy state classification unit 372.

[0114] The occupancy state classification unit 372 is a processing unit similar to the occupancy state classification unit 368, and is configured with one sparse 3D convolutional layer. It predicts the occupancy probability value of the sparse tensor to be processed, extracts a predetermined number of coordinates (the top k coordinates) with the highest occupancy probability values, and deletes the remaining coordinates. The occupancy state classification unit 372 also deletes the features corresponding to the deleted coordinates from F̂out.

[0115] For example, the occupancy state classification unit 372 may acquire the sparse tensor {Up(C_X″), F_X′} resulting from the computation by the CNN 371 (CNN_D4) as the sparse tensor to be processed. The occupancy state classification unit 372 may acquire N_X′ = |C_X′| corresponding to the points of a coordinate set C_X′ obtained by downscaling the coordinate set C_X before encoding by half, which is supplied from the score decoding unit 361. The occupancy state classification unit 372 may predict an occupancy probability value using the sparse tensor to be processed, extract the top N_X′ coordinate values ​​of the occupancy probability values, and delete the other coordinate values ​​to generate the sparse tensor X′ = {C_X′, F_X′} resulting from the computation. The occupancy state classification unit 372 may supply the sparse tensor X′ = {C_X′, F_X′} resulting from the computation to the CNN 373 (CNN_D5).

[0116] Like CNN365 (CNN_D1), CNN373 (CNN_D5) is a sparse 3D-CNN, primarily composed of sparse 3D convolutional layers and nonlinear transformation layers. That is, CNN373 (CNN_D5) acquires a sparse tensor as a processing target, performs a predetermined operation on the target sparse tensor, and outputs the resulting sparse tensor. For example, CNN373 (CNN_D5) may acquire the sparse tensor X' = {C_X', F_X'}, the result of the operation of the occupancy state classification unit 372, as the processing target sparse tensor. The scale of this sparse tensor X' = {C_X', F_X'} is half (x1 / 2) the scale before encoding. CNN373 (CNN_D5) is composed of two sparse 3D convolutional layers and two ReLU layers, and may perform a 2x upscale on the processing target sparse tensor. The CNN 373 (CNN_D5) may supply the sparse tensor of the obtained calculation result to the guide surface feature derivation unit 374.

[0117] The guide surface feature amount derivation unit 374 is a processing unit similar to the guide surface feature amount derivation unit 366, and performs processing related to the derivation of guide surface feature amounts. As in the case of the guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 374 acquires the sparse tensor {C^in, F^in} to be processed, derives the guide surface feature amount g(c_i) for each coordinate c_i ∈ C^in of the coordinate set C^in to be processed, connects the obtained guide surface feature amount set g(C^in) = {g(c_i)}_i to the feature amount set F^in to be processed, and returns the sparse tensor {C^in, [F^in, g(C^in)]} of the calculation result.

[0118] The guide surface feature amount derivation unit 374 may acquire the sparse tensor of the calculation result of the CNN 373 (CNN_D5) as the sparse tensor to be processed. Also, the guide surface feature amount derivation unit 374 may acquire the decoded guide surface supplied from the guide surface decoding unit 364. Then, the guide surface feature amount derivation unit 374 may use the decoded guide surface to perform the above-mentioned processing on the sparse tensor to be processed, thereby generating a sparse tensor of the calculation result. Also, the guide surface feature amount derivation unit 374 may supply the generated sparse tensor of the calculation result to the CNN 375 (CNN_D6).

[0119] Like CNN367 (CNN_D2), CNN375 (CNN_D6) is a sparse 3D-CNN, primarily composed of sparse 3D convolutional layers and nonlinear transformation layers. That is, CNN375 (CNN_D6) acquires a sparse tensor as a processing target, performs a predetermined calculation on the sparse tensor, and outputs the sparse tensor resulting from the calculation. For example, CNN375 (CNN_D6) may acquire the sparse tensor resulting from the calculation by the guide surface feature derivation unit 374 as the sparse tensor to be processed. However, since the sparse tensor to be processed has an increased dimension corresponding to the guide surface feature, the input / output dimensions of CNN375 (CNN_D6) are also changed accordingly. CNN375 (CNN_D6) may be composed of a three-layer IRN, and may perform calculations using the three-layer IRN on the sparse tensor to be processed. The CNN 375 (CNN_D6) may supply the sparse tensor {Up(C_X′), F_X} obtained as the calculation result to the occupancy state classification unit 376.

[0120] The occupancy state classification unit 376 is a processing unit similar to the occupancy state classification unit 368, and is configured with one sparse 3D convolutional layer. It predicts the occupancy probability value of the sparse tensor to be processed, extracts a predetermined number of coordinates (the top k coordinates) with the highest occupancy probability values, and deletes the other coordinates. The occupancy state classification unit 376 also deletes the features corresponding to the deleted coordinates from F̂out.

[0121] For example, the occupancy state classification unit 376 may acquire the sparse tensor {Up(C_X'), F_X} resulting from the computation of the CNN 375 (CNN_D6) as the sparse tensor to be processed. The occupancy state classification unit 376 may acquire N_X = |C_X|, corresponding to the number of points in the coordinate set C_X before encoding, supplied from the score decoding unit 361. The occupancy state classification unit 376 may predict an occupancy probability value using the sparse tensor to be processed, extract the top N_X coordinate values ​​with the occupancy probability value, and delete the remaining coordinate values ​​to generate the sparse tensor X = {C_X, F_X} resulting from the computation. The scale of this sparse tensor X = {C_X, F_X} is 1x (x1) compared to before encoding. In other words, the occupancy state classification unit 376 may generate (restore) a point cloud obtained by decoding a group of bit streams (score bit stream, guide surface bit stream, coordinate bit stream, and feature bit stream). This point cloud is also referred to as a decoded point cloud C^X. The occupancy state classification unit 376 may output the decoded point cloud C_X, which is the calculation result, to the outside of the decoding device 350.

[0122] That is, in the decoding device 350, the coordinate decoding unit 362 decodes the encoded data to generate a first coordinate set, which is a set of coordinates of points forming the point cloud. The feature decoding unit 363 decodes the encoded data to generate a first feature set, which is a set of feature values ​​of the coordinates. The guide surface feature derivation unit 366, the guide surface feature derivation unit 370, and the guide surface feature derivation unit 374 derive guide surface feature values, which are feature values ​​based on the positional relationship between the guide surface and the points, using the first coordinate set, the first feature set, and the guide surface, which is a reference surface formed in three-dimensional space. The CNNs 365, 367, 369, 371, 373, and 375 perform predetermined calculations using the first coordinate set and the first feature set reflecting the guide surface feature values, to derive a second coordinate set and a second feature set. Therefore, CNN365, CNN367, CNN369, CNN371, CNN373, and CNN375 can also be considered as calculation units.

[0123] With this configuration, the decoding device 350 can suppress a decrease in coding efficiency.

[0124] Note that the configurations of the CNN, guide surface feature derivation unit, occupancy state classification unit, etc. may be any and are not limited to the example of FIG. 11. For example, any number of CNNs, guide surface feature derivation units, and occupancy state classification units may be used. Furthermore, any number of downscaling operations and any number of conversion rates may be used. Furthermore, while FIG. 11 illustrates an example in which the calculation unit is a 3D-CNN, the calculation unit may have any configuration, such as a 3D-CNN, a multilayer perceptron, or another neural network, as in the calculation unit 122 of FIG. 5.

[0125] The guide surface feature derivation unit may be located anywhere in the CNN sequence. For example, as described above, the guide surface feature may be derived for each scale. For example, as shown in FIG. 11 , the guide surface feature derivation unit 366 may derive the guide surface feature at 1 / 4 scale, the guide surface feature derivation unit 370 may derive the guide surface feature at 1 / 2 scale, and the guide surface feature derivation unit 374 may derive the guide surface feature at 1 / 1 scale. Although omitted in the example of FIG. 11 , guide surface feature may also be derived at 1 / 8 scale. In other words, the scale of the second coordinate set and the second feature set may be larger than the scale of the first coordinate set and the first feature set. The calculation unit may derive the second coordinate set and the second feature set by recursively repeating a predetermined calculation multiple times. The calculation unit may upscale the input coordinate set and feature set in all or part of the predetermined calculation multiple times. Then, the guide surface feature amount derivation section may derive the guide surface feature amounts for some or all of the scales of the coordinate set and the feature amount set.

[0126] Alternatively, the guide surface may be generated (restored) by decoding. For example, as shown in Fig. 11, a guide surface decoding unit 364 may decode the coded data of the guide surface and generate the guide surface.

[0127] <Flow of Decoding Process> The decoding device 350 executes the decoding process to decode the bitstream and generate (restore) geometry as described above. An example of the flow of this decoding process will be described with reference to the flowcharts in FIGS. 12 and 13.

[0128] When the decoding process starts, the score decoding unit 361 decodes the score bit stream in step S351 of FIG. 12 to generate scores (N_X, N_X', N_X'') for each scale.

[0129] In step S352, the coordinate decoding unit 362 decodes the coordinate bit stream to generate a coordinate set C_Y.

[0130] In step S353, the feature decoding unit 363 decodes the feature bit stream to generate a feature set F_Y.

[0131] In step S354, the guide surface decoding unit 364 decodes the guide surface bitstream to generate a decoded guide surface.

[0132] In step S355, the CNN 365 (CNN_D1) sets {C^in, F^in} ← {C_Y, F_Y}. That is, the CNN 365 (CNN_D1) sets the sparse tensor Y = {C_Y, F_Y} as the sparse tensor {C^in, F^in} to be processed.

[0133] In step S356, CNN365 (CNN_D1), CNN367 (CNN_D2), CNN369 (CNN_D3), CNN371 (CNN_D4), CNN373 (CNN_D5), or CNN375 (CNN_D6) determines whether to input guide surface features to its own 3D-CNN. If it is determined that the guide surface features are to be input, the process proceeds to step S357. In other words, if the CNN to be processed is CNN367 (CNN_D2), CNN371 (CNN_D4), or CNN375 (CNN_D6), the process proceeds to step S357.

[0134] In step S357, the guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370, or the guide surface feature amount derivation unit 374 executes a guide surface feature amount set derivation process to derive a guide surface feature amount set g(C^in). This guide surface feature amount set derivation process is executed in the same manner as described with reference to the flowchart in FIG.

[0135] In step S358, the guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370, or the guide surface feature amount derivation unit 374 sets F^in ← [F^in, g(C^in)]. That is, the guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370, or the guide surface feature amount derivation unit 374 sets the concatenation of the feature amount set to be processed and the guide surface feature amount set g(C^in) as the feature amount set to be processed by the next 3D-CNN.

[0136] When the processing of step S358 is completed, the processing proceeds to Fig. 13. Also, in step S356 of Fig. 12, if it is determined that the guide surface feature amount is not to be input to its own 3D-CNN, the processing of steps S357 and S358 is omitted, and the processing proceeds to Fig. 13. In other words, if the CNN to be processed is CNN365 (CNN_D1), CNN369 (CNN_D3), or CNN373 (CNN_D5), the processing of steps S357 and S358 is omitted, and the processing proceeds to Fig. 13.

[0137] In step S371 of Figure 13, CNN365 (CNN_D1), CNN367 (CNN_D2), CNN369 (CNN_D3), CNN371 (CNN_D4), CNN373 (CNN_D5), or CNN375 (CNN_D6) performs 3D-CNN operations on the sparse tensor {C^in, F^in} to be processed, and derives the sparse tensor {C^out, F^out} as the operation result.

[0138] In step S372, the occupancy status classification unit 368, the occupancy status classification unit 372, or the occupancy status classification unit 376 determines whether to classify the occupancy status. If it is determined that the occupancy status is to be classified, the process proceeds to step S373. In other words, if the process of step S371 is performed by CNN 367 (CNN_D2), CNN 371 (CNN_D4), or CNN 375 (CNN_D6), the process proceeds to step S373.

[0139] In step S373, the occupancy state classification unit 368, the occupancy state classification unit 372, or the occupancy state classification unit 376 predicts the occupancy probability of each coordinate in the coordinate set C^out, extracts the top k coordinates with the highest occupancy probabilities, and deletes the remaining coordinates. At this time, the k is determined by applying a scale value from N_X, N_X', or N_X'' that corresponds to the sparse tensor to be processed. The occupancy state classification unit 368, the occupancy state classification unit 372, or the occupancy state classification unit 376 also deletes the features corresponding to the deleted coordinates in F^out.

[0140] When the processing of step S373 is completed, the processing proceeds to step S374. Also, if it is determined in step S372 that occupancy state classification is not to be performed, the processing of step S373 is omitted, and the processing proceeds to step S374. In other words, if the processing of step S371 is performed by CNN365 (CNN_D1), CNN369 (CNN_D3), or CNN373 (CNN_D5), the processing of step S373 is omitted, and the processing proceeds to step S374.

[0141] In step S374, the CNN 375 (CNN_D6) determines whether processing has been performed for all 3D-CNNs. If it is determined that processing has not been performed for all 3D-CNNs (i.e., that there are unprocessed 3D-CNNs (at least the CNN 375 (CNN_D6) is unprocessed)), the process proceeds to step S375.

[0142] In step S375, CNN365 (CNN_D1), CNN367 (CNN_D2), CNN369 (CNN_D3), CNN371 (CNN_D4), CNN373 (CNN_D5), or CNN375 (CNN_D6) sets {C^in, F^in} ← {C^out, F^out}. In other words, CNN365 (CNN_D1), CNN367 (CNN_D2), CNN369 (CNN_D3), CNN371 (CNN_D4), CNN373 (CNN_D5), or CNN375 (CNN_D6) sets the sparse tensor {C^out, F^out} resulting from the calculation it derived as the sparse tensor {C^in, F^in} to be processed by the next processing unit (guide surface feature derivation unit or occupancy state classification unit). When the processing of step S375 is completed, the process returns to step S356 of FIG. 12, and subsequent processing is executed. That is, the processes of steps S356 to S358 in FIG. 12 and the processes of steps S371 to S375 in FIG. 13 are executed for each 3D-CNN (which may include a guide surface feature amount derivation unit and an occupancy state classification unit).

[0143] Then, in step S374 of FIG. 13, if it is determined that the processing has been performed for all 3D-CNNs, the processing proceeds to step S376.

[0144] In step S376, the occupancy state classification unit 376 sets C_X←C^out, that is, the occupancy state classification unit 376 sets the coordinate set C^out of the calculation result as the decoding result (decoded point group C_X).

[0145] In step S377, the occupancy state classification unit 376 outputs the decoded point group C_X to the outside of the decoding device 350.

[0146] When the process of step S377 ends, the decoding process ends.

[0147] By performing each process in this manner, the decoding device 350 can suppress a decrease in coding efficiency.

[0148] <Method 1-1> Also, when the above-mentioned method 1 is applied, as shown in the second row from the top of the table in Figure 4, the guide surface feature may be a feature based on the point closest to the guide surface from the point (Method 1-1).

[0149] For example, in the first information processing device or the second information processing device, the guide surface feature amount may be a feature amount based on the nearest point on the guide surface with respect to the point as the reference.

[0150] <Method 1-1-1> Furthermore, when the above-mentioned method 1-1 is applied, as shown in the third row from the top of the table in Fig. 4, the guide surface feature amount may be a vector pointing from a point to the nearest point (Method 1-1-1). For example, as shown in Fig. 14, for each coordinate c_i = (x_i, y_i, z_i) of the coordinate set C^in, the nearest point p_i on the guide surface 142 is derived. The vector p_i - c_i ∈ R^3 is taken as the guide surface feature amount. That is, in this case, g(c_i) = p_i - c_i. By doing this, roughly speaking, the decoder can utilize information such as how far the coordinate is from the guide surface as a reference.

[0151] <Method 1-1-2> Also, when the above-mentioned method 1-1 is applied, as shown in the fourth row from the top of the table in Figure 4, the guide surface feature may be a feature related to the distance between a point and the nearest point (Method 1-1-2).

[0152] For example, the guide surface feature amount may be a signed distance. A signed distance is a scalar value obtained by multiplying the distance from a certain point to a boundary surface by +1 if the point is included within the boundary, and by -1 if not. In other words, if the distance from coordinate c_i to the nearest point p_i on the guide surface is expressed as ||p_i - c_i||, then g(c_i) is defined as in the following equation (2).

[0153] ...(2)

[0154] However, signed distance can only be defined for surfaces that have an inside and an outside. For example, if the guide surface is a mesh, the mesh must be watertight (in simple terms, a mesh with no holes). If the mesh is not watertight, the unsigned distance g(c_i) = ||p_i - c_i|| can be applied instead of signed distance. The unsigned distance simply indicates the distance, and can be calculated even for meshes that are not watertight.

[0155] Alternatively, a truncated signed distance, which limits the range of absolute values ​​in the signed distance, may be applied as the guide surface feature amount. That is, a maximum absolute value τ is determined, and if the absolute value of the signed distance is greater than τ, it is truncated to −τ or τ.

[0156] <Method 1-1-3> Furthermore, when the above-described method 1-1 is applied, as shown in the fifth row from the top of the table in FIG. 4, the guide surface feature amount may be the feature amount of the nearest point (method 1-1-3).

[0157] That is, some feature value h(p_i) at the nearest point p_i on the guide surface may be applied as the guide surface feature value g(c_i). That is, in this case, it is defined as g(c_i) = h(p_i). This feature value h(p_i) may be anything. For example, it may be the normal vector at the nearest point p_i, or the curvature of the guide surface at the nearest point p_i (a numerical value that indicates the degree of curvature of the surface, and Gaussian curvature or mean curvature can be specifically calculated).

[0158] <Method 1-2> The guide surface, which is a reference surface formed in three-dimensional space, may be configured using any 3D data. For example, when the above-mentioned method 1 is applied, the guide surface may be a mesh (Method 1-2), as shown in the sixth row from the top of the table in Figure 4.

[0159] For example, in the first information processing device and the second information processing device, the guide surface may be a mesh.

[0160] Generally, a mesh contains less information than a point cloud. By encoding the guide surface feature quantity using the mesh as the guide surface (reference surface), the amount of information to be encoded can be reduced compared to when encoding the point cloud as is. Therefore, it is possible to suppress a decrease in encoding efficiency.

[0161] <Method 1-2-1> Furthermore, when the guide surface is a mesh, the guide surface may be output as 3D data of the decoding result. For example, the restoration of the point cloud may be omitted, and the decoded guide surface may be output as the decoding result. By doing so, it is possible to suppress an increase in the load of the decoding process. Furthermore, the decoded guide surface may be fitted with the restored point cloud. In other words, when the above-mentioned method 1-2 is applied, the guide surface may be fitted to the point cloud as shown in the seventh row from the top of the table in Fig. 4 (Method 1-2-1).

[0162] For example, the three-dimensional spaces 380A and 380B in FIG. 15 are two-dimensional representations of three-dimensional spaces. In these three-dimensional spaces 380A and 380B, gray squares 381 represent points of the reconstructed point cloud. Although only one square is labeled in FIG. 15, all gray squares represent points of the point cloud. In contrast, assume that guide surface 382A is reconstructed as in three-dimensional space 380A. When the guide surface is a mesh, as in this example, the mesh surfaces and vertices may be misaligned with respect to the point cloud. In other words, if such a guide surface were used as the decoding result, the resolution would be lower than in the case of a point cloud, potentially reducing the quality of the decoding result.

[0163] Therefore, like guide surface 382B in three-dimensional space 380B, fitting may be performed on the guide surface using points of the point cloud (so that the surfaces and vertices of the guide surface approach the points of the point cloud). By doing so, it is possible to suppress a reduction in the definition of the guide surface and a reduction in the quality of the decoding result.

[0164] For example, the second information processing device may further include a fitting unit that fits the guide surface to the point cloud.

[0165] <Decoding Apparatus> When applying such method 1-2-1, the configuration of the encoding apparatus and the flow of the decoding process are the same as those described with reference to FIGS.

[0166] FIG. 16 is a block diagram showing an example of the main configuration of a decoding device when this method 1-2-1 is applied. Similar to the decoding device 350, the decoding device 400 shown in FIG. 16 is a device that decodes a bitstream in which the geometry of a point cloud (3D data) is encoded. For example, the decoding device 400 decodes bitstreams (guide surface bitstream, coordinate bitstream, feature bitstream, and point count bitstream) generated by encoding the point cloud (geometry) by the encoding device 300. However, the decoding device 400 outputs a decoded guide surface (mesh) as the decoding result.

[0167] Note that Fig. 16 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in the decoding device 400, there may be processing units that are not shown as blocks in Fig. 16, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 16.

[0168] As shown in FIG. 16, the decoding device 400 includes a decoding unit 411 and a fitting unit 412 .

[0169] The decoding unit 411 has the same configuration as the decoding device 350 (FIG. 11) and performs the same processing. That is, the explanation above using FIGS. 11 to 13 can be applied to the decoding unit 411. The decoding unit 411 decodes the point bit stream, coordinate bit stream, feature bit stream, and guide surface bit stream, and supplies the decoded point group C_X to the fitting unit 412. The decoding unit 411 also supplies the generated decoded guide surface (in this case, the guide surface is a mesh, so it is also referred to as a decoded guide mesh) to the fitting unit 412.

[0170] The fitting unit 412 acquires the decoded point group C_X and the decoded guide mesh, and fits the decoded guide mesh to the decoded point group C_X.

[0171] Any method may be used for this fitting. For example, an existing method for fitting a mesh to a point cloud may be applied. For example, the fitting unit 412 may perform mesh subdivision, which subdivides polygons, on the guide mesh, and mesh smoothing, which is a smoothing process. Then, the fitting unit 412 may perform fitting using Template Deformation for Point Cloud Fitting, as described in Non-Patent Document 3, on the guide mesh whose surface shape has been smoothed by increasing the number of triangles. This method is a process for fitting a template mesh to a point cloud. Here, the template mesh corresponds to the guide mesh, and the point cloud corresponds to the decoded point cloud.

[0172] However, as an initial condition, a small number of corresponding point pairs (information indicating which vertices of the template mesh correspond to which points in the point cloud) are required. Here, the nearest point in the point cloud can be searched for from each vertex of the template mesh, and the top few pairs with the shortest distance to the nearest point can be used as the initial corresponding point pairs. Note that since no local rotation is performed here, no rotation can be input and fitting can be performed with a rotation angle of 0. To achieve this, for example, all rotation matrices in the algorithm can be set to unit matrices.

[0173] The fitting unit 412 outputs the guide mesh after fitting to the outside of the decoding device 400 as a decoding result.

[0174] With this configuration, the decoding device 400 can output the guide mesh as the decoding result while suppressing a decrease in the quality of the decoding result.

[0175] <Flow of Decoding Process> An example of the flow of the decoding process executed by the decoding device 400 will be described with reference to the flowchart of FIG.

[0176] When the decoding process starts, in step S401, the decoding unit 411 executes the decoding process described with reference to FIGS. 8 and 9, decodes the bit stream, and generates a decoded point group and a guide mesh.

[0177] In step S402, the fitting unit 412 fits the guide mesh to the decoded point cloud. In step S403, the fitting unit 412 outputs the mesh resulting from the fitting as a decoding result to the outside of the decoding device 400. When the processing of step S403 ends, the decoding process ends.

[0178] By performing each process in this manner, the decoding device 400 can output the guide mesh as the decoding result while suppressing a decrease in the quality of the decoding result.

[0179] <Method 1-3> When the above-described method 1 is applied, the guide surface may be a triangle soup, as shown in the eighth row from the top of the table in FIG. 4 (method 1-3).

[0180] For example, in the first information processing device and the second information processing device, the guide surface may be a triangle soup.

[0181] In this case, the guide surface is stored as a type of mesh called a triangle soup and encoded using Trisoup coding. Trisoup coding is one option of G-PCC for encoding dense point clouds. The encoder estimates a set of triangles (=triangle soup) that approximates the input point cloud, as shown in Figure 18, and encodes the triangle soup. The triangle soup is a low-resolution octree and the triangular data in each octree node. The low-resolution octree is encoded using octree coding, and the intersections (also called vertices) between the triangles in each node and each edge are encoded. The decoder decodes the triangle soup and samples and outputs the point cloud from that surface.

[0182] <Encoding device> An example of the main configuration of an encoding device 300 when Methods 1-3 are applied is shown in Fig. 19. As shown in Fig. 19, in this case, the encoding device 300 has a Trisoup encoding unit 431 instead of the guide surface generation unit 311 and the guide surface encoding unit 312 in the example of Fig. 7. In addition, the encoding device 300 has a Trisoup decoding unit 432 instead of the guide surface decoding unit 313 in the example of Fig. 7.

[0183] The Trisoup encoding unit 431 performs Trisoup encoding on the input point group C_X, estimates Triangle soup internally, and generates encoded data (guide surface bit stream). The Trisoup encoding unit 431 supplies the generated guide surface bit stream to the Trisoup decoding unit 432. The Trisoup encoding unit 431 also outputs the guide surface bit stream to the outside of the encoding device 300. In other words, the Trisoup encoding unit 431 has the functions of both the guide surface generation unit 311 and the guide surface encoding unit 312.

[0184] The Trisoup decoding unit 432 acquires the guide surface bitstream supplied from the Trisoup encoding unit 431. The Trisoup decoding unit 432 Trisoup-decodes the guide surface bitstream, generates a triangle soup internally, and generates a point cloud sampled from that surface. However, the Trisoup decoding unit 432 does not use the point cloud, but rather uses the triangle soup generated internally during the decoding as the decoded guide surface, and supplies this to the guide surface feature derivation unit 315, the guide surface feature derivation unit 317, and the guide surface feature derivation unit 320.

[0185] With this configuration, the encoding device 300 can encode the guide surface as a triangle soup. That is, in this case as well, the encoding device 300 can reduce the amount of information to be encoded and suppress a decrease in encoding efficiency.

[0186] As described above, the Trisoup encoding unit 431 encodes the low-resolution octree. The coordinate encoding unit 324 performs octree encoding to encode the octree corresponding to the coordinate set C_Y. These two octrees are not necessarily identical, but if they are identical, the encoding device 300 may encode and output one of the octrees. In this case, the decoding device 350 may reuse one of the octrees obtained by decoding as the other octree. This can suppress an increase in the amount of code and further suppress a decrease in encoding efficiency.

[0187] Furthermore, even when one octree matches an octree obtained by downscaling the other octree, the encoding device 300 may encode and output only the high-resolution octree. In this case, the decoding device 350 may downscale the high-resolution octree obtained by decoding to restore the low-resolution octree. This can suppress an increase in the amount of code and further suppress a decrease in encoding efficiency.

[0188] <Flow of Encoding Process> An example of the flow of the encoding process executed by the encoding device 300 when Methods 1-3 are applied will be described with reference to the flowcharts of FIGS. 20 and 21.

[0189] When the encoding process starts, the Trisoup encoding unit 431 and the sparse tensor constructing unit 314 acquire an input point cloud C_X, which is a point cloud input to the encoding device 300, in step S431 of FIG.

[0190] In step S432, the Trisoup encoding unit 431 performs Trisoup encoding on the input point group C_X to generate a guide plane bit stream. The Trisoup encoding unit 431 outputs the guide plane bit stream to the outside of the encoding device 300.

[0191] In step S433, the Triangle Soup decoder 432 decodes the guide plane bitstream and generates Triangle Soup as a decoded guide plane.

[0192] The processes of steps S434 to S438 in Fig. 20 and steps S441 to S447 in Fig. 21 are executed in the same manner as the processes of steps S305 to S309 in Fig. 8 and steps S321 to S327 in Fig. 9. Then, when the process of step S447 in Fig. 21 ends, the encoding process ends.

[0193] By performing each process in this manner, the encoding device 300 can reduce the amount of information to be encoded, and can suppress a decrease in encoding efficiency.

[0194] <Regarding the Decoding Side> Note that the decoding device 350 when applying Methods 1-3 may be provided with a Trisoup decoding unit instead of the guide surface decoding unit 364 in Fig. 11. This Trisoup decoding unit is a processing unit similar to the Trisoup decoding unit 432 of the encoding device 300, and performs similar processing. In other words, this Trisoup decoding unit acquires the guide surface bit stream supplied to the decoding device 350, Trisoup-decodes the guide surface bit stream, generates a Triangle soup internally, sets it as a decoded guide surface, and supplies it to the guide surface feature derivation unit 366, the guide surface feature derivation unit 370, and the guide surface feature derivation unit 374. The other configurations are the same as those in Fig. 11.

[0195] The flow of the decoding process executed by the decoding device 350 in this case is basically the same as that described with reference to the flowcharts of Figures 12 and 13. However, in step S364, the Trisoup decoding unit Trisoup decodes the guide plane bit stream, and the obtained Triangle soup is used as the decoded guide plane.

[0196] <Method 1-4> When the above-mentioned method 1 is applied, the guide surface may be an implicit function that expresses a three-dimensional shape, as shown in the ninth row from the top of the table in FIG. 4 (method 1-4).

[0197] For example, in the first information processing device and the second information processing device, the guide surface may be an implicit function.

[0198] Implicit representation of a three-dimensional shape is a technique for representing a three-dimensional shape using a coordinate set where f(x, y, z) = 0, using a function f(x, y, z) of three-dimensional coordinate values ​​(x, y, z). For example, a unit sphere can be represented as f(x, y, z) = x^2 + y^2 + z^2 - 1. When representing a general three-dimensional shape, for example, the signed distance from a point (x, y, z) to a guide surface may be represented as f(x, y, z). Similarly, the unsigned distance or the truncated signed distance may be represented as f(x, y, z). In addition, the occupancy probability value at a point (x, y, z) may be represented as f(x, y, z). However, because the range of possible occupancy probability values ​​is [0, 1], in practice, an appropriately scaled value of the occupancy probability may be represented as f(x, y, z).

[0199] <Method 1-4-1> In addition, when the guide surface is an implicit function, the process of searching for the nearest point on the guide surface is generally not easy. Therefore, when the above-mentioned method 1-4 is applied, as shown in the 10th row from the top of the table in Fig. 4, the guide surface feature amount may be the output of the implicit function at the coordinates or the gradient vector of the output (Method 1-4-1).

[0200] For example, the implicit function value f(x_i, y_i, z_i) at the coordinates of the point c_i = (x_i, y_i, z_i) may be used as the guide surface feature g(c_i) (g(c_i) = f(x_i, y_i, z_i)). In addition, the value of the gradient vector ∇f at that coordinate value of the implicit function f or the Laplacian value Δf may also be applied together. Roughly speaking, when the implicit function f is a signed distance function, the gradient vector ∇f on the guide surface coincides with the normal vector. For example, a vector concatenating all of the above feature amounts may be applied as the guide surface feature amount (g(c_i) = [f(x_i, y_i, z_i), ∇f(x_i, y_i, z_i), Δf(x_i, y_i, z_i)]).

[0201] <Regarding the Encoding Device> The configuration of the encoding device 300 when applying this method 1-4-1 is the same as that described with reference to FIG. 7. However, the guide surface generation unit 311 estimates an implicit function as a guide surface from the input point group C_X. For example, the guide surface generation unit 311 generates a mesh from the input point group C_X, quantizes the three-dimensional space at an appropriate resolution (i.e., initializes a three-dimensional array), calculates the signed distance (or unsigned distance) from the center of each cell to the estimated mesh, and generates a three-dimensional array representing the signed distance. This three-dimensional array is supplied to the guide surface encoding unit 312 as a guide surface of the implicit function. This guide surface of the implicit function is also referred to as a guide implicit function.

[0202] The guide surface encoding unit 312 encodes the guide implicit function to generate a guide surface bitstream. Any encoding method may be used. For example, the guide surface encoding unit 312 may apply an existing implicit function encoding method. For example, the deep implicit volume compression described in Non-Patent Document 4 may be applied. This method can encode a truncated signed distance.

[0203] The guide surface decoding unit 313 decodes the guide surface bitstream and generates (restores) a guide implicit function. Therefore, the obtained decoded guide surface is also called a decoded guide implicit function. Any decoding method may be applied as long as it is compatible with the encoding method of the guide surface encoding unit 312.

[0204] The decoded guide implicit function is supplied to the guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, and the guide surface feature amount derivation unit 320. The guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, and the guide surface feature amount derivation unit 320 derive the guide surface feature amount using the decoded guide implicit function. The guide surface feature amount derivation unit 315, the guide surface feature amount derivation unit 317, and the guide surface feature amount derivation unit 320 may use the output of the implicit function at the coordinates of the point or the gradient vector of the output as the guide surface feature amount.

[0205] <Regarding the Encoding Process> The flow of the encoding process in this case is the same as that described with reference to the flowcharts in Figures 8 to 10. However, as described above in the description of the encoding device, in step S302, a guide implicit function is generated from the input point group. In step S303, the guide implicit function is coded to generate a guide plane bitstream. In step S304, the guide plane bitstream is decoded to generate a decoded guide implicit function.

[0206] In step S308, the output of the implicit function at the coordinates of the point and the gradient vector of the output are derived as the guide surface feature amount.

[0207] <Regarding the Decoding Device> The configuration of the decoding device 350 when applying this Method 1-4-1 is the same as that described with reference to Fig. 11. However, the guide surface decoding unit 364 decodes the acquired guide surface bitstream and generates (restores) a guide implicit function. Any decoding method may be applied as this decoding method as long as it corresponds to the encoding method of the guide surface encoding unit 312. The guide surface decoding unit 364 supplies the generated decoded guide implicit function to the guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370, and the guide surface feature amount derivation unit 374.

[0208] The guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370, and the guide surface feature amount derivation unit 374 derive the guide surface feature amount using the decoded guide implicit function. The guide surface feature amount derivation unit 366, the guide surface feature amount derivation unit 370, and the guide surface feature amount derivation unit 374 may use the output of the implicit function at the coordinates of the point or the gradient vector of the output as the guide surface feature amount.

[0209] <Regarding Decoding Process> The flow of the encoding process in this case is the same as that described with reference to the flowcharts in Figures 12 and 13. However, as described above in the description of the decoding device, in step S354, the guide surface bit stream is decoded to generate a decoded guide implicit function. In addition, in step S357, the output of the implicit function at the coordinates of the point and the gradient vector of the output are derived as guide surface feature amounts.

[0210] <Method 1-5> When the above-described method 1 is applied, the parameter set to be applied may be selected as shown in the eleventh row from the top of the table in FIG. 4 (method 1-5).

[0211] For example, the first information processing device may further include a parameter set selection unit that selects a parameter set to be applied to encoding the point cloud. Also, the first information processing device may further include a guide surface generation unit that generates a guide surface using the point cloud, and a guide surface encoding unit that encodes the guide surface by applying the selected parameter set and generates encoded data of the guide surface.

[0212] In guide surface coding, there may be parameter sets that affect the coding efficiency and decoding error of the guide surface. For example, when using mesh decimation and Draco coding, examples of coding parameter sets include the target number of triangles in decimation (the smaller the number, the better the coding efficiency but the larger the error) and the quantization parameter (QP (Quantization Parameter)) in the Draco encoder (the smaller the number, the better the coding efficiency but the larger the error). The encoder may select and apply the parameter set that provides the highest coding efficiency for the point cloud from a candidate set of parameter sets (also referred to as guide surface coding parameter sets) that are applied to coding of such guide surfaces.

[0213] In this way, it is possible to suppress a decrease in coding efficiency due to the applied parameter set.

[0214] <Encoding device> Fig. 22 is a block diagram showing an example of the main configuration of an encoding device when methods 1 to 5 are applied. The encoding device 500 shown in Fig. 22 is a device that encodes the geometry of a point cloud (3D data), similar to the encoding device 300. The encoding device 500 encodes the geometry by applying the above-mentioned methods 1 to 5.

[0215] Note that Fig. 22 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in encoding device 500, there may be processing units that are not shown as blocks in Fig. 22, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 22.

[0216] As shown in FIG. 22, the encoding device 500 includes a parameter set supplying unit 511 , an encoding unit 512 , a decoding unit 513 , and a parameter set selecting unit 514 .

[0217] The parameter set supply unit 511 selects one parameter set to be processed from the candidate set of guide surface encoding parameter sets, and supplies it to the encoding unit 512 and parameter set selection unit 514 .

[0218] The encoding unit 512 acquires an input point group C_X input to the encoding device 500. The encoding unit 512 also acquires a guide surface encoding parameter set supplied from the parameter set supply unit 511. The encoding unit 512 then applies the guide surface encoding parameter set to encode the input point group C_X and generate a bitstream. The encoding unit 512 may have a configuration similar to that of the encoding device 300 (FIG. 7), for example, and may perform similar processing. That is, the encoding unit 512 encodes the input point group C_X and generates a point number bitstream, a coordinate bitstream, a feature bitstream, and a guide surface bitstream. At this time, the encoding unit 512 applies the guide surface encoding parameter set supplied from the parameter set supply unit 511 in the guide surface encoding unit 312. That is, the guide surface encoding unit 312 of the encoding unit 512 encodes the guide surface using the guide surface encoding parameter set supplied from the parameter set supply unit 511 and generates a guide surface bitstream. The encoding unit 512 supplies the bit streams thus generated (score bit stream, coordinate bit stream, feature bit stream, and guide surface bit stream) to the decoding unit 513 and parameter set selection unit 514 .

[0219] The decoding unit 513 obtains the bit stream supplied from the encoding unit 512. The decoding unit 513 decodes the bit stream and generates (reconstructs) a decoded point group C_X. The decoding unit 513 may have a configuration similar to that of the decoding device 350 ( FIG. 11 ), for example, and may perform similar processing. That is, the decoding unit 513 decodes the point bit stream, coordinate bit stream, feature bit stream, and guide surface bit stream, and generates a decoded point group C_X. The decoding unit 513 supplies the generated decoded point group C_X to the parameter set selection unit 514.

[0220] The parameter set selection unit 514 acquires a guide surface encoding parameter set supplied from the parameter set supply unit 511. The parameter set selection unit 514 acquires an input point group C_X input to the encoding device 500. The parameter set selection unit 514 acquires bit streams (e.g., a point bit stream, a coordinate bit stream, a feature bit stream, and a guide surface bit stream) supplied from the encoding unit 512. The parameter set selection unit 514 acquires a decoded point group C_X supplied from the decoding unit 513.

[0221] Based on these, the parameter set selection unit 514 selects a guide plane coding parameter set that maximizes the coding efficiency of the input point group C_X. The parameter set selection unit 514 selects a bitstream corresponding to the selected guide plane coding parameter set and outputs it to the outside of the encoding device 500.

[0222] In selecting this parameter set, the parameter set selection unit 514 may select, for example, a parameter set that minimizes the value C = R + λ·D of the RD cost function. Here, R indicates the total size of the bitstream. D indicates the error between the decoded geometry and the input geometry. The coefficient λ is a parameter that determines whether to prioritize the bit size or the decoding error. As an index of the error between point clouds, for example, the D1 metric, which is an index that measures the distance between point clouds, or the D2 metric, which also takes normal direction into consideration, may be applied.

[0223] By selecting the guide plane coding parameter set to be applied so as to optimize it using the RD cost in this way, the coding device 500 can suppress a decrease in coding efficiency.

[0224] <Flow of Encoding Process> An example of the flow of encoding process executed by the encoding device 500 will be described with reference to the flowchart of FIG.

[0225] When the encoding process starts, in step S501, the parameter set supply unit 511 sets the coefficient λ of the RD cost function and a candidate set of guide surface encoding parameter sets.

[0226] In step S502, the parameter set supplying unit 511 selects the i-th guide surface encoding parameter set from the set candidate set. That is, the parameter set supplying unit 511 selects one unprocessed guide surface encoding parameter set.

[0227] In step S503, the encoding unit 512 executes the encoding process described with reference to the flowcharts of Figures 8 and 9, for example. At this time, the encoding unit 512 encodes the guide surface by applying the guide surface encoding parameters selected in step S502, and generates a guide surface bitstream. The encoding unit 512 then encodes the point cloud using the decoded guide surface generated by decoding the guide surface bitstream, and generates a point number bitstream, a coordinate bitstream, and a feature bitstream. When the applied guide surface encoding parameter set changes, the size of the guide surface bitstream and the decoded guide surface change. That is, the overall bitstream size and the decoded point cloud C_X also change. Therefore, the encoding unit 512 derives not only the guide surface bitstream, but all bitstreams.

[0228] In step S504, the parameter set selection unit 514 calculates the total size R_i of all the bit streams (the point number bit stream, the coordinate bit stream, the feature amount bit stream, and the guide surface bit stream) generated in step S503.

[0229] In step S505, the decoding unit 513 performs the decoding process described with reference to the flowcharts of Figures 12 and 13, decodes the bit streams (point bit stream, coordinate bit stream, feature bit stream, and guide surface bit stream) generated in step S503, and generates a decoded point group C_X.

[0230] In step S506, the parameter set selection unit 514 derives the error D_i between the decoded geometry (the decoded point group C_X generated in step S505) and the input geometry (the input point group C_X).

[0231] In step S507, the parameter set selection unit 514 derives the value of the RD cost function C_i = R_i + λ·D_i using the coefficient λ set in step S501, the bitstream size R_i derived in step S504, and the error D_i derived in step S506.

[0232] In step S508, the parameter set selection unit 514 determines whether all parameter sets have been processed. If it is determined that an unprocessed parameter set exists, the process returns to step S502, and the subsequent processes are executed. That is, the processes of steps S502 to S508 are executed for each parameter set, and the value C_i of the RD cost function is derived.

[0233] If it is determined in step S508 that the processing has been performed for all parameter sets, the processing proceeds to step S509.

[0234] In step S509, the parameter set selection unit 514 compares the values ​​C_i of the RD cost function corresponding to each parameter set derived as described above, and selects the parameter set for which the value C_i of the RD cost function is minimum.

[0235] In step S510, the parameter set selection unit 514 outputs a bitstream corresponding to the selected parameter set to the outside of the encoding device 500.

[0236] When the process of step S510 is completed, the decoding process ends.

[0237] By performing each process as described above, the encoding device 500 can select a guide surface encoding parameter set that optimizes the RD cost and apply it to the encoding of the input point group C_X (for the encoding of the guide surface). Therefore, the encoding device 500 can suppress a decrease in encoding efficiency.

[0238] <Regarding Decoding> Note that even when Methods 1-5 are applied, the decoding device 350 has the same configuration and performs the same decoding process as in Fig. 11. In other words, the explanations given with reference to Figs. 11 to 13 can also be applied to this case.

[0239] <Method 1-6> Furthermore, when the above-mentioned Method 1 is applied, attributes may be transmitted (Method 1-6) as shown in the bottom row of the table in Fig. 4. That is, although the transmission (encoding / decoding) of geometry has been described above, not only geometry but also attributes may be transmitted (encoding / decoding).

[0240] For example, in the first information processing device, the guide surface feature derivation unit may derive guide surface features using a coordinate set, a feature set including attribute information, and a guide surface including texture. Then, the calculation unit may perform calculation using the guide surface features to derive a new coordinate set and a feature set including attributes. Then, the coordinate encoding unit may encode the coordinate set to generate a coordinate bitstream. Furthermore, the feature encoding unit may encode the feature set including attributes to generate a feature bitstream. Furthermore, the guide surface encoding unit may encode the guide surface including texture to generate a guide surface bitstream. Furthermore, the score encoding unit may encode information indicating the score of each scale to generate a score bitstream.

[0241] For example, in the second information processing device, the coordinate decoding unit may decode the coordinate bit stream to generate a coordinate set. Also, the feature decoding unit may decode the feature bit stream to generate a feature set including attribute information. Also, the guide surface decoding unit may decode the guide surface bit stream to generate a decoded guide surface including texture. Also, the guide surface feature derivation unit may derive guide surface features using the coordinate set, the feature set including attributes, and the decoded guide surface including texture. Then, the calculation unit may perform calculation using the derived guide surface features to derive a new coordinate set and a feature set including attributes.

[0242] Consider the case where each point in the input point cloud has an attribute attached to it. The attribute can be any information. For example, it can include color information or reflectance intensity. Such point cloud attributes can be transmitted (encoded and decoded) in the same framework as geometry.

[0243] <Encoding Device> Fig. 24 is a block diagram showing an example of the main configuration of an encoding device when methods 1-6 are applied. The encoding device 600 shown in Fig. 24 is a device that encodes a point cloud (3D data), similar to the encoding device 300. The encoding device 600 performs encoding by applying the above-mentioned methods 1-6. In other words, the encoding device 600 encodes attributes together with geometry.

[0244] Note that Fig. 24 shows the main processing units, data flows, etc., and is not necessarily all that is shown in Fig. 24. In other words, in the encoding device 600, there may be processing units that are not shown as blocks in Fig. 24, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 24.

[0245] As shown in FIG. 24, the encoding device 600 has a guide surface generation unit 611, a guide surface encoding unit 612, a guide surface decoding unit 613, a sparse tensor construction unit 614, a guide surface feature derivation unit 615, a CNN 616 (CNN_E1), a guide surface feature derivation unit 617, a CNN 618 (CNN_E2), a CNN 619 (CNN_E3), a guide surface feature derivation unit 620, a CNN 621 (CNN_E4), a CNN 622 (CNN_E5), a CNN 623 (CNN_E6), a coordinate encoding unit 624, a feature encoding unit 625, and a score encoding unit 626.

[0246] The input point group input to the encoding device 600 is composed of geometry and attributes. That is, each point has coordinate values ​​(x, y, z) and attribute values ​​(for example, RGB values ​​in the case of color information).

[0247] The guide surface generation unit 611 is a processing unit similar to the guide surface generation unit 311 (FIG. 7) and performs similar processing. However, the guide surface generation unit 611 acquires an input point group composed of geometry and attributes, and generates a guide surface including the attributes. For example, if the guide surface is composed of a mesh, the attributes are formed as a texture. Therefore, in this case, the guide surface generation unit 611 generates a guide surface including a texture.

[0248] One way to store textures is to store the texture data as an image and add texture coordinate values ​​(values ​​that indicate where that vertex is located in the texture image) (also called UV coordinates) to each vertex of the mesh.

[0249] In addition to generating a guide surface (mesh) from the input point cloud, the guide surface generator 611 must also generate a texture from the input attributes. For each point in the input point cloud, the guide surface generator 611 searches for the nearest point on the guide mesh and copies the attributes of that point to the texel (one pixel in texture data is called a texel) corresponding to that nearest point. However, after the above processing, there is a possibility that an empty texel will exist. In that case, the guide surface generator 611 may use neighboring values ​​to fill in the empty texel value.

[0250] The guide surface encoding unit 612 is a processing unit similar to the guide surface encoding unit 312 ( FIG. 7 ) and performs similar processing. However, the guide surface encoding unit 612 encodes the guide surface, including the texture, supplied from the guide surface generation unit 611 to generate a guide surface bitstream. That is, the guide surface encoding unit 612 needs to encode texture data in addition to encoding mesh data. That is, the guide surface encoding unit 612 encodes not only geometry but also the texture coordinates and texture image of each vertex. Any encoding method may be used to encode the texture coordinates of each vertex. For example, Draco encoding may be used (the Draco encoder can encode texture coordinates in addition to meshes). Furthermore, existing image compression methods (e.g., JPEG (Joint Photographic Experts Group) or PNG (Portable Network Graphics)) may be used to encode the texture image. Therefore, the guide surface bitstream in this case includes not only a mesh-encoded bitstream but also a texture-encoded bitstream.

[0251] The guide surface decoding unit 613 is a processing unit similar to the guide surface decoding unit 313 (FIG. 7) and performs similar processing. However, the guide surface decoding unit 613 decodes the guide surface bitstream and generates a decoded guide surface including texture. In other words, the guide surface decoding unit 613 decodes the bitstream to generate not only geometry but also texture coordinates and texture images for each vertex. Any decoding method may be applied to generate texture coordinates for each vertex. For example, Draco decoding may be applied (a Draco decoder can generate texture coordinates in addition to meshes). Furthermore, an existing image decoding method (e.g., JPEG, PNG, etc.) may be applied to generate texture images.

[0252] The sparse tensor constructor 614 is a processing unit similar to the sparse tensor constructor 314 ( FIG. 7 ) and performs similar processing. However, the sparse tensor constructor 614 constructs an input sparse tensor X = {C_X, F_X} from an input point cloud composed of geometry and attributes. In this case, to input attributes, the feature set F_X is a vector that lists the attributes of each coordinate. In other words, each feature f_i ∈ F_X is the attribute value (e.g., RGB value) of each point. Furthermore, since the dimension of the feature vector increases, the input dimension of the CNN 616 (CNN_E1) ​​must also increase accordingly.

[0253] The guide surface feature amount derivation unit 615 is a processing unit similar to the guide surface feature amount derivation unit 315 (FIG. 7) and performs similar processing. However, the guide surface feature amount derivation unit 615 also includes feature amounts extracted from the texture in the guide surface feature amount in order to use attribute information on the texture for encoding and decoding. Specifically, the guide surface feature amount derivation unit 615 concatenates the texture value a_i (for example, RGB value in the case of color information) at the nearest point p_i on the guide mesh. In other words, the vectors are concatenated to obtain g(c_i) = [p_i - c_i, a_i].

[0254] CNN616 (CNN_E1) ​​is a processing unit similar to CNN316 (CNN_E1) ​​(Figure 7) and performs similar processing. However, in this case, in the sparse tensor processed by CNN616 (CNN_E1), F_X is changed to a vector that lists the attributes of each coordinate. In other words, each feature f_i ∈ F_X is the attribute value of each point (e.g., RGB value). In addition, since the dimension of the feature vector increases, the input dimension of CNN616 (CNN_E1) ​​must also increase accordingly.

[0255] The guide surface feature amount derivation unit 617 is a processing unit similar to the guide surface feature amount derivation unit 317 (FIG. 7) and performs similar processing. However, like the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617 also uses attribute information on the texture for encoding and decoding, so that the guide surface feature amount includes feature amounts extracted from the texture.

[0256] CNN618 (CNN_E2) is a processing unit similar to CNN318 (CNN_E2) (FIG. 7) and performs similar processing. However, in this case, as with CNN616 (CNN_E1), it is necessary to increase the input dimension of CNN618 (CNN_E2).

[0257] The CNN 619 (CNN_E3) is a processing unit similar to the CNN 319 (CNN_E3) (FIG. 7) and performs similar processing.

[0258] The guide surface feature amount derivation unit 620 is a processing unit similar to the guide surface feature amount derivation unit 320 ( FIG. 7 ) and performs similar processing. However, like the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 620 also uses attribute information on the texture for encoding and decoding, so that the guide surface feature amount includes feature amounts extracted from the texture.

[0259] CNN621 (CNN_E4) is a processing unit similar to CNN321 (CNN_E4) (FIG. 7) and performs similar processing. However, in this case, as with CNN616 (CNN_E1), it is necessary to increase the input dimension of CNN621 (CNN_E4).

[0260] The CNN 622 (CNN_E5) is a processing unit similar to the CNN 322 (CNN_E5) (FIG. 7) and performs similar processing.

[0261] The CNN 623 (CNN_E6) is a processing unit similar to the CNN 323 (CNN_E6) (FIG. 7) and performs similar processing.

[0262] The coordinate encoding unit 624 is a processing unit similar to the coordinate encoding unit 324 ( FIG. 7 ) and performs similar processing. The feature encoding unit 625 is a processing unit similar to the feature encoding unit 325 ( FIG. 7 ) and performs similar processing. In this case, however, the feature set encoded by the feature encoding unit 625 includes not only geometry information but also attribute information. The score encoding unit 626 is a processing unit similar to the score encoding unit 326 ( FIG. 7 ) and performs similar processing.

[0263] With this configuration, the encoding device 600 can encode not only geometry but also attributes.

[0264] <Flow of Encoding Process> An example of the flow of the encoding process executed by the encoding device 600 will be described with reference to the flowcharts of FIGS. 25 and 26.

[0265] In this case, when the encoding process starts, the guide surface generation unit 611 and the sparse tensor construction unit 614 acquire the input point group C_X, which is the point cloud input to the encoding device 600, and its attributes in step S601 of FIG.

[0266] In step S602, the guide surface generation unit 611 generates a guide surface and texture data from the input point group C_X and attributes.

[0267] In step S603, the guide surface coding unit 612 codes the generated guide surface and texture data to generate a guide surface bit stream, and outputs the guide surface bit stream to the outside of the coding device 600.

[0268] In step S604, the guide surface decoding unit 613 decodes the guide surface bit stream and generates (restores) a decoded guide surface and decoded texture data.

[0269] In step S605, the sparse tensor construction unit 614 generates an input sparse tensor X={C_X, F_X} (scaled 1) from the input point group C_X and attributes.

[0270] In step S606, the guide surface feature amount derivation unit 615 sets {C^in, F^in} ← (C_X, F_X). That is, the guide surface feature amount derivation unit 615 sets the input sparse tensor X={C_X, F_X} as the sparse tensor (C^in, F^in) to be processed.

[0271] In step S607, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 determines whether or not to input the guide surface feature amount to the next 3D-CNN. If it is determined that the guide surface feature amount is to be input to the next 3D-CNN, the process proceeds to step S608.

[0272] In step S608, the guide surface feature derivation unit 615, the guide surface feature derivation unit 617, or the guide surface feature derivation unit 620 executes a guide surface feature set derivation process and derives a guide surface feature set g(C^in) using the sparse tensor to be processed, the decoded guide surface, and the decoded texture data.

[0273] In step S609, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 sets F^in ← [F^in, g(C^in)]. That is, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 sets the concatenation of the feature amount set to be processed and the guide surface feature amount set g(C^in) as the feature amount set to be processed by the next 3D-CNN.

[0274] When the processing of step S609 is completed, the processing proceeds to Fig. 26. Also, if it is determined in step S607 of Fig. 25 that the guide surface feature amount will not be input to the next 3D-CNN, the processing proceeds to Fig. 26. That is, if there is no guide surface feature amount derivation unit immediately before the 3D-CNN, the processing of steps S608 and S609 is omitted. For example, in the case where the encoding device 600 has the configuration example shown in Fig. 24, when the next 3D-CNN is CNN616 (CNN_E1), CNN618 (CNN_E2), or CNN621 (CNN_E4), the processing of steps S608 and S609 is executed, and when the next 3D-CNN is CNN619 (CNN_E3), CNN622 (CNN_E5), or CNN623 (CNN_E6), the processing of steps S608 and S609 is omitted.

[0275] In step S621 of Figure 26, CNN616 (CNN_E1), CNN618 (CNN_E2), CNN619 (CNN_E3), CNN621 (CNN_E4), CNN622 (CNN_E5), or CNN623 (CNN_E6) performs 3D-CNN operations on the sparse tensor {C^in, F^in} to be processed, and derives the sparse tensor {C^out, F^out} as the operation result.

[0276] In step S622, CNN623 (CNN_E6) determines whether processing has been performed for all 3D-CNNs. If it is determined that processing has not been performed for all 3D-CNNs (i.e., that there are unprocessed 3D-CNNs (at least CNN623 (CNN_E6) is unprocessed)), the process proceeds to step S623.

[0277] In step S623, CNN616 (CNN_E1), CNN618 (CNN_E2), CNN619 (CNN_E3), CNN621 (CNN_E4), or CNN622 (CNN_E5) sets {C^in, F^in} ← {C^out, F^out}. In other words, CNN616 (CNN_E1), CNN618 (CNN_E2), CNN619 (CNN_E3), CNN621 (CNN_E4), or CNN622 (CNN_E5) sets the sparse tensor {C^out, F^out} of the calculation result derived by itself as the sparse tensor {C^in, F^in} to be processed by the next processing unit (guide surface feature derivation unit or 3D-CNN).

[0278] When the process of step S623 ends, the process returns to step S607 in Fig. 25, and the subsequent processes are executed. That is, the processes of steps S607 to S609 in Fig. 25 and the processes of steps S621 to S623 in Fig. 26 are executed for each 3D-CNN (if a guide surface feature amount derivation unit exists immediately before the 3D-CNN, the guide surface feature amount derivation unit is also included).

[0279] Then, in step S622 of FIG. 26, if it is determined that the processing has been performed for all 3D-CNNs (i.e., the processing has been performed for CNN623 (CNN_E6)), the processing proceeds to step S624.

[0280] In step S624, the CNN 623 (CNN_E6) sets {C_Y, F_Y} ← {C^out, F^out}. That is, the last CNN 623 (CNN_E6) sets the sparse tensor {C^out, F^out}, which is the calculation result derived by itself, as the output sparse tensor {C_Y, F_Y}.

[0281] In step S625, the coordinate encoding unit 624 encodes the coordinate set C_Y of the output sparse tensor {C_Y, F_Y} to generate a coordinate bitstream, and outputs the coordinate bitstream to the outside of the encoding device 600.

[0282] In step S626, the feature encoding unit 625 encodes the feature set F_Y of the output sparse tensor {C_Y, F_Y} to generate a feature bitstream, and outputs the feature bitstream to the outside of the encoding device 600.

[0283] In step S627, the score for each scale is coded to generate a score bitstream, which is then output to the outside of the coding device 600.

[0284] When the process of step S627 is completed, the encoding process ends.

[0285] <Flow of Guide Surface Feature Amount Set Deriving Process> Next, an example of the flow of guide surface feature amount set deriving process executed in step S608 of FIG. 25 will be described with reference to the flowchart of FIG.

[0286] When the guide surface feature amount set derivation process is started, in step S631, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 extracts coordinates c_i ∈ C^in. That is, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 selects one coordinate c_i from the coordinate set C^in to be processed, and sets it as the processing target.

[0287] In step S632, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 searches for the nearest point p_i on the decoded guide surface as viewed from the selected coordinate c_i.

[0288] In step S633, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 returns a vector g(c_i) = [p_i - c_i, a_i] in which the vector and attribute value up to the searched nearest point are arranged. That is, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 derives the guide surface feature amount g(c_i) based on the positional relationship between the guide surface and the point (p_i - c_i in this example) and the attribute value (a_i in this example).

[0289] In step S634, the guide surface feature amount derivation unit 615, the guide surface feature amount derivation unit 617, or the guide surface feature amount derivation unit 620 determines whether or not processing has been performed for all coordinates c_i ∈ C^in. If it is determined that an unprocessed coordinate c_i exists, the processing returns to step S631, and the subsequent processing is executed. That is, the processing of steps S631 to S634 is executed for each coordinate value c_i of the coordinate set C^in to be processed.

[0290] Then, in step S634, if it is determined that the process has been performed for all coordinate values ​​c_i, the guide surface feature amount set derivation process ends, and the process returns to FIG.

[0291] By performing each process in this manner, the encoding device 600 can encode not only geometry but also attributes.

[0292] <Decoding Device> Fig. 28 is a block diagram showing an example of the main configuration of a decoding device when methods 1-6 are applied. Similar to the decoding device 350, the decoding device 650 shown in Fig. 28 is a device that decodes a bitstream in which the geometry of a point cloud (3D data) is encoded. For example, the decoding device 650 decodes bitstreams (guide surface bitstream, coordinate bitstream, feature bitstream, and point count bitstream) generated by encoding the point cloud (geometry and attributes) by the encoding device 600, and generates (restores) the point cloud (geometry and attributes).

[0293] Note that Fig. 28 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, in the decoding device 650, there may be processing units that are not shown as blocks in Fig. 28, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 28.

[0294] As shown in FIG. 28, the decoding device 650 has a score decoding unit 661, a coordinate decoding unit 662, a feature decoding unit 663, a guide surface decoding unit 664, a CNN 665 (CNN_D1), a guide surface feature derivation unit 666, a CNN 667 (CNN_D2), an occupancy state classification unit 668, a CNN 669 (CNN_D3), a guide surface feature derivation unit 670, a CNN 671 (CNN_D4), an occupancy state classification unit 672, a CNN 673 (CNN_D5), a guide surface feature derivation unit 674, a CNN 675 (CNN_D6), and an occupancy state classification unit 676.

[0295] The score decoding unit 661 is a processing unit similar to the score decoding unit 361 ( FIG. 11 ) and performs similar processing. For example, the score decoding unit 661 may obtain and decode the score bitstream input to the decoding device 650, and generate (restore) information indicating the scores of the coordinate sets at each scale (N_X, N_X', N_X''). The score decoding unit 661 may supply N_X to the occupation state classification unit 676, N_X' to the occupation state classification unit 672, and N_X'' to the occupation state classification unit 668.

[0296] The coordinate decoding unit 662 is a processing unit similar to the coordinate decoding unit 362 ( FIG. 11 ) and performs similar processing. For example, the coordinate decoding unit 662 may acquire a coordinate bitstream input to the decoding device 650, decode it, and generate (restore) a coordinate set C_Y. The coordinate decoding unit 662 may supply the generated coordinate set C_Y to the CNN 665 (CNN_D1).

[0297] The feature decoding unit 663 is a processing unit similar to the feature decoding unit 363 ( FIG. 11 ) and performs similar processing. For example, the feature decoding unit 663 may acquire a feature bitstream input to the decoding device 650, perform entropy decoding, and generate (restore) a feature set F_Y. The feature decoding unit 663 may supply the generated feature set F_Y to the CNN 665 (CNN_D1).

[0298] The guide surface decoding unit 664 is a processing unit similar to the guide surface decoding unit 613 (FIG. 24), and performs processing related to the decoding of the guide surface bitstream. For example, the guide surface decoding unit 664 acquires the guide surface bitstream input to the decoding device 650, decodes it, and generates a decoded guide surface including texture. That is, the guide surface decoding unit 664 decodes the bitstream to not only generate geometry, but also generate texture coordinates and texture images for each vertex. This decoding method is arbitrary, as in the case of the guide surface decoding unit 613 (FIG. 24). The guide surface decoding unit 664 may supply the generated decoded guide surface (including texture) to the guide surface feature derivation unit 666, the guide surface feature derivation unit 670, and the guide surface feature derivation unit 674.

[0299] CNN665 (CNN_D1) is a processing unit similar to CNN365 (CNN_D1) (FIG. 11) and performs similar processing. However, in this case, in the sparse tensor processed by CNN665 (CNN_D1), the feature set includes not only geometry information but also attribute information.

[0300] The guide surface feature amount derivation unit 666 is a processing unit similar to the guide surface feature amount derivation unit 366 (FIG. 11) and performs similar processing. However, the guide surface feature amount derivation unit 666 also includes feature amounts extracted from the texture in the guide surface feature amount in order to use attribute information on the texture for encoding and decoding. Specifically, the guide surface feature amount derivation unit 666 concatenates the texture value a_i (for example, RGB value in the case of color information) at the nearest point p_i on the guide mesh. In other words, the vectors are concatenated to obtain g(c_i) = [p_i - c_i, a_i].

[0301] CNN667 (CNN_D2) is a processing unit similar to CNN367 (CNN_D2) (FIG. 11) and performs similar processing. However, in this case, the number of dimensions of the guide surface feature amount increases by the number of dimensions of the attribute value, so it is necessary to increase the input dimensions of CNN667 (CNN_D2).

[0302] The occupancy state classification unit 668 is a processing unit similar to the occupancy state classification unit 368 ( FIG. 11 ) and performs similar processing. For example, the occupancy state classification unit 668 may predict an occupancy probability value using the sparse tensor to be processed, extract the top N_X″ coordinate values ​​of the occupancy probability value, delete the other coordinate values, and generate a sparse tensor X″ = {C_X'', F_X″} as the calculation result. The occupancy state classification unit 668 may supply the sparse tensor X″ = {C_X'', F_X″} as the calculation result to the CNN 669 (CNN_D3).

[0303] The CNN669 (CNN_D3) is a processing unit similar to the CNN369 (CNN_D3) (FIG. 11) and performs similar processing.

[0304] The guide surface feature amount derivation unit 670 is a processing unit similar to the guide surface feature amount derivation unit 370 ( FIG. 11 ) and performs similar processing. However, like the guide surface feature amount derivation unit 666, the guide surface feature amount derivation unit 670 also uses attribute information on the texture for encoding and decoding, and therefore includes features extracted from the texture in the guide surface feature amount.

[0305] CNN671 (CNN_D4) is a processing unit similar to CNN371 (CNN_D4) (FIG. 11) and performs similar processing. However, in this case, the number of dimensions of the guide surface feature amount increases by the number of dimensions of the attribute value, so it is necessary to increase the input dimensions of CNN671 (CNN_D4).

[0306] The occupancy state classification unit 672 is a processing unit similar to the occupancy state classification unit 372 ( FIG. 11 ) and performs similar processing. For example, the occupancy state classification unit 672 may predict an occupancy probability value using the sparse tensor to be processed, extract the top N_X' coordinate values ​​of the occupancy probability value, delete the other coordinate values, and generate a sparse tensor X' = {C_X', F_X'} as the calculation result. The occupancy state classification unit 672 may supply the sparse tensor X' = {C_X', F_X'} as the calculation result to the CNN 673 (CNN_D5).

[0307] The CNN673 (CNN_D5) is a processing unit similar to the CNN373 (CNN_D5) (FIG. 11) and performs similar processing.

[0308] The guide surface feature amount derivation unit 674 is a processing unit similar to the guide surface feature amount derivation unit 374 ( FIG. 11 ) and performs similar processing. However, like the guide surface feature amount derivation unit 666, the guide surface feature amount derivation unit 674 also uses attribute information on the texture for encoding and decoding, and therefore includes features extracted from the texture in the guide surface feature amount.

[0309] CNN675 (CNN_D6) is a processing unit similar to CNN375 (CNN_D6) (Figure 11) and performs similar processing. However, in this case, the number of dimensions of the guide surface feature increases by the number of dimensions of the attribute value, so the input dimensions of CNN675 (CNN_D6) must be increased. Furthermore, CNN675 (CNN_D6) must increase the output dimensions by the number of attribute values.

[0310] The occupancy state classification unit 676 is a processing unit similar to the occupancy state classification unit 376 ( FIG. 11 ) and performs similar processing. For example, the occupancy state classification unit 676 may predict an occupancy probability value using the sparse tensor to be processed, extract the top N_X coordinate values ​​of the occupancy probability value, delete the other coordinate values, and generate a sparse tensor X = {C_X, F_X} as the calculation result. The occupancy state classification unit 676 may output the decoded point group C_X as the calculation result to outside the decoding device 650. The occupancy state classification unit 676 may also output attributes of the calculation result to outside the decoding device 650.

[0311] For example, if the attribute is an RGB value, each feature vector f_i ∈ F_X in the feature vector set F_X output by the CNN 675 (CNN_D6) is expanded three-dimensionally to become an RGB value. The occupancy state classification unit 676 sets the attribute value as the attribute of each point in the decoded point group C_X and outputs it.

[0312] With this configuration, the decoding device 650 can decode not only geometry but also attributes.

[0313] <Flow of Decoding Process> An example of the flow of the decoding process executed by the decoding device 650 will be described with reference to the flowcharts of FIGS. 29 and 30.

[0314] In this case, when the decoding process starts, the score decoding unit 661 decodes the score bit stream in step S651 of FIG. 29 to generate scores (N_X, N_X', N_X'') for each scale.

[0315] In step S652, the coordinate decoding unit 662 decodes the coordinate bit stream to generate a coordinate set C_Y.

[0316] In step S653, the feature decoding unit 663 decodes the feature bit stream to generate a feature set F_Y.

[0317] In step S654, the guide surface decoding unit 664 decodes the guide surface bitstream to generate a decoded guide surface and decoded texture data.

[0318] In step S655, the CNN 665 (CNN_D1) sets {C^in, F^in} ← {C_Y, F_Y}. That is, the CNN 665 (CNN_D1) sets the sparse tensor Y = {C_Y, F_Y} as the sparse tensor {C^in, F^in} to be processed.

[0319] In step S656, CNN665 (CNN_D1), CNN667 (CNN_D2), CNN669 (CNN_D3), CNN671 (CNN_D4), CNN673 (CNN_D5), or CNN675 (CNN_D6) determines whether to input guide surface features to its own 3D-CNN. If it is determined that the guide surface features are to be input, the process proceeds to step S657. In other words, if the CNN to be processed is CNN667 (CNN_D2), CNN671 (CNN_D4), or CNN675 (CNN_D6), the process proceeds to step S657.

[0320] In step S657, the guide surface feature amount derivation unit 666, the guide surface feature amount derivation unit 670, or the guide surface feature amount derivation unit 674 executes a guide surface feature amount set derivation process to derive a guide surface feature amount set g(C^in). This guide surface feature amount set derivation process is executed in the same manner as described with reference to the flowchart in Fig. 27. In other words, the guide surface feature amount derivation unit 666, the guide surface feature amount derivation unit 670, or the guide surface feature amount derivation unit 674 derives the guide surface feature amount set g(C^in) using the sparse tensor to be processed, the decoded guide surface, and the decoded texture data.

[0321] In step S658, the guide surface feature amount derivation unit 666, the guide surface feature amount derivation unit 670, or the guide surface feature amount derivation unit 674 sets F^in ← [F^in, g(C^in)]. That is, the guide surface feature amount derivation unit 666, the guide surface feature amount derivation unit 670, or the guide surface feature amount derivation unit 674 sets the concatenation of the feature amount set to be processed and the guide surface feature amount set g(C^in) as the feature amount set to be processed by the next 3D-CNN.

[0322] When the processing of step S658 is completed, the processing proceeds to Fig. 30. Also, in step S656 of Fig. 29, if it is determined that the guide surface feature amount is not to be input to its own 3D-CNN, the processing of steps S657 and S658 is omitted, and the processing proceeds to Fig. 30. In other words, if the CNN to be processed is CNN665 (CNN_D1), CNN669 (CNN_D3), or CNN673 (CNN_D5), the processing of steps S657 and S658 is omitted, and the processing proceeds to Fig. 30.

[0323] In step S671 of Figure 30, CNN665 (CNN_D1), CNN667 (CNN_D2), CNN669 (CNN_D3), CNN671 (CNN_D4), CNN673 (CNN_D5), or CNN675 (CNN_D6) performs 3D-CNN operations on the sparse tensor {C^in, F^in} to be processed, and derives the sparse tensor {C^out, F^out} as the operation result.

[0324] In step S672, the occupancy status classification unit 668, the occupancy status classification unit 672, or the occupancy status classification unit 676 determines whether to classify the occupancy status. If it is determined that the occupancy status is to be classified, the process proceeds to step S673. In other words, if the process of step S671 is performed by CNN 667 (CNN_D2), CNN 671 (CNN_D4), or CNN 675 (CNN_D6), the process proceeds to step S673.

[0325] In step S673, the occupancy state classification unit 668, 672, or 676 predicts the occupancy probability of each coordinate in the coordinate set C^out, extracts the top k coordinates with the highest occupancy probabilities, and deletes the remaining coordinates. At this time, the k is determined by applying a scale value from N_X, N_X', or N_X'' that corresponds to the sparse tensor being processed. The occupancy state classification unit 668, 672, or 676 also deletes the features corresponding to the deleted coordinates in F^out.

[0326] When the processing of step S673 is completed, the processing proceeds to step S674. Also, if it is determined in step S672 that occupancy state classification is not to be performed, the processing of step S673 is omitted, and the processing proceeds to step S674. In other words, if the processing of step S671 is performed by CNN 665 (CNN_D1), CNN 669 (CNN_D3), or CNN 673 (CNN_D5), the processing of step S673 is omitted, and the processing proceeds to step S674.

[0327] In step S674, CNN675 (CNN_D6) determines whether processing has been performed for all 3D-CNNs. If it is determined that processing has not been performed for all 3D-CNNs (i.e., that there are unprocessed 3D-CNNs (at least CNN675 (CNN_D6) is unprocessed)), processing proceeds to step S675.

[0328] In step S675, CNN665 (CNN_D1), CNN667 (CNN_D2), CNN669 (CNN_D3), CNN671 (CNN_D4), CNN673 (CNN_D5), or CNN675 (CNN_D6) sets {C^in, F^in} ← {C^out, F^out}. In other words, CNN665 (CNN_D1), CNN667 (CNN_D2), CNN669 (CNN_D3), CNN671 (CNN_D4), CNN673 (CNN_D5), or CNN675 (CNN_D6) sets the sparse tensor {C^out, F^out} of the calculation result derived by itself as the sparse tensor {C^in, F^in} to be processed by the next processing unit (guide surface feature derivation unit or occupancy state classification unit). When the processing of step S675 is completed, the process returns to step S656 of FIG. 29, and subsequent processing is executed. That is, the processes of steps S656 to S658 in FIG. 29 and the processes of steps S671 to S675 in FIG. 30 are executed for each 3D-CNN (which may include a guide surface feature amount derivation unit and an occupancy state classification unit).

[0329] Then, in step S674 of FIG. 30, if it is determined that the processing has been performed for all 3D-CNNs, the processing proceeds to step S676.

[0330] In step S676, the occupancy state classification unit 676 sets C_X ← C^out. That is, the occupancy state classification unit 676 sets the coordinate set C^out of the calculation result as the decoding result (decoded point group C_X). The occupancy state classification unit 676 also extracts attribute values ​​from F^out. For example, if the attribute is an RGB value, the occupancy state classification unit 676 extracts the last three dimensions of each feature vector of F^out and sets the vector as the attribute.

[0331] In step S677, the occupancy state classification unit 676 outputs the obtained decoded point group C_X to the outside of the decoding device 650. The occupancy state classification unit 676 also outputs the obtained attributes to the outside of the decoding device 650.

[0332] When the process of step S677 is completed, the decoding process ends.

[0333] By performing each process in this manner, the decoding device 350 can decode not only geometry but also attributes.

[0334] <Comparison> FIG. 31 shows an example of the results of comparing the RD costs when the present technology described above is applied and when the conventional method is applied.

[0335] The graph in Figure 31 shows an example of an RD curve plot when a certain point group (one frame, approximately 900,000 points, bit precision 10 bits) obtained by scanning a specific person is intra-encoded and intra-decoded.

[0336] In the graph of Figure 31, the vertical axis indicates mseF PSNR (p2point), or the so-called D1 Metric PSNR. In other words, if the position on the horizontal axis is the same, the higher the position on the vertical axis, the higher the quality of the decoded result. The horizontal axis indicates the bit size per point (bpp (bit per point)). In other words, if the position on the vertical axis is the same, the further to the left the position on the horizontal axis, the lower the code amount.

[0337] The solid line in the graph of Fig. 31 is an RD curve showing the relationship between bit size and PSNR when encoding a point cloud using a conventional method without using a guide surface. The dotted line in the graph of Fig. 31 is an RD curve showing the relationship between bit size and PSNR (Peak Signal-to-Noise Ratio) when encoding a point cloud using a guide surface by applying the present technology.

[0338] As shown in Figure 31, the dotted line is generally located above the solid line. Therefore, encoding using the method of applying the present technology (method using a guide surface) can obtain better results (high quality and low code amount) than conventional methods (methods not using a guide surface). In other words, by applying the present technology, it is possible to suppress a decrease in encoding efficiency.

[0339] <4. Supplementary Notes> <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0340] FIG. 32 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0341] In a computer 900 shown in FIG. 32, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0342] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0343] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0344] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0345] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.

[0346] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0347] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0348] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0349] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

[0350] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0351] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0352] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0353] This technology can be used, for example, as a technology aimed at improving productivity and safety at construction sites and resolving labor shortages (so-called smart construction), to create digital twins of construction sites through 3D surveying and use them for construction management. For example, this technology can be applied to 3D surveying using sensors mounted on drones or construction machinery, and feedback (for example, construction progress management and soil volume management) based on the 3D data obtained by the surveying (for example, point clouds).

[0354] Such surveys are generally performed multiple times at different times (e.g., every other day). In other words, point cloud data obtained by surveys at different times accumulates. Therefore, as the number of surveys increases, the storage capacity and transmission costs of point cloud data sequences can become a problem. There is redundancy between such point clouds from different times, such as having similar structures. In other words, there is temporal redundancy. By applying this technology, it is expected that point cloud compression utilizing redundancy in the temporal direction, i.e., point cloud sequence compression, will be effective.

[0355] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.

[0356] Furthermore, various types of information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, the associated pieces of data may be combined into one piece of data or may be separate pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to only part of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0357] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0358] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0359] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0360] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0361] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0362] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0363] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0364] The present technology can also be configured as follows: (1) An information processing device comprising: a guide surface feature derivation unit that derives guide surface feature amounts that are feature amounts based on a positional relationship between a guide surface and the points, using a first coordinate set that is a set of coordinates of points that form a point cloud, a first feature set that is a set of feature amounts of the coordinates, and a guide surface that is a reference surface formed in three-dimensional space, a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature set that reflects the guide surface feature amounts, to derive a second coordinate set and a second feature set, a coordinate encoding unit that encodes the derived second coordinate set, and a feature encoding unit that encodes the derived second feature set, wherein the code amount of encoded data of the second coordinate set and the second feature set is smaller than the code amount of encoded data of the first coordinate set and the first feature set. (2) The information processing device according to (1), wherein the scales of the second coordinate set and the second feature set are smaller than the scales of the first coordinate set and the first feature set. (3) The information processing device according to (2), wherein the calculation unit derives the second coordinate set and the second feature set by recursively repeating the predetermined calculation multiple times. (4) The information processing device according to (3), wherein the calculation unit downscales the input coordinate set and feature set in all or some of the multiple times of the predetermined calculation. (5) The information processing device according to (4), wherein the guide surface feature derivation unit derives the guide surface feature for some or all of the scales of the coordinate set and the feature set. (6) The information processing device according to any of (1) to (5), further comprising: a guide surface generation unit that generates the guide surface using the point cloud; a guide surface encoding unit that encodes the guide surface and generates encoded data of the guide surface; and a guide surface decoding unit that decodes the encoded data of the guide surface and generates the guide surface. (7) The information processing device according to any one of (1) to (6), wherein the guide surface feature amount is a feature amount based on a nearest point on the guide surface with the point as a reference.(8) The information processing device according to any one of (1) to (7), wherein the guide surface is a mesh. (9) The information processing device according to any one of (1) to (7), wherein the guide surface is a triangle soup. (10) The information processing device according to any one of (1) to (7), wherein the guide surface is an implicit function. (11) The information processing device according to any one of (1) to (10), further comprising: a parameter set selection unit that selects a parameter set to be applied to encoding the point cloud. (12) The information processing device according to (11), further comprising: a guide surface generation unit that generates the guide surface using the point cloud; and a guide surface encoding unit that encodes the guide surface by applying the selected parameter set and generates encoded data of the guide surface. (13) The information processing device according to any one of (1) to (12), wherein the guide surface feature derivation unit derives the guide surface feature using the coordinate set, the feature set including attribute information, and the guide surface including texture. (14) An information processing method comprising: using a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature set, which is a set of features of the coordinates, and a guide surface, which is a reference surface formed in three-dimensional space, deriving guide surface features, which are features based on the positional relationship between the guide surface and the points; performing an operation using the first coordinate set and the first feature set reflecting the guide surface features, to derive a second coordinate set and a second feature set; encoding the derived second coordinate set; and encoding the derived second feature set.

[0365] (21) An information processing device comprising: a coordinate decoding unit that decodes encoded data to generate a first coordinate set that is a set of coordinates of points that form a point cloud; a feature decoding unit that decodes encoded data to generate a first feature set that is a set of feature amounts of the coordinates; a guide surface feature derivation unit that derives guide surface feature amounts that are feature amounts based on a positional relationship between the guide surface and the points using the generated first coordinate set, the generated first feature set, and a guide surface that is a reference surface formed in three-dimensional space; and a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature set reflecting the guide surface feature amounts to derive a second coordinate set and a second feature set, wherein the code amount of encoded data of the first coordinate set and the first feature set is smaller than the code amount of encoded data of the second coordinate set and the second feature set. (22) The information processing device according to (21), wherein a scale of the second coordinate set and the second feature set is larger than a scale of the first coordinate set and the first feature set. (23) The information processing device according to (22), wherein the calculation unit derives the second coordinate set and the second feature set by recursively repeating the predetermined calculation multiple times. (24) The information processing device according to (23), wherein the calculation unit upscales the input coordinate set and feature set in all or a part of the multiple predetermined calculations. (25) The information processing device according to (24), wherein the guide surface feature derivation unit derives the guide surface feature for some or all scales of the coordinate set and the feature set. (26) The information processing device according to any of (21) to (25), further comprising a guide surface decoding unit that decodes encoded data of the guide surface and generates the guide surface. (27) The information processing device according to any of (21) to (26), wherein the guide surface feature is a feature based on a nearest point on the guide surface with the point as a reference. (28) The information processing device according to any of (21) to (27), wherein the guide surface is a mesh.(29) The information processing device according to (28), comprising a fitting unit that fits the guide surface to the point cloud. (30) The information processing device according to any of (21) to (27), in which the guide surface is a triangle soup. (31) The information processing device according to any of (21) to (27), in which the guide surface is an implicit function. (32) The information processing device according to any of (21) to (31), in which the guide surface feature derivation unit derives the guide surface feature using the coordinate set, the feature set including attribute information, and the guide surface including texture. (33) An information processing method comprising: decoding encoded data to generate a first coordinate set which is a set of coordinates of points forming a point cloud; decoding the encoded data to generate a first feature set which is a set of feature amounts of the coordinates; deriving guide surface feature amounts which are feature amounts based on the positional relationship between the guide surface and the points using the generated first coordinate set, the generated first feature set, and a guide surface which is a reference surface formed in three-dimensional space; performing a predetermined calculation using the first coordinate set and the first feature set reflecting the guide surface feature amounts to derive a second coordinate set and a second feature set; and the amount of code of the encoded data of the first coordinate set and the first feature set is smaller than the amount of code of the encoded data of the second coordinate set and the second feature set.

[0366] 100 Calculator, 101 Guide surface feature amount derivation unit, 102 CNN, 120 Calculator, 121 Guide surface feature amount derivation unit, 122 Calculation unit, 300 Encoding device, 311 Guide surface generation unit, 312 Guide surface encoding unit, 313 Guide surface decoding unit, 314 Sparse tensor construction unit, 315 Guide surface feature amount derivation unit, 316 CNN (CNN_E1), 317 Guide surface feature amount derivation unit, 318 CNN (CNN_E2), 319 CNN (CNN_E3), 320 Guide surface feature amount derivation unit, 321 CNN (CNN_E4), 322 CNN (CNN_E5), 323 CNN (CNN_E6), 324 Coordinate encoding unit, 325 Feature amount encoding unit, 326 Score encoding unit, 350 Decoding device, 361 Score decoding unit, 362 Coordinate decoding unit, 363 Feature decoding unit, 364 Guide surface decoding unit, 365 CNN (CNN_D1), 366 Guide surface feature derivation unit, 367 CNN (CNN_D2), 368 Occupancy state classification unit, 369 CNN (CNN_D3), 370 Guide surface feature derivation unit, 371 CNN (CNN_D4), 372 Occupancy state classification unit, 373 CNN (CNN_D5), 374 Guide surface feature derivation unit, 375 CNN (CNN_D6), 376 Occupancy state classification unit, 900 Computer

Claims

1. An information processing device comprising: a guide surface feature derivation unit that derives guide surface features, which are features based on the positional relationship between the guide surface and the points, using a first coordinate set, which is a set of coordinates of points that form a point cloud, a first feature set, which is a set of features of the coordinates, and a guide surface, which is a reference surface formed in three-dimensional space; a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature set reflecting the guide surface features, to derive a second coordinate set and a second feature set; a coordinate encoding unit that encodes the derived second coordinate set; and a feature encoding unit that encodes the derived second feature set, wherein the amount of code for the encoded data of the second coordinate set and the second feature set is smaller than the amount of code for the encoded data of the first coordinate set and the first feature set.

2. The information processing device of claim 1, wherein the calculation unit recursively repeats the specified calculation multiple times, and in all or a portion of the multiple times, downscales the input coordinate set and feature set to derive the second coordinate set and the second feature set having a smaller scale than the first coordinate set and the first feature set, and the guide surface feature derivation unit derives the guide surface feature for each of the scales of the coordinate set and the feature set, or for all scales.

3. The information processing device of claim 1, further comprising: a guide surface generation unit that generates the guide surface using the point cloud; a guide surface encoding unit that encodes the guide surface and generates encoded data for the guide surface; and a guide surface decoding unit that decodes the encoded data for the guide surface and generates the guide surface.

4. The information processing device according to claim 1, wherein the guide surface feature amount is a feature amount based on a nearest point on the guide surface with respect to the point.

5. The information processing device according to claim 1, wherein the guide surface is a mesh.

6. The information processing device according to claim 1, wherein the guide surface is a triangle soup.

7. The information processing device according to claim 1, wherein the guide surface is an implicit function.

8. The information processing device according to claim 1, further comprising a parameter set selection unit for selecting a parameter set to be applied to encoding the point cloud.

9. The information processing device according to claim 1, wherein the guide surface feature amount derivation unit derives the guide surface feature amount using the coordinate set, the feature amount set including attribute information, and the guide surface including a texture.

10. An information processing method comprising: using a first coordinate set, which is a set of coordinates of points forming a point cloud, a first feature set, which is a set of features of the coordinates, and a guide surface, which is a reference surface formed in three-dimensional space, to derive guide surface features, which are features based on the positional relationship between the guide surface and the points; performing a predetermined calculation using the first coordinate set and the first feature set reflecting the guide surface features, to derive a second coordinate set and a second feature set; encoding the derived second coordinate set; encoding the derived second feature set; and a code amount of the encoded data of the second coordinate set and the second feature set is less than a code amount of the encoded data of the first coordinate set and the first feature set.

11. An information processing device comprising: a coordinate decoding unit that decodes encoded data to generate a first coordinate set, which is a set of coordinates of points that form a point cloud; a feature decoding unit that decodes encoded data to generate a first feature set, which is a set of features of the coordinates; a guide surface feature derivation unit that uses the generated first coordinate set, the generated first feature set, and a guide surface that is a reference surface formed in three-dimensional space to derive guide surface features, which are features based on the positional relationship between the guide surface and the points; and a calculation unit that performs a predetermined calculation using the first coordinate set and the first feature set reflecting the guide surface features, to derive a second coordinate set and a second feature set, wherein the code amount of the encoded data of the first coordinate set and the first feature set is smaller than the code amount of the encoded data of the second coordinate set and the second feature set.

12. The information processing device described in claim 11, wherein the calculation unit recursively repeats the specified calculation multiple times, and in all or a portion of the multiple times, upscales the input coordinate set and feature set to derive the second coordinate set and the second feature set having a larger scale than the first coordinate set and the first feature set, and the guide surface feature derivation unit derives the guide surface features for each of the scales of the coordinate set and the feature set, or for some or all of the scales.

13. The information processing device according to claim 11, further comprising: a guide surface decoding unit that decodes the encoded data of the guide surface and generates the guide surface.

14. The information processing device according to claim 11, wherein the guide surface feature amount is a feature amount based on a nearest point on the guide surface with respect to the point.

15. The information processing device according to claim 11, wherein the guide surface is a mesh.

16. The information processing device according to claim 11, further comprising a fitting unit that fits the guide surface to the point cloud.

17. The information processing device according to claim 11, wherein the guide surface is a triangle soup.

18. The information processing device according to claim 11, wherein the guide surface is an implicit function.

19. The information processing device according to claim 11, wherein the guide surface feature amount derivation unit derives the guide surface feature amount using the coordinate set, the feature amount set including attribute information, and the guide surface including texture.

20. An information processing method comprising: decoding encoded data to generate a first coordinate set which is a set of coordinates of points forming a point cloud; decoding the encoded data to generate a first feature set which is a set of features of the coordinates; deriving guide surface features which are features based on the positional relationship between the guide surface and the points using the generated first coordinate set, the generated first feature set, and a guide surface which is a reference surface formed in three-dimensional space; performing a predetermined calculation using the first coordinate set and the first feature set reflecting the guide surface features, to derive a second coordinate set and a second feature set; and a code amount of the encoded data of the first coordinate set and the first feature set is less than a code amount of the encoded data of the second coordinate set and the second feature set.

Images