Information processing apparatus and method

Abstract

The present disclosure relates to an information processing device and method that make it possible to encode the geometry of a point cloud using a neural implicit function. The present invention generates, on the basis of a parameter vector of a neural implicit function representing the geometry of 3D data, a feature quantity vector representing a lower spatial correlation than the spatial correlation of the parameter vector, and encodes the feature quantity vector. The present invention also decodes the encoded data to generate a feature quantity vector representing a lower spatial correlation than a spatial correlation of a parameter vector of a neuroimplicit function representing a geometry of the 3D data, and generates the geometry from the feature quantity vector. The present disclosure can be applied to, for example, an information processing apparatus, an electronic device, an information processing method, or a program.

Description

Technical Field

[0001] The present disclosure relates to information processing devices and methods, and more particularly to information processing devices and methods adapted to be able to encode the geometry of a point cloud using a coordinate-based network. Background Art

[0002] Conventionally, there is a method for encoding the attributes of a point cloud using a coordinate-based network (for example, see Non-Patent Document 1). According to this method, the attributes are converted into implicit neural representations, and the parameters of the coordinate-based network are encoded using LoD correlation (the difference between low resolution and high resolution).

[0003] Citation List

[0004] Non-patent literature

[0005] Non-patent literature 1: Berivan Isik, Philip A. Chou, Sung Jin Hwang, Nick Johnston, George Toderici, "LVAC: Learned Volumetric Attribute Compression for Point Clouds using Coordinate Based Networks", arXiv: 2111.08988v1 [cs.GR] November 17, 2021 Summary of the invention

[0006] Problems to be solved by the present invention

[0007] However, there is no published method for encoding the geometry of point clouds using coordinate-based networks.

[0008] The present disclosure is made in view of the above circumstances, and is suitable for encoding the geometry of a point cloud using a coordinate-based network.

[0009] Solution to the problem

[0010] According to one aspect of the present technology, an information processing device is an information processing device, which includes: a generation unit, which generates a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector based on a parameter vector of a coordinate-based network representing the geometry of 3D data; and an encoding unit, which encodes the feature vector.

[0011] An information processing method according to one aspect of the present technology is an information processing method including: generating a feature vector representing a spatial correlation lower than that of the parameter vector based on a parameter vector of a coordinate-based network representing a geometry of 3D data; and encoding the feature vector.

[0012] According to another aspect of the present technology, an information processing device is an information processing device including: a decoding unit that decodes encoded data to generate a feature vector representing a lower spatial correlation than a parameter vector of a coordinate-based network representing a geometric shape of 3D data; and a generation unit that generates a geometric shape based on the feature vector.

[0013] An information processing method according to another aspect of the present technology is an information processing method, which includes: decoding encoded data to generate a feature vector representing a spatial correlation lower than the spatial correlation of a parameter vector of a coordinate-based network representing the geometry of 3D data; and generating a geometry based on the feature vector.

[0014] An information processing device and method according to an aspect of the present technology generates a feature vector representing a spatial correlation lower than that of the parameter vector based on a parameter vector of a coordinate-based network representing a geometry of 3D data, and encodes the feature vector.

[0015] According to another aspect of the present technology, an information processing device and method decodes encoded data to generate a feature vector representing a lower spatial correlation than a parameter vector of a coordinate-based network representing a geometric shape of 3D data, and generates a geometric shape based on the feature vector. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart illustrating an example of the flow of an attribute encoding process using a coordinate-based network.

[0017] Figure 2 is a diagram showing an example of a state of a binary tree construction.

[0018] Figure 3 is a diagram showing an example of a binary tree.

[0019] Figure 4 is a diagram illustrating an example of a method for encoding geometric shapes using a coordinate-based network.

[0020] Figure 5 It is a block diagram showing a main configuration example of a geometric shape encoding device.

[0021] Figure 6is a diagram showing an example of a parameter vector grid and an example of a CNN feature vector grid.

[0022] Figure 7 is a diagram showing an example of spatial correlation.

[0023] Figure 8 is a flowchart showing an example of the flow of geometric shape encoding processing.

[0024] Fig. 9 is a flowchart showing an example of the flow of CNN feature vector grid generation processing.

[0025] Fig.10 It is a block diagram showing a main configuration example of a geometry decoding device.

[0026] Fig.11 is a flowchart showing an example of the flow of geometry decoding processing.

[0027] Fig.12 is a flowchart showing an example of the flow of parameter vector mesh generation processing.

[0028] Fig.13 is a flowchart showing an example of the flow of geometric shape encoding processing.

[0029] Fig.14 It is a block diagram showing a main configuration example of a geometric shape encoding device.

[0030] Fig.15 is a diagram showing an example of a mask grid and an example of a parameter vector grid.

[0031] Fig.16 is a flowchart showing an example of the flow of geometric shape encoding processing.

[0032] Fig.17 It is a block diagram showing a main configuration example of a geometry decoding device.

[0033] Fig.18 is a flowchart showing an example of the flow of geometry decoding processing.

[0034] Fig.19 is a flowchart showing an example of the process of CNN feature vector grid encoding processing.

[0035] Fig. 20 is a flowchart showing an example of the flow of CNN feature vector grid decoding processing.

[0036] Fig.21 is a diagram showing examples of correlations that are different in scale from each other.

[0037] Fig. 22is a diagram illustrating an example of a method for encoding geometric shapes using a coordinate-based network.

[0038] Fig.23 It is a block diagram showing a main configuration example of a geometric shape encoding device.

[0039] Fig.24 is a diagram showing an example of a parameter vector grid and an example of a CNN feature vector grid.

[0040] Fig.25 is a flowchart showing an example of the flow of geometric shape encoding processing.

[0041] Fig.26 It is a block diagram showing a main configuration example of a geometry decoding device.

[0042] Fig. 27 is a flowchart showing an example of the flow of geometry decoding processing.

[0043] Fig.28 It is a block diagram showing a main configuration example of a geometric shape encoding device.

[0044] Fig.29 is a diagram showing an example of a parameter vector grid and an example of a CNN feature vector grid.

[0045] Fig.30 is a flowchart showing an example of the flow of geometric shape encoding processing.

[0046] Fig.31 It is a block diagram showing a main configuration example of a geometry decoding device.

[0047] Fig.32 is a flowchart showing an example of the flow of geometry decoding processing.

[0048] Fig.33 is a diagram showing a configuration example of a 3D-CNN.

[0049] Fig.34 : is a block diagram showing a main configuration example of a point cloud encoding device.

[0050] Fig.35 is a flowchart showing an example of the flow of point cloud encoding processing.

[0051] Fig.36 : is a block diagram showing a main configuration example of a point cloud decoding device.

[0052] Fig.37 is a flowchart showing an example of the flow of point cloud decoding processing.

[0053] Fig.38 is a block diagram showing a main configuration example of a computer. DETAILED DESCRIPTION

[0054] Modes for carrying out the present disclosure (hereinafter, referred to as embodiments) will be described below. Note that the description is made in the following order.

[0055] 1. Documents supporting technical content and technical terminology, etc.

[0056] 2.LVAC

[0057] 3. Encoding geometric shapes using coordinate-based networks

[0058] 4. Generate parameter vector by optimizing Lagrangian function

[0059] 5. Application of Sparse 3D Convolution

[0060] 6. Encoding CNN feature vector using mask grid

[0061] 7. Multi-resolution CNN feature vectors

[0062] 8. Implementation Method (Point Cloud Coding Device)

[0063] 9. Implementation Method (Point Cloud Decoding Device)

[0064] 10. Additional Notes

[0065] <1. Documents supporting technical content and technical terminology, etc.>

[0066] In addition to the contents described in the embodiments, the scope disclosed in the present technology also includes the contents described in the following non-patent documents and the like known at the time of filing this application, the contents of other documents cited in the following non-patent documents, and the like.

[0067] Non-Patent Document 1: (described above)

[0068] That is, the contents described in the aforementioned non-patent document, the contents described in other documents cited in the aforementioned non-patent document, and the like constitute the basis for determining the support requirements.

[0069] <2.LVAC>

[0070] <Point Cloud>

[0071] Traditionally, as 3D data representing the three-dimensional structure of a three-dimensional object, there is a point cloud that represents the object as a collection of multiple points. The data of the point cloud (also called point cloud data) includes the geometry (position information) and attributes (attribute information) of each point in the point cloud. Each geometry indicates the position of the corresponding point in three-dimensional space. Each attribute indicates the attribute of the corresponding point. The attribute may include arbitrary information. For example, each attribute may include color information, reflectivity information, normal information, etc. of the corresponding point. As described above, the point cloud has a relatively simple data structure, and by using enough points, any three-dimensional object can be represented with sufficient accuracy.

[0072] <Voxel representation>

[0073] However, since the point cloud has a relatively large amount of data, it is necessary to compress the data volume by encoding, etc. For example, for the geometry of each point, the higher the position accuracy, the larger the data volume. In view of this, a method of using voxels to represent geometric shapes has been considered. Voxel refers to an area obtained by dividing a three-dimensional space area including an object. In the point cloud, the position of each point is located at a predetermined position (for example, the center) in the voxel. In other words, the voxel is used to indicate whether the point exists in the corresponding voxel. Therefore, the geometry of each point can be quantified based on voxels. Therefore, the increase in the amount of data of each geometric shape can be suppressed. Note that the method of using voxels to represent geometric shapes is also referred to as voxel representation in this article.

[0074] A voxel can be divided into multiple voxels. That is, by repeatedly dividing each voxel in a recursive manner, the size of each voxel can be further reduced. The smaller the size of each voxel, the higher the resolution. That is, the position of each point can be represented more accurately. In other words, the effect of reducing the amount of data for each geometric shape by the aforementioned quantization is suppressed.

[0075] Note that only the voxels where a point exists are divided in this voxel representation. Each voxel where a point exists is divided into eight voxels (2×2×2). Among the 8 voxels, only the voxel where a point exists is further divided into 8 voxels. In this way, the voxel where a point exists is divided recursively until the voxel takes its smallest unit. Thus, a hierarchical structure is formed.

[0076] <Octtree representation>

[0077] As described above, the voxel representation indicates whether a point exists in each voxel. In other words, the voxels within each level are defined as nodes, and for each divided voxel, 0 and 1 are used to indicate whether a point exists. Therefore, a tree structure (octree) can be used to represent each geometric shape. The method of using the octree to represent geometric shapes is also referred to as octree representation in this article. The bit patterns of each node are arranged and encoded in a predetermined order. By using the octree representation, scalable geometric shape decoding can be achieved. That is, geometric shapes of any level (resolution) can be obtained by decoding only the necessary information.

[0078] In addition, according to the voxel representation, as described above, the division of voxels where no points exist can be omitted. Therefore, according to the octree representation, nodes where no points exist can also be omitted. Therefore, the increase in the amount of data for each geometric shape can be suppressed.

[0079] <lvac>

[0080] Meanwhile, non-patent document 1 discloses a method for encoding attributes of point clouds using a coordinate-based network (LVAC (Learning Volume Attribute Compression of Point Clouds Using Coordinate-Based Networks)). According to this method, attributes have been converted into implicit neural representations, and parameters of the coordinate-based network have been encoded using LoD correlation (difference between low resolution and high resolution).

[0081] Figure 1 is a flowchart showing an example of the flow of attribute encoding processing. Figure 1 As shown, the encoder divides 3D data such as point clouds into multiple blocks and constructs a binary tree in which leaf nodes represent occupied blocks (step S11). Figure 2 As shown, the encoder recursively divides the region into two to build a tree structure (binary tree) in which the nodes represent the parameter vector (z) in each region.

[0082] Figure 3 is a diagram showing an example of a binary tree. The relationships represented by the following equations (1) to (3) are satisfied, where zl,n represents the direct current (DC) component of the nth node at level l. Note that n L 、n R Represent the indexes of the left and right child nodes respectively, and w * Indicates the number of points in each of the left and right child nodes.

[0083]

[0084]

[0085] Using a binary tree, the encoder derives a global DC component ( Figure 3 z in 0,0 ) and {number of leaf nodes}-1 AC component ( Figure 3 δz in 1,1 ,δz 2,1 ,δz 2,3 )(Step S12). The global DC component refers to the parameter vector in the entire region (the root node of the binary tree). The AC component refers to the difference between a parent node and its child nodes.

[0086] The encoder determines the quantization width and scales the value (step S13). Next, the encoder quantizes and entropy encodes the global DC component and AC component to generate and output a bit stream (step S14).

[0087] On the other hand, the decoder performs the reverse process. For example, the decoder performs entropy decoding and inverse scaling on the bitstream to obtain the global DC component and AC component. Then, the decoder combines the global DC component and AC component together to obtain the parameter vector ( Figure 3 One of the child nodes is obtained by adding the AC component to the parent node. Based on the relationship represented by equation (1), another child node is obtained using the parent node and one child node. Next, the decoder inputs the parameter vector and the coordinate value into the coordinate-based network to calculate the attribute.

[0088] According to LAVC, the correlation between LoDs (ie, the AC components are closer to 0) is used in this way to compress the parameter vector.

[0089] However, methods for encoding the geometry of 3D data (eg, point clouds) using coordinate-based networks have not been disclosed.

[0090] <3. Encoding geometric shapes using coordinate-based networks>

[0091] <Method 1>

[0092] Thus, as in LVAC, a coordinate-based network is used to encode the geometry of 3D data (e.g., point clouds). That is, the geometry is transformed into an implicit neural representation, and the parameter vector of the implicit neural representation is encoded.

[0093] For example, Figure 4 As shown in the top row of the table, the spatial correlation of the parameter vector of the coordinate-based network for geometric shapes is reduced and encoded (method 1). This method 1 can be applied to an encoder that encodes geometric shapes, for example.

[0094] For example, an information processing device includes: a generation unit that generates a feature vector representing a spatial correlation lower than that of the parameter vector based on a parameter vector of a coordinate-based network representing a geometric shape of 3D data; and an encoding unit that encodes the feature vector.

[0095] In addition, an information processing method includes: generating a feature vector representing a spatial correlation lower than that of the parameter vector based on a parameter vector of a coordinate-based network representing a geometric shape of 3D data; and encoding the feature vector.

[0096] Implicit function representation refers to a method of representing a 3D shape based on a function. The 3D shape is represented by an implicit function fθ that inputs coordinate values ​​(x, y, z) and outputs the occupancy value at the coordinate value: R 3 →R denotes. Note that R denotes a set of real numbers. A coordinate-based network refers to an implicit function that is approximated using a neural network (NN).

[0097] A neural network refers to a composite function of multiple linear and nonlinear transformations. Linear and nonlinear transformations are performed alternately on the input vector x to obtain an output vector y. In the linear transformation, matrix multiplication is performed on the vector, and the bias vector is added. In the nonlinear transformation, a nonlinear function is applied to each element of the vector. The parameters thus obtained can be trained with the weight matrix and bias vector of all linear transformations. These linear and nonlinear transformations are stacked in multiple layers to optimize the parameters so that the complex transformation (function) from x to y can be approximated. Note that the neural network is also called a fully connected network or a multilayer perceptron.

[0098] Furthermore, changing the transformation from input x to output y in a neural network according to the value of another vector c is also called conditioning, and this vector c is also called a conditional vector. Changing only the conditional vector without changing all the weights of the network allows a change in the transformation from x to y.

[0099] Examples of the adjustment method include the following methods.

[0100] Concatenation: The input vector x and the conditional vector c are concatenated in the dimension direction and input to the network.

[0101] Modulation: The output vector of the intermediate layer of the network undergoes a linear transformation of each element according to the value of the conditional vector.

[0102] Meta-Network: Use another neural network to predict all the weights of the network based on the conditional vector.

[0103] The use of such a neural network enables the representation of complex shapes and smooth and continuous surfaces. The use of the parameter vector adjustment enables the representation of various shapes by changing the parameter vector alone without changing the weights of the network. That is, the parameter vector corresponds to the conditional vector.

[0104] Therefore, the geometry of the point cloud can be encoded using a coordinate-based network. The overall encoding efficiency can also be improved by encoding the geometry. The encoding efficiency can also be improved by reducing spatial correlation. The processing can also be parallelized by reducing spatial correlation. Therefore, the processing time of the decoding process can be reduced. The encoding processing volume can also be reduced.

[0105] Furthermore, when method 1 is applied, for example, as from Figure 4 As shown in the second row from the top of the table, a three-dimensional (3D)-CNN with a 3D convolution layer can be used to generate a CNN feature vector (method 1-1). For example, the generating unit can generate a feature vector using a 3D convolutional neural network (3D-CNN) which is a neural network with a three-dimensional convolution layer.

[0106] 3D-CNN refers to a fully connected neural network with a 3D convolution layer (or 3D deconvolution layer) as a linear layer, and 3D-CNN is composed of a 3D convolution layer and a pooling layer (e.g., downsampling calculation, such as average pooling to calculate the average value). The 3D convolution layer performs a 3D convolution calculation. Note that the 3D convolution calculation corresponds to a two-dimensional convolution calculation whose input, output, and filter are three-dimensionally extended, and the 3D convolution calculation refers to the calculation of convolving a filter of a predetermined size (three-dimensional) into each data item in a three-dimensional area. The filter coefficients correspond to trainable weights.

[0107] Furthermore, when method 1-1 is applied, for example, as from Figure 4 As shown in the third row from the top of the table, a parameter vector can be generated based on a geometric shape using 3D-CNN, and a CNN feature vector can be generated based on the parameter vector using 3D-CNN (method 1-1-1). For example, the generation unit can generate a parameter vector that is maintained in a grid shape based on a geometric shape, and use 3D-CNN to generate a feature vector using the parameter vector as input. The generation unit can also use 3D-CNN to generate a parameter vector using a geometric shape as input.

[0108] <Geometric shape encoding device>

[0109] Figure 5 : is a block diagram showing an example of the configuration of a geometric shape encoding device as one aspect of an information processing device to which the present technology is applied. Figure 5 The geometric shape encoding device 100 shown is a device for encoding the geometric shape (3D data) of a point cloud. The geometric shape encoding device 100 applies the aforementioned method 1-1-1 to encode the geometric shape.

[0110] Notice, Figure 5 The main processing unit, data flow, etc. are shown; therefore, the geometric shape encoding device 100 is not limited to Figure 5 That is, the geometric shape encoding device 100 may include Figure 5 The processing units not shown in block form and may involve Figure 5 Processing or data flow not indicated by arrows etc.

[0111] like Figure 5 As shown, the geometric shape encoding device 100 includes a parameter vector grid generation unit 111, a CNN feature vector grid generation unit 112 and an encoding unit 113.

[0112] The parameter vector mesh generation unit 111 performs processing related to the generation of the parameter vector mesh. For example, the parameter vector mesh generation unit 111 can obtain a geometric shape to be input to the geometric shape encoding device 100. In addition, the parameter vector mesh generation unit 111 can generate a parameter vector maintained in a mesh shape according to the geometric shape. For example, the parameter vector mesh generation unit 111 can use 3D-CNN (3D-CNN_E1) ​​to generate a parameter vector (parameter vector mesh) according to the geometric shape. That is, the parameter vector mesh generation unit 111 can input the acquired geometric shape to 3D-CNN_E1, and transform the geometric shape into Figure 6 A parameter vector grid 131 is shown.

[0113] The parameter vector grid refers to a four-dimensional tensor (four-dimensional array) in which one-dimensional parameter vectors are arranged in a three-dimensional grid shape. Under the condition that the parameter vector has dimension Cp and the parameter vector grid has a block size Bp, for 3D-CNN_E1, the input dimension needs to be 1, the output dimension needs to be Cp, and the downscaling factor needs to be 1 / Bp. As long as this condition is met, both the hyperparameters and the structure have degrees of freedom.

[0114] The parameter vector grid generation unit 111 may supply the generated parameter vector grid to the CNN feature vector grid generation unit 112 .

[0115] The CNN feature vector mesh generation unit 112 performs processing related to the generation of the CNN feature vector mesh. For example, the CNN feature vector mesh generation unit 112 may acquire the parameter vector mesh (the parameter vector maintained in a mesh shape) supplied from the parameter vector mesh generation unit 111. In addition, the CNN feature vector mesh generation unit 112 may generate a feature vector (CNN feature vector) from the parameter vector using 3D-CNN (3D-CNN_E2). That is, the CNN feature vector mesh generation unit 112 may acquire the parameter vector mesh 131 ( Figure 6 ) is input to 3D-CNN_E2, and the parameter vector grid 131 is transformed into Figure 6 The CNN feature vector grid 132 is shown. Therefore, it can be said that the CNN feature vector grid generation unit 112 is a generation unit.

[0116] Under the condition that the CNN feature vector has dimension Cf and the CNN feature vector grid has block size Bf, for 3D-CNN_E2, the input dimension needs to be Cp, the output dimension needs to be Cf, and the downscaling factor needs to be Bp / Bf to account for nonlinear spatial correlation. In addition, 3D-CNN_E2 has at least two 3D convolutional layers and one nonlinear transformation layer. In addition, at least one of the 3D convolutional layers has a kernel size greater than 1. As long as this condition is met, both the hyperparameters and the structure have degrees of freedom.

[0117] The CNN feature vector grid generation unit 112 may supply the generated CNN feature vector grid to the encoding unit 113 .

[0118] The encoding unit 113 performs processing related to the encoding of the CNN feature vector grid. For example, the encoding unit 113 may obtain the CNN feature vector grid supplied from the CNN feature vector grid generation unit 112. In addition, the encoding unit 113 may encode the CNN feature vector. For example, the encoding unit 113 may quantize the acquired CNN feature vector grid and entropy encode the quantization level, thereby generating encoded data (i.e., a bitstream) of the CNN feature vector grid. The encoding unit 113 may quantize and entropy encode all values ​​of the CNN feature vector grid independently of each other. At this time, the encoding unit 113 may apply a quantization width that is common in the spatial direction and different for each dimension to quantization. In addition, the encoding unit 113 may apply a probability table that is common in the spatial direction and different for each dimension to entropy encoding.

[0119] The encoding unit 113 outputs the generated bit stream (encoded data of the CNN feature vector grid) to the outside of the geometry encoding device 100. The bit stream can be provided to the decoding device via an arbitrary transmission path or using an arbitrary recording medium.

[0120] As described above, quantization and entropy encoding using a quantization width and probability table that are common in the spatial direction and different for each dimension allow the encoding unit 113 to encode (quantize and entropy encode) the CNN feature vectors independently of each other. In other words, the encoding unit 113 is able to encode (quantize and entropy encode) the CNN feature vectors in parallel. By encoding the CNN feature vectors in parallel, encoding can be performed at a higher speed (i.e., to reduce processing time).

[0121] For example, in the case of LVAC for attributes, the coding efficiency is improved by encoding the parameter vector using the correlation between the levels of the binary tree (also called LoD correlation). When the simple correlation with the surrounding blocks is high, the LoD correlation becomes high. l+1,nR LoD correlation is used under the assumption that z is close to 0. That is, when this assumption holds, the compression efficiency becomes higher. This is equivalent to the case where the values ​​of adjacent parameter vectors are close to each other (e.g., z l+1,nR and z l+1,nL close to each other).

[0122] However, in the case of geometric shapes, neighboring blocks can be in a similar relationship. Figure 7 , it is assumed that block 141 and block 142 each correspond to a block of a geometric shape, and the black lines in block 141 and block 142 each indicate the position where a point group exists. In this case, the position of the point group in block 141 and the position of the point group in block 142 are in a roughly line-symmetrical relationship. Therefore, the simple difference in the geometry between block 141 and block 142 is large. However, by obtaining a difference at the boundary between block 141 and block 142 by line-symmetrical movement of one of block 141 and block 142, the difference can be made smaller than the simple difference. In the case of a geometric shape, as described above, adjacent blocks are in a similar relationship (or in a relationship close enough to a similar relationship), and can be matched or approximated to each other by rotational movement, translation, scaling, symmetry operations, combinations thereof, and the like.

[0123] In addition, candidates for the geometric shape of the processing object block can be predicted based on the geometric shapes of neighboring blocks and prior knowledge. In the case of the geometric shape, there may be complex correlations as described above.

[0124] Such complex correlations can be used for compression; however, as described above, there is a possibility that it cannot be adequately captured by LoD correlation. Note that, as described above, a relationship in the spatial direction that does not rely on the structure of a binary tree is also called a correlation in the spatial direction (correlativity) or redundancy in the spatial direction.

[0125] As described above, when the entropy model (probability model) of entropy coding is common in the spatial direction, the high correlation in the spatial direction is not applicable to the entropy model. Therefore, the probability is high that the probability drops to the lower side. Therefore, the amount of coding cannot be fully reduced, so that there is still room for improvement in coding efficiency.

[0126] In the case of the geometry encoding device 100, the CNN feature vector grid generation unit 112 inputs a parameter vector grid to 3D-CNN_E2 to generate a CNN feature vector grid. Therefore, the parameter vector is subjected to 3D convolution calculation, and the feature vector takes a more random value in the spatial direction. That is, the correlation in the spatial direction can be reduced. That is, the CNN feature vector grid generation unit 112 generates a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector based on the parameter vector of the coordinate-based network representing the geometry of the 3D data. Therefore, compared with the case where the geometry is encoded using LoD correlation, the geometry encoding device 100 can improve the encoding efficiency.

[0127] In addition, the geometry coding device 100 can encode (quantize and entropy encode) CNN feature vectors in parallel while suppressing a decrease in encoding efficiency. In other words, the geometry coding device 100 can perform encoding at a higher speed (ie, can reduce processing time).

[0128] Note that in LAVC for attributes, a parameter vector is obtained by optimizing a Lagrangian function. Since gradient calculation based on back propagation is performed, the optimization requires a large amount of processing. In contrast, according to the geometry encoding device 100, the parameter vector mesh generation unit 111 inputs a geometry to the 3D-CNN_E1 having a three-dimensional convolutional layer to generate a parameter vector mesh. Therefore, the geometry encoding device 100 can reduce the amount of processing compared to the case of using optimization of the Lagrangian function.

[0129] Note that the parameter vector grid generation unit 111 and the CNN feature vector grid generation unit 112 can be integrated into one to serve as a generation unit 121. That is, the generation unit 121 can generate a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector of the coordinate-based network representing the geometry according to the geometry of the 3D data. At this time, the generation unit 121 can use a 3D-CNN with a three-dimensional convolutional layer to generate a feature vector (CNN feature vector). For example, the generation unit 121 can generate a parameter vector maintained in a grid shape according to the geometry, and use a 3D-CNN to generate a feature vector using the parameter vector as input. At this time, the generation unit 121 can use a 3D-CNN to generate a parameter vector using the geometry as input.

[0130] In addition, the aforementioned 3D-CNN_E1 and 3D-CNN_E2 may be implemented using a single 3D-CNN. For example, the generating unit 121 may input the geometric shape input to the geometric shape encoding device 100 to the 3D-CNN to transform the geometric shape into Figure 6 The CNN feature vector grid 132 shown. In addition, in this case, the geometry encoding device 100 can produce a similar beneficial effect.

[0131] <Process flow of geometry encoding processing>

[0132] As described above, the geometry encoding device 100 performs geometry encoding processing to encode the geometry. Refer to Figure 8 the flowchart of to describe an example of the process flow of this geometry encoding processing.

[0133] When the geometry encoding processing starts, in step S101, the parameter vector grid generation unit 111 generates a parameter vector grid according to the geometry using 3D-CNN_E1.

[0134] In step S102, the CNN feature vector grid generation unit 112 performs CNN feature vector grid generation processing to generate a CNN feature vector grid according to the parameter vector grid using 3D-CNN_E2. That is, the CNN feature vector grid generation unit 112 generates a feature vector representing a lower spatial correlation than the spatial correlation of the parameter vector based on the parameter vector of the coordinate-based network representing the geometry of the 3D data.

[0135] In step S103, the encoding unit 113 quantizes and entropy-encodes the CNN feature vector grid to generate encoded data (i.e., bitstream) of the CNN feature vector grid.

[0136] When the processing in step S103 ends, the geometry encoding processing ends.

[0137] <Process flow of CNN feature vector grid generation processing>

[0138] Next, refer to Fig. 9 the flowchart of to describe an example of the process flow of the CNN feature vector grid generation processing performed in Figure 8 step S102 shown.

[0139] When the CNN feature vector grid generation processing starts, in step S131, the CNN feature vector grid generation unit 112 inputs the parameter vector grid generated in Figure 8 step S101 shown.

[0140] In step S132, the CNN feature vector grid generation unit 112 initializes the four-dimensional array input using the parameter vector grid.

[0141] In step S133, the CNN feature vector grid generation unit 112 selects the processing object calculation layer of 3D-CNN_E2.

[0142] In step S134, the CNN feature vector grid generation unit 112 initializes the four-dimensional array output.

[0143] In step S135, the CNN feature vector grid generation unit 112 selects the processing object indexes i, j and k of the output in the spatial direction from the unprocessed indexes in a predetermined order.

[0144] In step S136, the CNN feature vector grid generation unit 112 calculates the processing object indexes i, j and k in the processing object calculation layer.

[0145] In step S137, the CNN feature vector grid generation unit 112 determines whether all indexes have been processed. When there are unprocessed indexes, the CNN feature vector grid generation process returns to step S136. Then various types of processing in and after step S136 are performed. That is, while updating the processing object indexes i, j, and k, various types of processing from step S135 to step S137 are repeatedly performed until all indexes are subjected to calculation in the processing object calculation layer.

[0146] When it is determined in step S137 that all indexes have been subjected to calculation in the processing object calculation layer, the CNN feature vector grid generation process proceeds to step S138.

[0147] In step S138, the CNN feature vector mesh generation unit 112 determines whether all calculation layers have been processed. When it is determined that there is an unprocessed calculation layer, the CNN feature vector mesh generation process proceeds to step S139.

[0148] In step S139, the CNN feature vector mesh generation unit 112 uses the output (the calculation result in step S136) as input. When the processing in step S139 ends, the CNN feature vector mesh generation processing returns to step S133. That is, while updating the processing object calculation layer, various types of processing from step S133 to step S139 are repeatedly performed until calculations are performed in all calculation layers.

[0149] When it is determined in step S138 that calculations have been performed in all calculation layers, the CNN feature vector grid generation process proceeds to step S140. In step S140, the CNN feature vector grid generation unit 112 outputs the output (the calculation result in step S136) as a CNN feature vector grid (i.e., supplies the output to the encoding unit 113). When the process in step S140 ends, the CNN feature vector grid generation process ends. Then, the process flow returns to Figure 8 .

[0150] By performing various types of processing as described above, the geometry encoding device 100 is able to encode the geometry of the point cloud using a coordinate-based network. The geometry encoding device 100 is also able to improve the overall encoding efficiency by encoding the geometry. The geometry encoding device 100 is also able to improve the encoding efficiency by reducing spatial correlation. The geometry encoding device 100 is also able to encode feature vectors in parallel while suppressing the reduction in encoding efficiency by reducing spatial correlation. Therefore, the geometry encoding device 100 is able to perform encoding processing at a higher speed (i.e., the processing time can be reduced). The geometry encoding device 100 is also able to reduce the processing time of the decoding process. The geometry encoding device 100 is also able to reduce the amount of encoding processing by using 3D-CNN to generate a parameter vector grid.

[0151] <Method 1>

[0152] The aforementioned method 1 can also be applied to a decoder that decodes, for example, coded data of a geometric shape. For example, an information processing device includes: a decoding unit that decodes the coded data to generate a feature vector that represents a spatial correlation lower than the spatial correlation of a parameter vector of a coordinate-based network representing the geometric shape of 3D data; and a generating unit that generates a geometric shape based on the feature vector.

[0153] Additionally, an information processing method includes decoding encoded data to generate a feature vector representing a spatial correlation lower than a spatial correlation of a parameter vector of a coordinate-based network representing a geometric shape of 3D data; and generating a geometric shape from the feature vector.

[0154] Thus, the geometry of the point cloud encoded by reducing the spatial correlation of the parameter vectors of the coordinate-based network can be decoded. Thus, by implementing geometry encoding, an overall improvement in coding efficiency can be achieved. The coding efficiency can be further improved due to the reduced spatial correlation.

[0155] When in the case of an encoder, method 1-1 and method 1-1-1 can also be applied. For example, the generation unit can generate a parameter vector based on the feature vector, and generate a geometric shape using a coordinate-based network to which the parameter vector is applied. The generation unit can also generate a parameter vector using a 3D-CNN with a three-dimensional inverse (3D) convolution layer with a feature vector as input. The 3D deconvolution layer performs a 3D deconvolution calculation. The 3D deconvolution calculation is the inverse process of the 3D convolution calculation.

[0156] By using 3D-CNN as described above, the calculation process can be parallelized for each element. Therefore, compared with the case where the global DC component and the AC component are combined together as in the LAVC for the attribute, the parameter vector can be generated at a higher speed (i.e., the processing time for generating the parameter vector is reduced). In particular, in the case where the global DC component and the AC component are combined together, the processing time for generating the parameter vector increases as the number of blocks increases. On the other hand, the use of 3D-CNN makes the processing time basically constant, regardless of the number of blocks. That is, even when the number of blocks increases, the increase in the processing time for generating the parameter vector can be suppressed.

[0157] <Geometric shape decoding device>

[0158] Fig.10 : is a block diagram showing an example of the configuration of a geometry decoding device as one aspect of an information processing device to which the present technology is applied. Fig.10 The geometric shape decoding device 200 shown is a device for decoding the coded data (3D data) of the geometric shape of the point cloud. The geometric shape decoding device 200 applies the aforementioned method 1-1-1 to decode the coded data of the geometric shape. For example, the geometric shape decoding device 200 decodes the bit stream generated by the geometric shape encoding device 100 (i.e., the coded data of the CNN feature vector grid of the geometric shape) to generate (reconstruct) the geometric shape.

[0159] Notice, Fig.10 The main processing unit, data flow, etc. are shown; therefore, the geometry decoding device 200 is not limited to Fig.10 That is, the geometry decoding device 200 may include Fig.10 The processing units not shown in block form and may involve Fig.10 Processing or data flow not indicated by arrows etc.

[0160] like Fig.10 As shown, the geometry decoding device 200 includes a decoding unit 211 , a parameter vector grid generating unit 212 and a geometry generating unit 213 .

[0161] The decoding unit 211 performs processing related to the decoding of the bitstream. For example, the decoding unit 211 may obtain a bitstream to be input to the geometry decoding device 200 (i.e., the encoded data of the CNN feature vector grid of the geometry). In addition, the decoding unit 211 may decode the bitstream. For example, the decoding unit 211 may entropy decode the acquired bitstream to generate (reconstruct) quantization levels, and may inversely quantize the quantization levels to generate (reconstruct) the CNN feature vector grid. That is, the decoding unit 211 may decode the encoded data to generate a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector of the coordinate-based network representing the geometry of the 3D data. At this time, the decoding unit 211 uses the same quantization width and probability table as the quantization width and probability table used for the encoder (e.g., the encoding unit 113). In addition, the decoding unit 211 may supply the generated CNN feature vector grid to the parameter vector grid generation unit 212.

[0162] The parameter vector mesh generation unit 212 performs processing related to the generation of the parameter vector mesh. For example, the parameter vector mesh generation unit 212 may acquire the CNN feature vector mesh supplied from the decoding unit 211. In addition, the parameter vector mesh generation unit 212 may generate (reconstruct) a parameter vector from the CNN feature vector mesh using 3D-CNN (3D-CNN_D1). That is, the parameter vector mesh generation unit 212 may generate (reconstruct) a parameter vector from the acquired CNN feature vector mesh 132 ( Figure 6 ) is input to 3D-CNN_D1 to transform the CNN feature vector grid 132 into a parameter vector grid 131 ( Figure 6 ).

[0163] In order to consider nonlinear spatial correlation, for 3D-CNN_D1, the input dimension needs to be Cf, the output dimension needs to be Cp, and the upscaling factor needs to be Bf / Bp. For the purpose of upscaling, the geometric shape decoding device 200 uses a 3D deconvolution layer instead of a 3D convolution layer. In addition, 3D-CNN_D1 has at least two 3D deconvolution layers and one nonlinear layer. At least one of the 3D deconvolution layers has a kernel size greater than 1. As long as this condition is met, both the hyperparameters and the structure have degrees of freedom.

[0164] The parameter vector mesh generation unit 212 may supply the generated parameter vector mesh to the geometry generation unit 213 .

[0165] The geometry generation unit 213 performs processing related to the generation of the geometry. For example, the geometry generation unit 213 may acquire the parameter vector mesh supplied from the parameter vector mesh generation unit 212. In addition, the geometry generation unit 213 may obtain the geometry of the 3D data from the parameter vector mesh and the query point using a coordinate-based network representing the geometry of the 3D data. Then, the geometry generation unit 213 may output the generated geometry to the outside of the geometry decoding device 200.

[0166] For example, the geometry generation unit 213 first constructs a query point. The geometry generation unit 213 can set the query point by any method. For example, a grid point arranged at a predetermined interval in a three-dimensional area (in space) can be set for the query point. In this case, parameter values ​​such as the intervals between the grid points can be shared in advance between the encoder (e.g., the geometry encoding device 100) and the decoder (e.g., the geometry decoding device 200). In addition, the parameter value can be sent from the encoder to the decoder, where the parameter value is associated with the bitstream. In addition, for example, the geometry generation unit 213 can set the query point. The method of setting the query point is arbitrary. For example, the geometry generation unit 213 can set the query point according to a condition. For example, the geometry generation unit 213 can construct a query point only for the target area.

[0167] Then, the geometry generation unit 213 derives the occupancy probability f for all query points x θ (x local :z), where z represents the parameter vector of the block to which x belongs, and x local represents the local coordinates in the block of x. That is, for each query point, the coordinates are used as the input to the coordinate-based network, the parameter vector is used as the conditioning vector, and the output is used as the occupancy probability. However, f θ represents a neural network (coordinate-based network), and θ represents a parameter. Then, the geometric shape generation unit 213 selects one of the query points whose occupancy probability value is high, and outputs information about the selected query point to the outside of the geometric shape decoding device 200. That is, the geometric shape generation unit 213 takes the position information about the query point as input, uses the coordinate-based network to which the parameter vector is applied, derives the occupancy probability of the query point, and outputs information about the query point whose occupancy probability is high.

[0168] The method of selecting the query point for outputting information is arbitrary. For example, the geometry generation unit 213 may select a query point whose occupancy probability value is greater than or equal to a predetermined threshold. Note that the threshold only needs to be shared in advance between the encoder (e.g., the geometry encoding device 100) and the decoder (e.g., the geometry decoding device 200). In addition, the threshold may be sent from the encoder to the decoder, where the threshold is associated with the bitstream.

[0169] Alternatively, the geometry generation unit 213 may select a predetermined number of query points in descending order of their occupancy probability values. Note that the predetermined number may be sent from the encoder to the decoder, where the predetermined number is associated with the bitstream.

[0170] As described above, the geometry decoding device 200 generates a parameter vector using 3D-CNN, and thus can parallelize the calculation process for each element. Therefore, compared with the case where the global DC component and the AC component are combined together as in the LAVC for the attribute, the geometry decoding device 200 can generate the parameter vector at a higher speed (i.e., can reduce the processing time for generating the parameter vector). In particular, in the case where the global DC component and the AC component are combined together, the processing time for generating the parameter vector increases as the number of blocks increases. On the other hand, the geometry decoding device 200 uses 3D-CNN, which makes the processing time substantially constant, regardless of the number of blocks. That is, even when the number of blocks increases, the geometry decoding device 200 can suppress the increase in the processing time for generating the parameter vector.

[0171] Note that in the foregoing description, the coordinates of the query point are used as inputs to the coordinate-based network; however, the parameters to be input are arbitrary and are not limited to this example. For example, in addition to the coordinates (x, y, z) of the query point, the geometry generation unit 213 can also input time information (t) to the coordinate-based network. In this case, the input corresponds to four-dimensional information (x, y, z, t). Therefore, the geometry generation unit 213 is able to generate (reconstruct) a geometry that changes in the time direction. That is, encoding and decoding of a geometry that changes in the time direction can be achieved.

[0172] Alternatively, in addition to the coordinates (x, y, z) of the query point, the geometry generation unit 213 may also input the sight direction to the coordinate-based network. In this case, the input corresponds to five-dimensional information Therefore, the geometric shape generation unit 213 can generate (reconstruct) a geometric shape having a corresponding relationship with the sight line direction. For example, the geometric shape generation unit 213 can generate (reconstruct) a geometric shape having a property (e.g., reflection, etc.) that varies according to the sight line direction. That is, encoding and decoding of a geometric shape having a property that varies according to the sight line direction can be achieved.

[0173] Of course, other parameters can be added to the input. Alternatively, multiple parameters can be added. For example, time information (t) and the direction of sight can be added. Add both to the input.

[0174] Furthermore, in the foregoing description, the occupancy probability is used as an output from the coordinate-based network; however, the parameter to be output is arbitrary and is not limited to this example. For example, the occupancy probability value may be obtained in a manner in which the coordinates of the query point, etc. are input into the coordinate-based network. That is, the geometry generation unit 213 may output the occupancy probability value of the query point whose occupancy probability is high. Alternatively, when the coordinates of the query point, etc. are input into the coordinate-based network, information in other formats such as a signed distance or an unsigned distance is obtained. The geometry generation unit 213 may use this information as the occupancy probability.

[0175] In addition, through the coordinate-based network, in addition to the occupancy probability, attributes can also be obtained. For example, the coordinate-based network can output a vector that associates the occupancy probability with the attribute. Therefore, the geometric shape generation unit 213 is able to generate (reconstruct) a geometric shape with attributes. That is, encoding and decoding of geometric shapes with attributes can be achieved.

[0176] Note that the parameter vector mesh generation unit 212 and the geometric shape generation unit 213 can be integrated into one to serve as the generation unit 221. That is, the generation unit 221 can use a feature vector representing a spatial correlation lower than the spatial correlation of a parameter vector of a coordinate-based network representing the geometric shape of the 3D data as an input, and generate (reconstruct) the geometric shape of the 3D data. At this time, the generation unit 221 can generate a parameter vector based on the feature vector, and generate a geometric shape using a coordinate-based network to which the parameter vector is applied. In this case, the generation unit 221 can generate a parameter vector using a 3D-CNN with a three-dimensional deconvolution layer with the feature vector as input. In addition, the generation unit 221 can use the position information about the query point as input, use the coordinate-based network to which the parameter vector is applied to derive the occupancy probability of the query point, and output information about the query point with a high occupancy probability. In this case, the generation unit 221 can output the occupancy probability value of the query point with a high occupancy probability. In addition, the generation unit 221 can set a query point and use the set query point as input. In addition, the generation unit 221 can further use time information as input. In addition, the generation unit 221 can also use the line of sight direction as input.

[0177] <Flow of geometric shape decoding processing>

[0178] As described above, the geometry decoding apparatus 200 performs a geometry decoding process to decode the coded data (ie, bit stream) of the geometry. Fig.11 The flowchart describes an example of the process of the geometry decoding process.

[0179] When the geometry decoding process starts, in step S201, the decoding unit 211 performs entropy decoding and inverse quantization on the bit stream to generate (reconstruct) a CNN feature vector grid. That is, the decoding unit 211 decodes the encoded data to generate a feature vector representing a lower spatial correlation than the spatial correlation of the parameter vector of the coordinate-based network representing the geometry of the 3D data.

[0180] In step S202, the parameter vector mesh generation unit 212 performs a parameter vector mesh generation process to generate (reconstruct) a parameter vector mesh from the CNN feature vector mesh using 3D-CNN_D1.

[0181] In step S203, the geometry generation unit 213 generates (reconstructs) a geometry using the parameter vector grid and the coordinate-based network. For example, the geometry generation unit 213 sets the query point as described above, obtains the occupancy probability of each query point using the position information about the query point as an input to the coordinate-based network and the parameter vector as a condition vector of the coordinate-based network, and generates the geometry of the 3D data (point cloud) using the position information about one of the query points whose occupancy probability is high.

[0182] When the processing in step S203 is completed, the geometry decoding processing is completed.

[0183] <Parameter Vector Mesh Generation Process>

[0184] Next, refer to Fig.12 Flow chart of Fig.11 An example of the flow of the parameter vector mesh generation process performed in step S202 shown in FIG.

[0185] When the parameter vector mesh generation process starts, in step S231, the parameter vector mesh generation unit 212 inputs Fig.11 The CNN feature vector grid generated in step S201 is shown.

[0186] In step S232, the parameter vector grid generation unit 212 initializes the four-dimensional array input using the CNN feature vector grid.

[0187] In step S233 , the parameter vector mesh generation unit 212 selects a processing target calculation layer of 3D-CNN_D1 .

[0188] In step S234, the parameter vector grid generation unit 212 initializes the four-dimensional array output.

[0189] In step S235, the parameter vector mesh generation unit 212 selects the processing object indexes i, j, and k of the output in the spatial direction from the unprocessed indexes in a predetermined order.

[0190] In step S236 , the parameter vector mesh generation unit 212 calculates the processing object indexes i, j, and k in the processing object calculation layer.

[0191] In step S237, the parameter vector mesh generation unit 212 determines whether all indexes have been processed. When there are unprocessed indexes, the parameter vector mesh generation process returns to step S236. Then various types of processing in and after step S236 are performed. That is, while updating the processing object indexes i, j, and k, various types of processing from step S235 to step S237 are repeatedly performed until all indexes are subjected to calculation in the processing object calculation layer.

[0192] When it is determined in step S237 that all the indices have been subjected to calculation in the processing target calculation layer, the parameter vector mesh generation process proceeds to step S238.

[0193] In step S238, the parameter vector mesh generation unit 212 determines whether all calculation layers have been processed. When it is determined that there is an unprocessed calculation layer, the parameter vector mesh generation process proceeds to step S239.

[0194] In step S239, the parameter vector mesh generation unit 212 uses the output (the calculation result in step S236) as input. When the processing in step S239 ends, the parameter vector mesh generation processing returns to step S233. That is, while updating the processing target calculation layer, various types of processing from step S233 to step S239 are repeatedly performed until calculations are performed in all calculation layers.

[0195] When it is determined in step S238 that calculation has been performed in all calculation layers, the parameter vector mesh generation process proceeds to step S240. In step S240, the parameter vector mesh generation unit 212 outputs the output (the calculation result in step S236) as a parameter vector mesh (i.e., supplies the output to the geometry generation unit 213). When the process in step S240 ends, the parameter vector mesh generation process ends. Then, the process flow returns to step S240. Fig.11 .

[0196] By performing various types of processing as described above, the geometry decoding device 200 generates a parameter vector using 3D-CNN, and thus is able to parallelize the calculation processing for each element. Therefore, compared to the case where the global DC component and the AC component are combined together as in the LAVC for the attribute, the geometry decoding device 200 is able to generate a parameter vector at a higher speed (i.e., the processing time for generating the parameter vector can be reduced). In particular, in the case where the global DC component and the AC component are combined together, the processing time for generating the parameter vector increases as the number of blocks increases. On the other hand, the geometry decoding device 200 uses 3D-CNN, which makes the processing time substantially constant, regardless of the number of blocks. That is, even when the number of blocks increases, the geometry decoding device 200 is able to suppress the increase in the processing time for generating the parameter vector.

[0197] <4. Generate parameter vector by optimizing Lagrangian function>

[0198] <Method 1-1-2>

[0199] Note that the parameter vector grid generation unit 111 can also derive the parameter vector by applying the optimization of the objective function in the same manner as in the case of LAVC for attributes, instead of using 3D-CNN_E1. This technique is generally referred to as an automatic decoder. For example, the parameter vector grid generation unit 111 can set an objective function that can be differentiated with respect to the parameter vector, and derive a parameter vector that minimizes (or maximizes) the objective function by gradient descent. An example of the objective function is a Lagrangian function to be used in LVAC (i.e., a function that outputs a weighted sum of an approximation of the bit size and a reconstruction distortion with a parameter vector as input).

[0200] That is to say, Figure 4 As shown in the fourth row from the top of the table, the geometry encoding device 100 can generate a parameter vector according to the geometry by optimizing the Lagrangian function, and use 3D-CNN to generate a CNN feature vector according to the parameter vector (method 1-1-2).

[0201] For example, in an information processing device, the information processing device includes: a generating unit that generates a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector based on a parameter vector of a coordinate-based network representing a geometric shape of 3D data; and an encoding unit that encodes the feature vector, and the generating unit can generate the parameter vector by optimizing a Lagrangian function. For example, in the geometric shape encoding device 100, the generating unit 121 (parameter vector grid generating unit 111) can generate the parameter vector by optimizing a Lagrangian function.

[0202] <Flow of geometric shape coding processing>

[0203] Reference Fig.13 An example of the flow of the geometry encoding processing in this case is described by a flowchart.

[0204] When the geometry encoding process starts, in step S301 , the parameter vector mesh generation unit 111 divides 3D data (eg, point cloud) into blocks.

[0205] In step S302 , the parameter vector grid generating unit 111 initializes the parameter vector grid.

[0206] In step S303, the parameter vector mesh generation unit 111 optimizes the parameter vector mesh by gradient descent to minimize the Lagrangian function (objective function). For example, the Lagrangian function is defined by the following equation (4).

[0207]

[0208] Note that Z with circumflex represents a parameter vector obtained by encoding and decoding V. Also in equation (4), D represents the distortion between the geometry obtained using Z (with circumflex) and the geometry input to the geometry encoding device 100. Also in equation (4), R represents an estimate of the bit size. Also in equation (4), λ represents a hyperparameter. Also in equation (4), θ represents a parameter of the neural network.

[0209] The parameter vector mesh generation unit 111 can obtain the parameter vector by minimizing the Lagrangian function.

[0210] When the parameter vector grid is obtained as described above, with Figure 8 The various types of processing in steps S101 and S102 shown are similarly performed in steps S304 and S305. When the processing in step S305 ends, the geometry encoding processing ends.

[0211] Also in this case, the geometry encoding device 100 is capable of encoding the geometry of the point cloud using a coordinate-based network. The geometry encoding device 100 is also capable of improving the overall encoding efficiency by encoding the geometry. At this point, it is expected that the geometry encoding device 100 improves compression performance by directly optimizing the bit size and reconstruction distortion. The geometry encoding device 100 is also capable of improving encoding efficiency by reducing spatial correlation. The geometry encoding device 100 is also capable of encoding feature vectors in parallel while suppressing a reduction in encoding efficiency by reducing spatial correlation. Therefore, the geometry encoding device 100 is capable of performing encoding processing at a higher speed (i.e., the processing time can be reduced).

[0212] <5. Application of sparse 3D convolution>

[0213] <Method 1-2>

[0214] In the above 3D-CNN, when performing a normal 3D convolution calculation, an empty block with a point number of 0 is also subjected to a 3D convolution calculation, so that a parameter vector and a CNN feature vector are derived. Therefore, when the input geometry is sparse, there is a possibility that the coding efficiency is reduced and unnecessary computational cost increases.

[0215] Therefore, 3D-CNN can use sparse 3D convolutional layers instead of 3D convolutional layers. That is, as Figure 4 As shown in the fifth row from the top of the table, a 3D-CNN with a sparse 3D convolutional layer can be used to generate a CNN feature vector (Method 1-2).

[0216] For example, in an information processing device, the information processing device includes: a generation unit that generates a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector based on a parameter vector of a coordinate-based network representing a geometric shape of 3D data; and an encoding unit that encodes the feature vector, and the generation unit can use a 3D-CNN with a sparse three-dimensional convolutional layer to generate the feature vector. Alternatively, the generation unit can generate the parameter vector using a 3D-CNN with a sparse three-dimensional convolutional layer with a geometric shape as input, and generate the feature vector using the 3D-CNN with the parameter vector as input. In addition, the generation unit can generate the parameter vector using a 3D-CNN with a sparse three-dimensional convolutional layer with a geometric shape as input, and generate the feature vector using a 3D-CNN with a sparse three-dimensional convolutional layer with the parameter vector as input.

[0217] The sparse 3D convolution layer performs sparse 3D convolution calculations. According to the sparse 3D convolution calculations, 3D convolution calculations are performed on occupied blocks, and 3D convolution calculations for blocks that are considered empty blocks are skipped (omitted). Therefore, for example, when sparse geometric shapes are processed, a decrease in encoding efficiency and an increase in unnecessary computational cost can be suppressed.

[0218] <Geometric shape encoding device>

[0219] Fig.14 1 is a block diagram showing a main configuration example of the geometry encoding device 100 in this case. Note that Fig.14 The main processing unit, data flow, etc. are shown; therefore, the geometric shape encoding device 100 is not limited to Fig.14 That is, the geometric shape encoding device 100 may include Fig.14 The processing units not shown in block form and may involve Fig.14 Processing or data flow not indicated by arrows etc.

[0220] Also in this case, the geometric shape encoding device 100 includes a parameter vector grid generation unit 111, a CNN feature vector grid generation unit 112 and an encoding unit 113.

[0221] The parameter vector mesh generation unit 111 (generation unit 121) can generate a parameter vector mesh using 3D-CNN_E1 with a sparse 3D convolution layer. In addition, the CNN feature vector mesh generation unit 112 (generation unit 121) can generate a CNN feature vector mesh using 3D-CNN_E2 with a sparse 3D convolution layer.

[0222] Note that the mask grid pairs {0, 1} (D×D×D) and the parameter vector grid R M×C Input to and output from a 3D-CNN with sparse 3D convolutional layers. Here, D represents the number of blocks on one side of the grid, C represents the number of dimensions of the input / output vectors, and M represents the number of occupied blocks of the mask grid (the total number of blocks that each take the value "1").

[0223] The mask grid refers to mapping information indicating the occupancy state for each block. For example, a value of "1" indicates that the relevant block is an occupied block (a block including a point), and a value of "0" indicates that the relevant block is an empty block. According to the sparse 3D convolution calculation, in the mask grid, the 3D convolution calculation is performed on the blocks regarded as occupied blocks, and the 3D convolution calculation for the blocks regarded as empty blocks is skipped (omitted).

[0224] For example, suppose in Fig.15 In the mask grid 321 shown, black squares indicate occupied blocks and white squares indicate empty blocks. The parameter vector grid generation unit 111 inputs the mask grid 321 to 3D-CNN_E1 to obtain Fig.15 Parameter vector grid 322 is shown. In parameter vector grid 322, gray blocks indicate blocks for which 3D convolution calculation is performed, and white blocks indicate blocks for which 3D convolution calculation is skipped. That is, parameter vectors are obtained only when occupied blocks in mask grid 321 are subjected to 3D convolution calculation.

[0225] As described above, pairs of mask grids and parameter vector grids are input to and output from a 3D-CNN with a sparse 3D convolution layer. That is, the parameter vector grid generation unit 111 (generation unit 121) can input the mask grid 321 and the geometry of the point cloud (voxel data) to 3D-CNN_E1. However, the mask grid is information indicating the position of the point and is equivalent to its geometry with reduced resolution (i.e., downscaled geometry). Therefore, 3D-CNN_E1 is able to perform sparse 3D convolution calculations by inputting the geometry of the point cloud (voxel data) alone. In this case as well, pairs of mask grids and parameter vector grids are output, in which only occupied blocks are subjected to 3D convolution calculations. That is, for example, it is obtained Fig.15 The mask grid 321 and parameter vector grid 322 are shown. The parameter vector grid generation unit 111 can supply the obtained mask grid and parameter vector grid to the CNN feature vector grid generation unit 112.

[0226] The CNN feature vector grid generation unit 112 (generation unit 121) can obtain the mask grid and the parameter vector grid, and input the mask grid and the parameter vector grid to the 3D-CNN_E2 with a sparse 3D convolution layer. Thus, a mask grid and a CNN feature vector grid are obtained, in which only occupied blocks are subjected to 3D convolution calculations. Note that for 3D-CNN_E2 in this case, the block size Bp of the parameter vector grid and the block size Bf of the CNN feature vector grid need to match each other. The CNN feature vector grid generation unit 112 (generation unit 121) can supply the mask grid and the CNN feature vector grid to the encoding unit 113.

[0227] In this case, the encoding unit 113 includes a CNN feature vector grid encoding unit 301 and a mask grid encoding unit 302, such as Fig.14 shown.

[0228] The CNN feature vector grid encoding unit 301 performs processing related to the encoding of the CNN feature vector grid. For example, the CNN feature vector grid encoding unit 301 can obtain the CNN feature vector grid supplied from the CNN feature vector grid generation unit 112 (i.e., the CNN feature vector grid in which only the occupied blocks are subjected to 3D convolution calculation). In addition, the CNN feature vector grid encoding unit 301 can encode the CNN feature vector grid. For example, the CNN feature vector grid encoding unit 301 can quantize and entropy encode the CNN feature vector grid. In addition, the CNN feature vector grid encoding unit 301 outputs the generated bit stream (encoded data of the CNN feature vector grid) to the outside of the geometry encoding device 100. Hereinafter, the bit stream is also referred to as a parameter bit stream. The parameter bit stream (encoded data of the CNN feature vector grid) can be provided to a decoding device via any transmission path or using any recording medium.

[0229] Note that the CNN feature vector grid encoding unit 301 can be implemented by applying Figure 5 The CNN feature vector grid is encoded by a method similar to the method of the encoding unit 113 in the case shown. That is, the aforementioned description of the encoding of the CNN feature vector grid by the encoding unit 113 in <3. Encoding geometric shapes using a coordinate-based network> or <4. Generating parameter vectors by optimizing the Lagrangian function> can be applied to the encoding of the CNN feature vector grid by the CNN feature vector grid encoding unit 301.

[0230] The mask grid encoding unit 302 performs processing related to the encoding of the mask grid. For example, the mask grid encoding unit 302 can obtain the mask grid supplied from the CNN feature vector grid generation unit 112. In addition, the mask grid encoding unit 302 can encode the mask grid to generate encoded data (i.e., a bitstream) of the mask grid. For example, the mask grid encoding unit 302 can perform lossless encoding by treating the mask grid as a voxel without attributes. For example, the mask grid encoding unit 302 can perform encoding by a geometric lossless encoding method that complies with the geometry-based point cloud compression (G-PCC) standard (see, for example, "G-PCC Future Enhancements", ISO / IEC 23090-9:2019 (E), ISO / IEC JTC 1 / SC 29 / WG 11N18887, 2019-12-20). The mask grid encoding unit 302 outputs the generated bitstream (encoded data of the mask grid) to the outside of the geometry encoding device 100. Note that, hereinafter, this bit stream is also referred to as a mask bit stream. This mask bit stream (encoded data of the mask grid) can be provided to a decoding device via an arbitrary transmission path or using an arbitrary recording medium having this mask bit stream associated with a parameter bit stream.

[0231] When sparse geometry is a processing target, by applying the 3D-CNN with a sparse 3D convolutional layer as described above, the geometry encoding device 100 is able to suppress a decrease in encoding efficiency and an increase in unnecessary computational cost.

[0232] <Flow of geometric shape coding processing>

[0233] Reference Fig.16 , describing the geometry encoding process in this case (to be performed by Fig.14 An example of a process of geometric shape encoding processing) performed by the geometric shape encoding device 100 shown.

[0234] When the geometry encoding process starts, in step S331, the parameter vector mesh generation unit 111 generates a parameter vector mesh (and a mask mesh) according to the geometry (and a mask mesh) using 3D-CNN_E1 that performs sparse 3D convolution calculations.

[0235] In step S332, the CNN feature vector grid generation unit 112 performs CNN feature vector grid generation processing to generate a CNN feature vector grid and a mask grid based on the parameter vector grid and the mask grid using 3D-CNN_E2 that performs sparse 3D convolution calculations.

[0236] In step S333, the CNN feature vector grid encoding unit 301 quantizes and entropy encodes the CNN feature vector grid to generate encoded data (ie, parameter bitstream) of the CNN feature vector grid.

[0237] In step S334 , the mask grid encoding unit 302 encodes the mask grid to generate encoded data of the mask grid (ie, a mask bitstream).

[0238] When the processing in step S334 is completed, the geometric shape encoding processing is completed.

[0239] By performing various types of processing as described above, the geometry encoding device 100 is able to skip 3D convolution calculations for empty blocks, and when sparse geometry is a processing object, the geometry encoding device 100 is able to suppress a decrease in encoding efficiency and an increase in unnecessary computational costs.

[0240] <Method 1-2>

[0241] The above methods 1-2 can be applied to a decoder. That is, a sparse 3D deconvolution layer can be used instead of a 3D deconvolution layer. For example, in an information processing device, the information processing device includes: a decoding unit that decodes the encoded data to generate a feature vector representing a spatial correlation lower than the spatial correlation of a parameter vector of a coordinate-based network representing a geometric shape of the 3D data; and a generating unit that generates a geometric shape based on the feature vector, and the generating unit can use a 3D-CNN with a sparse 3D deconvolution layer to generate a parameter vector using the feature vector as input.

[0242] The sparse 3D deconvolution layer performs sparse 3D deconvolution calculation. According to the sparse inverse 3D convolution calculation, the 3D deconvolution calculation is performed on the occupied block, and the 3D deconvolution calculation for the block regarded as the empty block is skipped (omitted). Therefore, for example, when a sparse geometric shape is a processing object, a decrease in coding efficiency and an increase in unnecessary computational cost can be suppressed.

[0243] <Geometric shape decoding device>

[0244] Fig.17 2 is a block diagram showing a main configuration example of the geometry decoding device 200 in this case. Note that Fig.17 The main processing unit, data flow, etc. are shown; therefore, the geometry decoding device 200 is not limited to Fig.17 That is, the geometry decoding device 200 may include Fig.17 The processing units not shown in block form and may involve Fig.17 Processing or data flow not indicated by arrows etc.

[0245] Also in this case, the geometry decoding device 200 includes a decoding unit 211 , a parameter vector mesh generation unit 212 , and a geometry generation unit 213 .

[0246] like Fig.17 As shown, the decoding unit 211 includes a CNN feature vector grid decoding unit 351 and a mask grid decoding unit 352.

[0247] The CNN feature vector grid decoding unit 351 performs processing related to the decoding of the encoded data (i.e., parameter bitstream) of the CNN feature vector grid. For example, the CNN feature vector grid decoding unit 351 can obtain a parameter bitstream to be input to the geometry decoding device 200. In addition, the CNN feature vector grid decoding unit 351 can decode the parameter bitstream. For example, the CNN feature vector grid decoding unit 351 can entropy decode the acquired parameter bitstream to generate (reconstruct) quantization levels, and can inverse quantize the quantization levels to generate (reconstruct) the CNN feature vector grid. At this time, the CNN feature vector grid decoding unit 351 uses the same quantization width and probability table as the encoder (e.g., the CNN feature vector grid encoding unit 301). In addition, the CNN feature vector grid decoding unit 351 can supply the generated CNN feature vector grid to the parameter vector grid generation unit 212.

[0248] Note that the CNN feature vector grid decoding unit 351 can be decoded by applying Fig.10 The CNN feature vector grid is generated (reconstructed) in a similar manner to the method of the decoding unit 211 in the case shown. That is, the aforementioned description of the decoding of the encoded data of the CNN feature vector grid by the decoding unit 211 in <3. Encoding geometric shapes using a coordinate-based network> can be applied to the decoding of the encoded data of the CNN feature vector grid by the CNN feature vector grid decoding unit 351.

[0249] The mask grid decoding unit 352 performs processing related to the decoding of the encoded data of the mask grid. For example, the mask grid decoding unit 352 can obtain a mask bitstream to be input to the geometry decoding device 200. The mask bitstream can be supplied with the mask bitstream associated with, for example, a parameter bitstream. In addition, the mask grid decoding unit 352 can decode the mask bitstream to generate (reconstruct) a mask grid. At this time, the mask grid decoding unit 352 can perform decoding by a decoding method that has a corresponding relationship with the encoding method of the mask grid encoding unit 302. For example, the mask grid decoding unit 352 can perform decoding by a geometry decoding method that complies with G-PCC. The mask grid decoding unit 352 can supply the generated mask grid to the parameter vector grid generation unit 212.

[0250] The parameter vector grid generation unit 212 (generation unit 221) can obtain the mask grid and the CNN feature vector grid, and input the mask grid and the CNN feature vector grid to the 3D-CNN_D1 with a sparse 3D deconvolution layer. A mask grid and a parameter vector grid are thereby obtained, in which only occupied blocks are subjected to 3D deconvolution calculations. Note that for 3D-CNN_D1 in this case, the block size Bp of the parameter vector grid and the block size Bf of the CNN feature vector grid need to match each other. The parameter vector grid generation unit 212 can supply the mask grid and the parameter vector grid to the geometry generation unit 213.

[0251] The geometry generation unit 213 (generation unit 221) may obtain the mask grid and the parameter vector grid, and obtain the geometry of the 3D data according to the mask grid, the parameter vector grid, and the query point using a coordinate-based network representing the geometry of the 3D data. That is, in this case, the geometry generation unit 213 may transform the parameters into voxels only for the blocks in the mask grid that are considered as occupied blocks (i.e., blocks with a value of "1"). The geometry generation unit 213 may output the generated geometry to the outside of the geometry decoding device 200.

[0252] Note that the processing performed by the geometry generation unit 213 on this occupied block is the same as Fig.10 That is, the aforementioned description in <3. Encoding geometric shapes using a coordinate-based network> can be applied to this case.

[0253] When sparse geometry is the object of processing, by applying the 3D-CNN with a sparse 3D deconvolution layer as described above, the geometry decoding device 200 is able to suppress a decrease in encoding efficiency and an increase in unnecessary computational cost.

[0254] <Flow of geometric shape decoding processing>

[0255] Reference Fig.18 , which describes the geometry decoding process in this case (to be performed by Fig.17 An example of a process of geometric shape decoding processing) performed by the geometric shape decoding device 200 shown.

[0256] When the geometry decoding process starts, in step S361, the CNN feature vector grid decoding unit 351 performs entropy decoding and inverse quantization on the parameter bit stream to generate (reconstruct) a CNN feature vector grid.

[0257] In step S362 , the mask grid decoding unit 352 decodes the mask bit stream to generate (reconstruct) a mask grid.

[0258] In step S363, the parameter vector mesh generation unit 212 performs parameter vector mesh generation processing to generate (reconstruct) a parameter vector mesh from the mask mesh and the CNN feature vector mesh using 3D-CNN_D1 that performs sparse 3D deconvolution.

[0259] In step S364, the geometry generation unit 213 generates (reconstructs) a geometry using the mask grid, the parameter vector grid, and the coordinate-based network. For example, the geometry generation unit 213 sets the query point as described above, obtains the occupancy probability of each query point using the position information about the query point as the input to the coordinate-based network and the parameter vector as the condition vector of the coordinate-based network, and generates the geometry of the 3D data (point cloud) using the position information about one of the query points whose occupancy probability is high. At this time, the geometry generation unit 213 performs this processing only on the blocks in the mask grid that are regarded as occupied blocks (i.e., blocks that take the value of "1") to generate the geometry.

[0260] When the processing in step S364 is completed, the geometry decoding processing is completed.

[0261] When a sparse geometry is a processing target, by performing various types of processing as described above, the geometry decoding device 200 is able to suppress a decrease in encoding efficiency and an increase in unnecessary calculation cost.

[0262] <6. Encoding CNN feature vectors using mask grids>

[0263] <Method 1-2-1>

[0264] At the same time, the mask grid is equivalent to the downscaled input geometry. That is, it can be said that the mask grid is information indicating the rough geometry. Therefore, for example, information about the shape of the detailed geometry in each block can be obtained based on the rough geometry. In other words, there is a possibility of obtaining information about the parameter vector and the CNN feature vector based on the mask grid. It is expected that the coding efficiency of the CNN feature vector grid can be improved by encoding the CNN feature vector grid using this information. In other words, it can be said that for methods that do not use this information to encode the CNN feature vector grid, there is room for improving the coding efficiency of the CNN feature vector grid.

[0265] Therefore, when applying methods 1-2, the mask grid can be used to encode the CNN feature vector grid, such as Figure 4 As shown in the bottom row of the table (Method 1-2-1).

[0266] <Derivation of predicted values>

[0267] For example, a predicted value of a CNN feature vector can be obtained based on a mask grid, and the prediction residual can be encoded. For example, in an information processing device, the information processing device includes: a generation unit that generates a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector based on a parameter vector of a coordinate-based network representing the geometry of 3D data; and an encoding unit that encodes the feature vector, the encoding unit can derive a predicted value of the feature vector based on a mask grid used in a 3D-CNN, derive a prediction residual as a difference between the feature vector and the predicted value, and encode the prediction residual.

[0268] exist Fig.14 In the illustrated geometry encoding device 100, in this case, the CNN feature vector grid generation unit 112 also supplies the generated mask grid to the mask grid encoding unit 302. The mask grid encoding unit 302 encodes the mask grid in a manner similar to the case described above in <5. Application of sparse 3D convolution>.

[0269] However, in this case, the CNN feature vector grid generation unit 112 may supply the generated CNN feature vector grid and the mask grid to the CNN feature vector grid encoding unit 301. The CNN feature vector grid encoding unit 301 may acquire the CNN feature vector grid and the mask grid. Then, the CNN feature vector grid encoding unit 301 may obtain the predicted value (i.e., the predicted vector) of the CNN feature vector based on the acquired mask grid. Additionally, the CNN feature vector grid encoding unit 301 may obtain the prediction residual as the difference between the CNN feature vector and the predicted vector (i.e., the difference vector). Furthermore, the CNN feature vector grid encoding unit 301 may encode the difference vector (the difference vector grid). For example, the CNN feature vector grid encoding unit 301 may perform quantization and entropy encoding on the difference vector grid.

[0270] <Flow of CNN Feature Vector Grid Encoding Process>

[0271] In this case, the geometry encoding device 100 ( Fig.14 ) performs the CNN feature vector grid encoding process in step S333 of the geometry encoding process ( Fig.16 ).

[0272] Refer to Fig.19 for an example of the flow of the CNN feature vector grid encoding process described by the flowchart.

[0273] When the CNN feature vector grid encoding process starts, in step S401, the CNN feature vector grid encoding unit 301 generates a predicted vector grid using the 3D-CNN_D2 that performs sparse 3D convolution according to the mask grid. For example, the CNN feature vector grid encoding unit 301 inputs the mask grid into the 3D-CNN_D2 to obtain the predicted vector grids h'0,..., h' M-1 . Note that M represents the total number of occupied blocks in the mask grid. Additionally, regarding the 3D-CNN_D2, the input needs to be one-dimensional, the output needs to be Cf-dimensional, and the downscaling factor needs to be 1. Additionally, regarding the 3D-CNN_D2, the CNN feature vector h, the predicted vector h', and the difference vector δh each need to be Cf-dimensional.

[0274] In step S402, the CNN feature vector grid encoding unit 301 generates a difference vector grid using the CNN feature vector grid and the predicted vector grid. For example, for each CNN feature vector h, the CNN feature vector grid encoding unit 301 obtains the difference vector δh (= h - h') of the predicted vector h' and generates the difference vector grid.

[0275] In step S403, the CNN feature vector grid encoding unit 301 performs quantization and entropy encoding on the difference vector grid.

[0276] When the processing in step S403 ends, the CNN feature vector grid encoding process ends. Then, the processing flow returns to Fig.16 .

[0277] By obtaining the prediction vector using the mask grid as described above, if the prediction accuracy of the prediction vector is high, the geometric shape encoding device 100 can improve the encoding efficiency of the CNN feature vector grid.

[0278] Note that the 3D-CNN_E2 described as being used to generate the CNN feature vector can be configured to output the differential vector grid δh0,..., δh M-1 . In this case, the details of the processing to be performed by the geometric shape encoding device 100 are similar to those Fig.14 shown, but the meaning of the vector to be output is changed.

[0279] <Flow of CNN Feature Vector Grid Decoding Process>

[0280] Note that in this case, the CNN feature vector grid decoding unit 351 of the geometric shape decoding device 200 ( Fig.17 ) decodes the parameter vector to obtain the differential vector grid. Therefore, the CNN feature vector grid decoding unit 351 obtains the CNN feature vector grid from the differential vector grid. That is, in step S361 of the geometric shape decoding process ( Fig.18 ), the CNN feature vector grid decoding unit 351 performs the CNN feature vector grid decoding process to obtain the CNN feature vector grid from the differential vector grid.

[0281] Refer to Fig. 20 for an example of the flow of the CNN feature vector grid decoding process described by the flowchart.

[0282] When the CNN feature vector grid decoding process starts, in step S431, the CNN feature vector grid decoding unit 351 uses the 3D-CNN_D2 that performs sparse 3D convolution to generate a prediction vector grid according to the mask grid. For example, the CNN feature vector grid decoding unit 351 inputs the mask grid into the 3D-CNN_D2 to obtain the prediction vector grid h'0,..., h' M-1 .

[0283] In step S432, the CNN feature vector grid decoding unit 351 performs entropy decoding and inverse quantization on the parameter bitstream. For example, the CNN feature vector grid decoding unit 351 decodes and inversely quantizes the parameter bitstream to generate the (reconstructed) differential vector grid δh0,..., δh M-1 .

[0284] In step S433, the CNN feature vector grid decoding unit 351 generates a CNN feature vector grid using the difference vector grid and the prediction vector grid. For example, the CNN feature vector grid decoding unit 351 adds the prediction vector grid and the difference vector grid together (h = h' + δh) to obtain the CNN feature vector grids h0, ..., h M-1 .

[0285] When the processing in step S433 is completed, the CNN feature vector grid decoding process is completed. Then, the processing flow returns to Fig.18 .

[0286] By using the mask grid to obtain the prediction vector as described above, if the prediction accuracy of the prediction vector is high, the geometric shape decoding device 200 can improve the encoding efficiency of the CNN feature vector grid.

[0287] <Derivation of probability distribution>

[0288] In addition, for example, a mask grid can be used to derive a probability distribution of an entropy model to be applied to the encoding of a CNN feature vector. For example, in an information processing device, the information processing device includes: a generation unit that generates a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector based on a parameter vector of a coordinate-based network representing the geometry of 3D data; and an encoding unit that encodes the feature vector, and the encoding unit can obtain parameters of an entropy model of the feature vector based on the mask grid used in the 3D-CNN, and entropy encode the feature vector using the parameters.

[0289] Also in this case, the CNN feature vector grid generation unit 112 can supply the generated CNN feature vector grid and mask grid to the CNN feature vector grid encoding unit 301. The CNN feature vector grid encoding unit 301 can obtain the CNN feature vector grid and the mask grid. Then, the CNN feature vector grid encoding unit 301 can input the acquired mask grid into 3D-CNN_D3, thereby deriving the parameters of the probability distribution of the entropy model to be applied to the encoding of the CNN feature vector (for example, the mean and variance in the case of a Gaussian distribution). In this case, with respect to 3D-CNN_D3, the input needs to be one-dimensional, the output needs to be Cf×{the number of parameters in the distribution}-dimensional, and the downscaling factor needs to be 1. In addition, in the case of a Gaussian distribution, the number of parameters is represented by the mean and variance.

[0290] Then, the CNN feature vector grid encoding unit 301 applies the probability distribution with the obtained parameters to the entropy model of the CNN feature vector to encode the CNN feature vector. Therefore, the shape of the geometric shape (the distribution state of the point) can be reflected in the variance, etc.; therefore, an improvement in coding efficiency can be expected.

[0291] <7. Multi-resolution CNN feature vector>

[0292] <Correlation of multiple scales>

[0293] For example, in Fig.21 2×2 block correlations 511 (correlations with small scales) and 3×3 block correlations 512 (correlations with large scales) appear in the parameter vector grid 501 shown. As described above, there can be correlations with multiple scales as correlations between parameter vectors of a coordinate-based network. However, for methods using a single resolution CNN feature vector grid as in the previous example, it is difficult to reduce correlations with different multiple scales at the same time.

[0294] For example, when the resolution of the CNN feature vector grid is set to low (coarse), the encoder is expected to be able to reduce the correlation with the large scale. However, in this case, during the downscaling process, the detailed information about the high resolution originally present in the parameter vector grid is partially lost. Therefore, there is a possibility that it is difficult for the decoder to reconstruct the detailed information based on the CNN feature vector grid. This may lead to an increase in decoding distortion.

[0295] On the other hand, when the resolution of the CNN feature vector grid is set to high (fine), the encoder is expected to be able to reduce the correlation with small scales. However, in this case, for example, in order to allow the decoder to reconstruct the parameter vector grid based on the CNN feature vector grid, the kernel size of the 3D convolution needs to be set to high. For example, it is necessary to change the kernel size of the 3D convolution layer of the 3D-CNN in each of the encoder and the decoder from 3×3×3 to 5×5×5. However, the 3D convolution calculation amount is proportional to the cube of the kernel size; therefore, in this case, there is a possibility of increased calculation amount. For example, in the case where the kernel size of the 3D convolution layer of the 3D-CNN in each of the encoder and the decoder is changed from 3×3×3 to 5×5×5 as described above, there is a possibility that the calculation amount increases to about (5^3) / (3^3)=about 4.6 times.

[0296] <Method 1-3, Method 1-3-1>

[0297] Therefore, if Fig. 22 As shown in the top row of the table, CNN feature vectors of multiple resolutions (i.e., multiple CNN feature vectors (grids) of different resolutions) can be generated (methods 1-3). For example, when methods 1-3 are applied, as from Fig. 22 As shown in the second row from the top of the table, a high-resolution CNN feature vector (grid) and a low-resolution CNN feature vector (grid) can be generated (Method 1-3-1).

[0298] The resolution of the CNN feature vector grid depends on the downscaling factor of the 3D-CNN in the encoder and the upscaling factor of the 3D-CNN in the decoder. That is, in order to use CNN feature vector grids of multiple resolutions, the 3D-CNN in the encoder only needs to be expanded to have multiple downscaling factors, and the 3D-CNN in the decoder only needs to be expanded to have multiple upscaling factors.

[0299] Note that, for example, in the case of using CNN feature vector grids of two resolutions, high resolution and low resolution, the high resolution CNN feature vector grid can be considered to correspond to the AC component vector, and the low resolution CNN feature vector grid corresponds to the DC component vector.

[0300] For example, in an information processing device, the information processing device includes: a generating unit that generates a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector based on a parameter vector of a coordinate-based network representing a geometric shape of 3D data; and an encoding unit that encodes the feature vector, and the generating unit can generate a first feature vector as a feature vector of a first resolution and a second feature vector as a feature vector of a second resolution. In addition, the encoding unit can encode the first feature vector and the second feature vector. Note that the second resolution can be higher than the first resolution.

[0301] In this case, the generating unit may generate the first feature vector using the first 3D-CNN and generate the second feature vector using a second 3D-CNN different from the first 3D-CNN. Note that the three-dimensional convolutional layer of the first 3D-CNN may be different from the three-dimensional convolutional layer of the second 3D-CNN.

[0302] The spatial correlation with large scales can be reduced by using a low-resolution CNN feature vector grid. The spatial correlation with small scales can also be reduced by using a high-resolution CNN feature vector grid. In other words, by using CNN feature vector grids of multiple resolutions, the correlation of multiple resolutions can be reduced while suppressing the increase in computational complexity.

[0303] <Geometric shape encoding device>

[0304] Fig.23 1 is a block diagram showing a main configuration example of the geometry encoding device 100 in this case. Note that Fig.23 The main processing unit, data flow, etc. are shown; therefore, the geometric shape encoding device 100 is not limited to Fig.23 That is, the geometric shape encoding device 100 may include Fig.23 The processing units not shown in block form and may involve Fig.23 Processing or data flow not indicated by arrows etc.

[0305] In this case, as in Figure 5 In the case shown in FIG. 1 , the geometric shape coding device 100 includes a parameter vector grid generation unit 111 (generation unit 121). The geometric shape coding device 100 also includes a CNN feature vector grid generation unit 112-1 (generation unit 121) and a CNN feature vector grid generation unit 112-2 (generation unit 121), instead of having Figure 5 The CNN feature vector grid generation unit 112 of the configuration shown. The geometric shape encoding device 100 also includes an encoding unit 113-1 and an encoding unit 113-2, instead of having Figure 5 The encoding unit 113 has the configuration shown.

[0306] In this case, the parameter vector mesh generation unit 111 supplies the generated parameter vector mesh to each of the CNN feature vector mesh generation unit 112 - 1 and the CNN feature vector mesh generation unit 112 - 2 .

[0307] Each of the CNN feature vector grid generation unit 112-1 and the CNN feature vector grid generation unit 112-2 is Figure 5 The CNN feature vector grid generation unit 112 shown in FIG. 1 is a similar processing unit. In other words, Figure 5 Each of the illustrated CNN feature vector grid generation unit 112 - 1 and CNN feature vector grid generation unit 112 - 2 and CNN feature vector grid generation unit 112 has a similar configuration and performs similar processing.

[0308] However, the CNN feature vector grid generation unit 112-1 and the CNN feature vector grid generation unit 112-2 generate CNN feature vector grids of different resolutions. For example, the CNN feature vector grid generation unit 112-1 generates a CNN feature vector grid of low resolution (first resolution), and the CNN feature vector grid generation unit 112-2 generates a CNN feature vector grid of high resolution (second resolution). Note that there is no particular limitation on the value of "low resolution" and the value of "high resolution", as long as "low resolution" is lower than "high resolution".

[0309] The CNN feature vector mesh generation unit 112-1 can use 3D-CNN (3D-CNN_E2_L) to generate a low-resolution CNN feature vector from the parameter vector mesh (parameter vector maintained in a mesh shape) supplied from the parameter vector mesh generation unit 111. For example, the CNN feature vector mesh generation unit 112-1 inputs the 3D-CNN_E2_L Fig.24 The parameter vector grid 531 shown is used to transform the parameter vector grid 531 into a low-resolution CNN feature vector grid 533. In addition, the CNN feature vector grid generation unit 112-1 can supply the generated low-resolution CNN feature vector grid to the encoding unit 113-1.

[0310] In the case where the CNN feature vector has dimension Cf and the CNN feature vector grid has a block size Bf, for 3D-CNN_E2_L, the input dimension needs to be Cp, the output dimension needs to be Cf, and the downscaling factor needs to be Bp / Bf in order to account for nonlinear spatial correlation. In addition, 3D-CNN_E2_L has at least two 3D convolutional layers and one nonlinear transformation layer. In addition, at least one of the 3D convolutional layers has a kernel size greater than 1. As long as this condition is met, both the hyperparameters and the structure have degrees of freedom.

[0311] The CNN feature vector mesh generation unit 112-2 can generate a high-resolution CNN feature vector using 3D-CNN (3D-CNN_E2_H) from the parameter vector mesh (parameter vector maintained in a mesh shape) supplied from the parameter vector mesh generation unit 111. For example, the CNN feature vector mesh generation unit 112-2 inputs the 3D-CNN_E2_H Fig.24 The parameter vector grid 531 shown is used to transform the parameter vector grid 531 into a high-resolution CNN feature vector grid 532. In addition, the CNN feature vector grid generation unit 112-2 can supply the generated high-resolution CNN feature vector grid to the encoding unit 113-2.

[0312] In the case where the CNN feature vector has dimension Cf and the CNN feature vector grid has a block size Bf, for 3D-CNN_E2_H, the input dimension needs to be Cp, the output dimension needs to be Cf, and the downscaling factor needs to be Bp / Bf in order to account for nonlinear spatial correlation. In addition, 3D-CNN_E2_H has at least two 3D convolutional layers and one nonlinear transformation layer. In addition, at least one of the 3D convolutional layers has a kernel size greater than 1. As long as this condition is met, both the hyperparameters and the structure have degrees of freedom.

[0313] However, the downscaling factor of 3D-CNN_E2_L is smaller than the downscaling factor of 3D-CNN_E2_H. The downscaling factor of 3D-CNN_E2_L and the downscaling factor of 3D-CNN_E2_H can be determined by any method. For example, a combination of downscaling factors that exhibit the maximum combined performance in the dataset used for training can be selected from a set of datasets used for training and multiple candidates for downscaling factors (the downscaling factor can be determined by a typical grid search).

[0314] Each of the encoding units 113-1 and 113-2 is Figure 5 The coding unit 113 shown in FIG. Figure 5 Each of the illustrated encoding units 113 - 1 and 113 - 2 and the encoding unit 113 has a similar configuration and performs similar processing.

[0315] The encoding unit 113-1 encodes the low-resolution CNN feature vector grid supplied from the CNN feature vector grid generation unit 112-1. For example, the encoding unit 113-1 may quantize the low-resolution CNN feature vector grid and entropy encode the quantization level, thereby generating encoded data (also referred to as a low-resolution bitstream) of the low-resolution CNN feature vector grid. The encoding unit 113-1 may quantize and entropy encode all values ​​of the low-resolution CNN feature vector grid independently of each other. At this time, the encoding unit 113-1 may apply a quantization width that is common in the spatial direction and different for each dimension to quantization. In addition, the encoding unit 113-1 may apply a probability table that is common in the spatial direction and different for each dimension to entropy encoding. The encoding unit 113-1 outputs the generated low-resolution bitstream to the outside of the geometry encoding device 100. The low-resolution bitstream can be provided to a decoding device via an arbitrary transmission path or using an arbitrary recording medium.

[0316] The encoding unit 113-2 encodes the high-resolution CNN feature vector grid supplied from the CNN feature vector grid generation unit 112-2. For example, the encoding unit 113-2 may quantize the high-resolution CNN feature vector grid and entropy encode the quantization level, thereby generating encoded data of the high-resolution CNN feature vector grid (also referred to as a high-resolution bitstream). The encoding unit 113-2 may quantize and entropy encode all values ​​of the high-resolution CNN feature vector grid independently of each other. At this time, the encoding unit 113-2 may apply a quantization width that is common in the spatial direction and different for each dimension to quantization. In addition, the encoding unit 113-2 may apply a probability table that is common in the spatial direction and different for each dimension to entropy encoding. The encoding unit 113-2 outputs the generated high-resolution bitstream to the outside of the geometry encoding device 100. The high-resolution bitstream can be provided to a decoding device via any transmission path or using any recording medium.

[0317] Note that the geometry encoding apparatus 100 may merge the low-resolution bitstream and the high-resolution bitstream into one bitstream.

[0318] In addition, encoding unit 113-1 and encoding unit 113-2 do not share the quantization width and probability table of the entropy model. This is because the distribution of values ​​(ie, the entropy model) may be different depending on the resolution of the CNN feature vector grid.

[0319] The geometric shape encoding device 100 having the aforementioned configuration can generate feature vectors of multiple resolutions each representing a spatial correlation lower than the spatial correlation of the parameter vector based on the parameter vector of the coordinate-based network representing the geometry of the 3D data. Therefore, the geometric shape encoding device 100 can reduce the correlation with multiple scales while suppressing the increase in the amount of calculation, and can also improve the encoding efficiency.

[0320] <Flow of geometric shape coding processing>

[0321] Reference Fig.25 The flowchart describes an example of the flow of the geometry encoding process to be performed by the geometry encoding device 100 in this case.

[0322] When the geometry encoding process starts, in step S501, the parameter vector mesh generation unit 111 generates a parameter vector mesh according to the geometry using 3D-CNN_E1.

[0323] In step S502, the CNN feature vector mesh generation unit 112-1 performs CNN feature vector mesh generation processing to generate a low-resolution (first resolution) CNN feature vector mesh based on the parameter vector mesh using 3D-CNN_E2_L. That is, the CNN feature vector mesh generation unit 112-1 generates a low-resolution feature vector representing a lower spatial correlation than the spatial correlation of the parameter vector based on the parameter vector of the coordinate-based network representing the geometry of the 3D data. Note that according to Fig. 9 A similar process to the exemplary process shown is used to perform the CNN feature vector grid generation process in this case.

[0324] In step S503, the CNN feature vector mesh generation unit 112-2 performs CNN feature vector mesh generation processing to generate a high-resolution (second resolution) CNN feature vector mesh based on the parameter vector mesh using 3D-CNN_E2_H. That is, the CNN feature vector mesh generation unit 112-2 generates a high-resolution feature vector representing a lower spatial correlation than the spatial correlation of the parameter vector based on the parameter vector of the coordinate-based network representing the geometry of the 3D data. Note that according to Fig. 9 A similar process to the exemplary process shown is used to perform the CNN feature vector grid generation process in this case.

[0325] In step S504, the encoding unit 113-1 quantizes and entropy encodes the low-resolution CNN feature vector grid generated in step S502 to generate a low-resolution bitstream.

[0326] In step S505, the encoding unit 113-2 quantizes and entropy encodes the high-resolution CNN feature vector grid generated in step S503 to generate a high-resolution bitstream.

[0327] When the processing in step S505 is completed, the geometric shape encoding processing is completed.

[0328] By performing various types of processing as described above, the geometry encoding device 100 can reduce correlations with multiple scales while suppressing an increase in the amount of calculation, and can also improve encoding efficiency.

[0329] <Method 1-3, Method 1-3-1>

[0330] The aforementioned methods 1-3 and 1-3-1 can be applied to a decoder. For example, in an information processing device, the information processing device includes: a decoding unit that decodes the encoded data to generate a feature vector representing a spatial correlation lower than the spatial correlation of a parameter vector of a coordinate-based network representing the geometry of the 3D data; and a generating unit that generates a geometric shape based on the feature vector, the decoding unit can decode the first encoded data to generate a first feature vector as a feature vector of a first resolution, and can decode the second encoded data different from the first encoded data to generate a second feature vector as a feature vector of a second resolution. In addition, the generating unit can generate a transformation vector based on the first feature vector, generate an addition result vector by adding the second feature vector to the transformation vector for each element, and generate a geometric shape based on the addition result vector. In addition, the second resolution can be higher than the first resolution. In addition, in this case, the generating unit can generate a parameter vector based on the addition result vector, and generate a geometric shape using a coordinate-based network to which the parameter vector is applied.

[0331] In this case, the generating unit may generate a transformation vector using a first 3D-CNN with the first feature vector as input, and generate a parameter vector using a second 3D-CNN different from the first 3D-CNN with the addition result vector as input. Note that the three-dimensional convolutional layer of the first 3D-CNN may be different from the three-dimensional convolutional layer of the second 3D-CNN.

[0332] The spatial correlation with large scales can be reduced by using a low-resolution CNN feature vector grid. The spatial correlation with small scales can also be reduced by using a high-resolution CNN feature vector grid. In other words, by using CNN feature vector grids of multiple resolutions, the correlation of multiple resolutions can be reduced while suppressing the increase in computational complexity.

[0333] <Geometric shape decoding device>

[0334] Fig.26 2 is a block diagram showing a main configuration example of the geometry decoding device 200 in this case. Note that Fig.26 The main processing unit, data flow, etc. are shown; therefore, the geometry decoding device 200 is not limited to Fig.26 That is, the geometry decoding device 200 may include Fig.26 The processing units not shown in block form and may involve Fig.26 Processing or data flow not indicated by arrows etc.

[0335] In this case, the geometry decoding apparatus 200 includes a decoding unit 211-1, a decoding unit 211-2, a parameter vector mesh generation unit 212-1, a parameter vector mesh generation unit 212-2, an addition unit 551, and a geometry generation unit 213. Note that the parameter vector mesh generation unit 212-1, the parameter vector mesh generation unit 212-2, the addition unit 551, and the geometry generation unit 213 may be integrated into one to serve as the generation unit 221.

[0336] Each of the decoding unit 211-1 and the decoding unit 211-2 is Fig.10 The decoding unit 211 shown in FIG. Fig.10 Each of the illustrated decoding unit 211 - 1 and decoding unit 211 - 2 and decoding unit 211 has a similar configuration and performs similar processing.

[0337] The decoding unit 211-1 decodes the low-resolution bitstream supplied to the geometry decoding device 200 to generate (reconstruct) a low-resolution CNN feature vector grid, and supplies the low-resolution CNN feature vector grid to the parameter vector grid generation unit 212-1. At this time, the decoding unit 211-1 can entropy decode the acquired low-resolution bitstream to generate (reconstruct) quantization levels, and can inverse quantize the quantization levels to generate (reconstruct) a low-resolution CNN feature vector grid. That is, the decoding unit 211-1 can decode the encoded data to generate a low-resolution feature vector representing a lower spatial correlation than the spatial correlation of the parameter vector of the coordinate-based network representing the geometry of the 3D data. At this time, the decoding unit 211-1 uses the same quantization width and probability table as the quantization width and probability table used for the encoder (e.g., encoding unit 113-1).

[0338] The decoding unit 211-2 decodes the high-resolution bitstream supplied to the geometry decoding device 200 to generate (reconstruct) a high-resolution CNN feature vector grid, and supplies the high-resolution CNN feature vector grid to the parameter vector grid generation unit 212-2. At this time, the decoding unit 211-2 can entropy decode the acquired high-resolution bitstream to generate (reconstruct) quantization levels, and can inverse quantize the quantization levels to generate (reconstruct) a high-resolution CNN feature vector grid. That is, the decoding unit 211-2 can decode the encoded data to generate a high-resolution feature vector representing a lower spatial correlation than the spatial correlation of the parameter vector of the coordinate-based network representing the geometry of the 3D data. At this time, the decoding unit 211-2 uses the same quantization width and probability table as the quantization width and probability table used for the encoder (e.g., encoding unit 113-2).

[0339] Each of the parameter vector mesh generation unit 212-1 and the parameter vector mesh generation unit 212-2 is connected to Fig.10 The parameter vector grid generation unit 212 shown is similar to the processing unit. That is, Fig.10 Each of the illustrated parameter vector mesh generation unit 212 - 1 and the parameter vector mesh generation unit 212 - 2 and the parameter vector mesh generation unit 212 has a similar configuration and performs similar processing.

[0340] The parameter vector mesh generation unit 212-1 generates a transformation vector mesh from the low-resolution CNN feature vector mesh supplied from the decoding unit 211-1 using 3D-CNN (3D-CNN_D1_L). For example, the parameter vector mesh generation unit 212-1 generates a transformation vector mesh. Fig.24 The low-resolution CNN feature vector grid 533 shown is input to 3D-CNN_D1_L to transform the low-resolution CNN feature vector grid 533 into a transformation vector grid. The parameter vector grid generation unit 212-1 supplies the transformation vector grid to the addition unit 551.

[0341] Note that the number of input dimensions of 3D-CNN_D1_L is equal to the number of dimensions of the low-resolution CNN feature vector, and the number of output dimensions of 3D-CNN_D1_L is equal to the number of dimensions of the high-resolution CNN feature vector. In addition, the upscaling factor of 3D-CNN_D1_L is equal to the scaling ratio between the low-resolution CNN feature vector grid and the high-resolution CNN feature vector grid. For the purpose of upscaling, the geometry decoding device 200 uses a 3D deconvolution layer instead of a 3D convolution layer. In addition, 3D-CNN_D1_L has at least two 3D deconvolution layers and one nonlinear layer. At least one of the 3D deconvolution layers has a kernel size greater than 1. As long as this condition is met, both the hyperparameters and the structure have degrees of freedom.

[0342] The adding unit 551 adds the high-resolution CNN feature vector grid supplied from the decoding unit 211-2 to the transformation vector grid supplied from the parameter vector grid generating unit 212-1 for each element. Fig.24 The high-resolution CNN feature vector grid 532 shown is added to the transformation vector. Since 3D-CNN_D1_L has the aforementioned characteristics, the addition unit 551 is able to add the transformation vector grid to the high-resolution CNN feature vector grid for each element. The addition unit 551 supplies the addition result (also referred to as the addition result vector grid) to the parameter vector grid generation unit 212-2.

[0343] The parameter vector mesh generation unit 212-2 generates (reconstructs) a parameter vector mesh using the 3D-CNN (3D-CNN_D1_H) from the addition result vector mesh supplied from the addition unit 551. For example, the parameter vector mesh generation unit 212-2 inputs the addition result vector mesh to the 3D-CNN_D1_H to transform the addition result vector mesh into Fig.24 The parameter vector mesh 531 is shown. The parameter vector mesh generation unit 212-2 supplies the parameter vector mesh to the geometry generation unit 213.

[0344] Note that the number of input dimensions of 3D-CNN_D1_H is equal to the number of dimensions of the high-resolution CNN feature vector, and the number of output dimensions of 3D-CNN_D1_H is equal to the number of dimensions of the parameter vector grid. In addition, the upscaling factor of 3D-CNN_D1_H is equal to the scaling ratio between the high-resolution CNN feature vector grid and the parameter vector grid. For the purpose of upscaling, the geometry decoding device 200 uses a 3D deconvolution layer instead of a 3D convolution layer. In addition, 3D-CNN_D1_H has at least two 3D deconvolution layers and one nonlinear layer. At least one of the 3D deconvolution layers has a kernel size greater than 1. As long as this condition is met, both the hyperparameters and the structure have degrees of freedom.

[0345] As in Fig.10 In the illustrated case, the geometry generation unit 213 generates the geometry of the 3D data based on the parameter vector grid and query points supplied from the parameter vector grid generation unit 212-2 using a coordinate-based network representing the geometry of the 3D data, and outputs the geometry to the outside of the geometry decoding device 200.

[0346] The geometry decoding device 200 having the aforementioned configuration can generate a parameter vector using feature vectors of multiple resolutions each representing a spatial correlation lower than the spatial correlation of a parameter vector of a coordinate-based network representing the geometry of 3D data. Therefore, the geometry decoding device 200 can reduce the correlation with multiple scales while suppressing an increase in the amount of calculation, and can also improve encoding efficiency.

[0347] <Flow of geometric shape decoding processing>

[0348] Reference Fig. 27 The flowchart describes an example of the flow of the geometry decoding process to be performed by the geometry decoding device 200 in this case.

[0349] When the geometry decoding process starts, in step S551, the decoding unit 211-1 entropy decodes and inverse quantizes the low-resolution bitstream to generate (reconstruct) a low-resolution CNN feature vector grid. That is, the decoding unit 211-1 decodes the encoded data to generate a low-resolution feature vector representing a lower spatial correlation than the spatial correlation of the parameter vector of the coordinate-based network representing the geometry of the 3D data.

[0350] In step S552, the parameter vector mesh generation unit 212-1 performs parameter vector mesh generation processing to generate a transformation vector mesh from the low-resolution CNN feature vector mesh generated in step S551 using 3D-CNN_D1_L.

[0351] In step S553, the decoding unit 211-2 performs entropy decoding and inverse quantization on the high-resolution bitstream to generate (reconstruct) a high-resolution CNN feature vector grid. That is, the decoding unit 211-2 decodes the encoded data to generate a high-resolution feature vector representing a lower spatial correlation than the spatial correlation of the parameter vector of the coordinate-based network representing the geometry of the 3D data.

[0352] In step S554, the adding unit 551 adds the high-resolution CNN feature vector grid generated in step S553 and the transformation vector grid generated in step S552 for each element to generate an addition result vector grid.

[0353] In step S555, the parameter vector mesh generation unit 212-2 performs parameter vector mesh generation processing to generate (reconstruct) a parameter vector mesh from the addition result vector mesh generated in step S554 using 3D-CNN_D1_H.

[0354] In step S556, the geometry generation unit 213 generates (reconstructs) a geometry using the parameter vector mesh generated in step S555 and the coordinate-based network. For example, the geometry generation unit 213 sets query points as described above, obtains the occupancy probability of each query point using the position information about the query points as an input to the coordinate-based network and the parameter vector as a condition vector of the coordinate-based network, and generates the geometry of the 3D data (point cloud) using the position information about one of the query points whose occupancy probability is high.

[0355] When the processing in step S556 is completed, the geometry decoding processing is completed.

[0356] By performing various types of processing as described above, correlation with multiple scales can be reduced while suppressing an increase in the amount of calculation, and encoding efficiency can also be improved.

[0357] <Method 1-3-2>

[0358] When applying methods 1-3, such as Fig. 22 As shown in the third row from the top of the table, CNN feature vectors (grids) of N (N: 2 or a greater integer) resolutions can be generated (method 1-3-2). That is, CNN feature vectors of three or more types of resolutions can be used. In this case as well, the spatial correlation of the scales corresponding to each resolution can be reduced by using a CNN feature vector grid of resolutions. In other words, the correlation of multiple resolutions can be reduced while suppressing the increase in the amount of calculation.

[0359] <Geometric shape encoding device>

[0360] Fig.28 1 is a block diagram showing a main configuration example of the geometry encoding device 100 in this case. Note that Fig.28 The main processing unit, data flow, etc. are shown; therefore, the geometric shape encoding device 100 is not limited to Fig.28 That is, the geometric shape encoding device 100 may include Fig.28 The processing units not shown in block form and may involve Fig.28 Processing or data flow not indicated by arrows etc.

[0361] Similarly, in the case of generating CNN feature vector grids of N types of resolutions, the configuration of the geometric shape encoding device 100 is also the same as Fig.23 The situation shown is basically similar. That is, the geometry encoding device 100 includes a CNN feature vector grid generation unit 112 and an encoding unit 113 for each resolution. That is, the geometry encoding device 100 includes N CNN feature vector grid generation units 112 (CNN feature vector grid generation units 112-1 to 112-N) and N encoding units 113 (encoding units 113-1 to 113-N).

[0362] In this case, the parameter vector mesh generation unit 111 supplies the generated parameter vector mesh to each of the CNN feature vector mesh generation units 112 - 1 to 112 -N.

[0363] The CNN feature vector grid generation unit 112-1 uses the 3D-CNN (3D-CNN_E2_1) having a corresponding relationship with the first resolution, generates a CNN feature vector of the first resolution according to the parameter vector grid supplied from the parameter vector grid generation unit 111, and supplies the CNN feature vector of the first resolution to the encoding unit 113-1. The encoding unit 113-1 encodes (e.g., quantizes and entropy encodes) the CNN feature vector of the first resolution, and outputs the encoded data (first resolution bit stream) to the outside of the geometry encoding device 100.

[0364] The CNN feature vector grid generation unit 112-2 uses the 3D-CNN (3D-CNN_E2_2) having a corresponding relationship with the second resolution, generates a CNN feature vector of the second resolution according to the parameter vector grid supplied from the parameter vector grid generation unit 111, and supplies the CNN feature vector of the second resolution to the encoding unit 113-2. The encoding unit 113-2 encodes (e.g., quantizes and entropy encodes) the CNN feature vector of the second resolution, and outputs the encoded data (second resolution bit stream) to the outside of the geometry encoding device 100.

[0365] Similarly, the CNN feature vector grid generation unit 112-N uses a 3D-CNN (3D-CNN_E2_N) having a corresponding relationship with the Nth resolution, generates a CNN feature vector of the Nth resolution according to the parameter vector grid supplied from the parameter vector grid generation unit 111, and supplies the CNN feature vector of the Nth resolution to the encoding unit 113-N. The encoding unit 113-N encodes (e.g., quantizes and entropy encodes) the CNN feature vector of the second resolution, and outputs the encoded data (Nth resolution bit stream) to the outside of the geometry encoding device 100.

[0366] That is, the CNN feature vector grid generation unit 112-K (1≤K≤N) uses the 3D-CNN (3D-CNN_E2_K) having a corresponding relationship with the K-th resolution, generates a CNN feature vector of the K-th resolution according to the parameter vector grid supplied from the parameter vector grid generation unit 111, and supplies the CNN feature vector of the K-th resolution to the encoding unit 113-K. The encoding unit 113-K encodes (e.g., quantizes and entropy encodes) the CNN feature vector of the K-th resolution, and outputs the encoded data (K-th resolution bitstream) to the outside of the geometry encoding device 100.

[0367] The first to N-th resolutions are different resolutions from each other, and these resolutions increase from the first resolution toward the N-th resolution. That is, the first resolution is the lowest resolution, and the N-th resolution is the highest resolution. Therefore, the downscaling factor of 3D-CNN_E2_K (1≤K≤N) is smaller than the downscaling factor of 3D-CNN_E2_(K+1). Other features of 3D-CNN_E2_K are similar to Fig.23 Those in the exemplary case shown.

[0368] Therefore, according to Fig.29 The parameter vector grid 601 shown is used to generate, for example, a CNN feature vector grid 602 of a first resolution, a CNN feature vector grid 603 of a second resolution, and a CNN feature vector grid 604 of a third resolution. That is, CNN feature vector grids of three or more types of resolutions can be generated.

[0369] Therefore, even when there is correlation with more various scales, the geometric shape encoding device 100 can reduce the correlation with each scale. That is, the correlation of multiple resolutions can be reduced while suppressing the increase in the amount of calculation.

[0370] Note that Fig.28 As shown, various types of processing for resolution (various types of processing to be performed by CNN feature vector grid generation units 112-1 to 112-N and various types of processing to be performed by encoding units 113-1 to 113-N) can be performed independently of each other. Therefore, the geometric shape encoding device 100 can perform various types of processing for resolution in parallel with each other, for example.

[0371] <Flow of geometric shape coding processing>

[0372] Reference Fig.30 The flowchart describes an example of the flow of the geometry encoding process to be performed by the geometry encoding device 100 in this case.

[0373] When the geometry encoding process starts, in step S601, the parameter vector mesh generation unit 111 generates a parameter vector mesh according to the geometry using 3D-CNN_E1.

[0374] In step S602 , the generation unit 121 selects a processing target resolution (Kth resolution) from unprocessed resolutions.

[0375] In step S603, the CNN feature vector mesh generation unit 112-K performs CNN feature vector mesh generation processing to generate a CNN feature vector mesh of the processing object resolution (K-th resolution) according to the parameter vector mesh using 3D-CNN_E2_K (i.e., 3D-CNN_E2 according to the processing object resolution). That is, the CNN feature vector mesh generation unit 112-K generates a feature vector of the K-th resolution representing a spatial correlation lower than the spatial correlation of the parameter vector based on the parameter vector of the coordinate-based network representing the geometry of the 3D data. Note that according to Fig. 9 The CNN feature vector grid generation process in this case is performed using a process similar to the exemplary process shown.

[0376] In step S604, the encoding unit 113-K quantizes and entropy encodes the CNN feature vector grid of the processing target resolution (Kth resolution) generated in step S603 to generate a bit stream of the processing target resolution (Kth resolution bit stream).

[0377] In step S605, the generation unit 121 determines whether all resolutions have been processed. When there is an unprocessed resolution, the geometry encoding process returns to step S602. Then, various types of processing in and after step S602 are performed. That is, various types of processing from step S602 to step S605 are performed for each resolution. When it is determined in step S605 that all resolutions have been processed, the geometry encoding process ends.

[0378] By performing various types of processing as described above, the geometry encoding device 100 can reduce the correlation with multiple scales while suppressing an increase in the amount of calculation, even when there is correlation with more various scales, and can also improve encoding efficiency.

[0379] <Method 1-3-2>

[0380] The above method 1-3-2 can be applied to the decoder. That is, CNN feature vectors of three or more types of resolutions can also be used in the decoder. In this case, the spatial correlation of the scale corresponding to each resolution can also be reduced by the CNN feature vector grid of the resolution. In other words, the correlation of multiple resolutions can be reduced while suppressing the increase in the amount of calculation.

[0381] <Geometric shape decoding device>

[0382] Fig.31 2 is a block diagram showing a main configuration example of the geometry decoding device 200 in this case. Note that Fig.31 The main processing unit, data flow, etc. are shown; therefore, the geometry decoding device 200 is not limited to Fig.31 That is, the geometry decoding device 200 may include Fig.31 The processing units not shown in block form and may involve Fig.31 Processing or data flow not indicated by arrows etc.

[0383] Similarly, in the case of using CNN feature vector grids of N types of resolutions, the configuration of the geometric shape decoding device 200 is also the same as Fig.26 The situation shown is basically similar. That is, the geometry decoding device 200 includes a decoding unit 211 and a parameter vector grid generation unit 212 for each resolution. That is, the geometry encoding device 100 includes N decoding units 211 (decoding units 211-1 to 211-N) and N parameter vector grid generation units 212 (parameter vector grid generation units 212-1 to 212-N). The geometry decoding device 200 also includes (N-1) addition units 551 (addition units 551 to 551-(N-1)). Note that the parameter vector grid generation units 212-1 to 212-N, the addition units 551 to 551-(N-1) and the geometry generation unit 213 can be integrated into one to serve as a generation unit 221.

[0384] The decoding unit 211-1 decodes the first resolution bit stream supplied to the geometry decoding device 200 to generate (reconstruct) a CNN feature vector grid of the first resolution. The decoding unit 211-2 decodes the second resolution bit stream supplied to the geometry decoding device 200 to generate (reconstruct) a CNN feature vector grid of the second resolution. Similarly, the decoding unit 211-N decodes the Nth resolution bit stream supplied to the geometry decoding device 200 to generate (reconstruct) a CNN feature vector grid of the Nth resolution. That is, the decoding unit 211-K (1≤K≤KN) decodes the Kth resolution bit stream supplied to the geometry decoding device 200 to generate (reconstruct) a CNN feature vector grid of the Kth resolution.

[0385] At this time, for example, the decoding unit 211-K may entropy decode the acquired K-th resolution bitstream to generate (reconstruct) quantization levels, and may inversely quantize the quantization levels to generate (reconstruct) a CNN feature vector grid of the K-th resolution. In addition, the decoding unit 211-K uses the same quantization width and probability table as those used for the encoder (e.g., the encoding unit 113-K).

[0386] The decoding unit 211-1 supplies the generated CNN feature vector grid of the first resolution to the parameter vector grid generation unit 212-1. The decoding units 211-2 to 211-N each supply the generated CNN feature vector grid to the addition unit 551. That is, in the case of 2≤K, the decoding unit 211-K supplies the generated CNN feature vector grid of the Kth resolution to the addition unit 551-(K-1).

[0387] The parameter vector grid generating unit 212-1 generates a first transformation vector grid from the CNN feature vector grid of the first resolution supplied from the decoding unit 211-1 using 3D-CNN (3D-CNN_D1_1), and supplies the first transformation vector grid to the adding unit 551-1.

[0388] The addition unit 551-1 adds the CNN feature vector grid of the second resolution supplied from the decoding unit 211-2 to the first transformation vector grid for each element to generate a first addition result vector grid. The parameter vector grid generation unit 212-2 generates a second transformation vector grid based on the first addition result vector grid supplied from the addition unit 551-1 using 3D-CNN (3D-CNN_D1_2), and supplies the second transformation vector grid to the addition unit 551-2.

[0389] Hereinafter, similarly, the addition unit 551-K adds the CNN feature vector grid of the (K+1)th resolution supplied from the decoding unit 211-(K+1) to the Kth transform vector grid supplied from the parameter vector grid generation unit 212-K for each element to generate a Kth addition result vector grid. The parameter vector grid generation unit 212-(K+1) generates the (K+1)th transform vector grid based on the Kth addition result vector grid supplied from the addition unit 551-K using the 3D-CNN (3D-CNN_D1_(K+1)), and supplies the (K+1)th transform vector grid to the addition unit 551-(K+1).

[0390] Then, the addition unit 551-(N-1) adds the CNN feature vector grid of the Nth resolution supplied from the decoding unit 211-N to the (N-1)th transformation vector grid supplied from the parameter vector grid generation unit 212-(N-1) for each element to generate the (N-1)th addition result vector grid. The parameter vector grid generation unit 212-N generates the Nth transformation vector grid based on the (N-1)th addition result vector grid supplied from the addition unit 551-(N-1) using the 3D-CNN (3D-CNN_D1_N), and supplies the Nth transformation vector grid to the geometry generation unit 213.

[0391] Note that the input dimension of 3D-CNN_D1_K is equal to the dimension of the CNN feature vector of the Kth resolution, and the output dimension of 3D-CNN_D1_K is equal to the dimension of the CNN feature vector of the (K+1)th resolution. In addition, the upscaling factor of 3D-CNN_D1_K is equal to the scaling ratio between the CNN feature vector grid of the Kth resolution and the CNN feature vector grid of the (K+1)th resolution. Other features of 3D-CNN_D1_K are similar to Fig.26 Those in the exemplary case shown.

[0392] As in Fig.10 In the illustrated case, the geometry generation unit 213 generates the geometry of the 3D data based on the parameter vector grid and query points supplied from the parameter vector grid generation unit 212-N using a coordinate-based network representing the geometry of the 3D data, and outputs the geometry to the outside of the geometry decoding device 200.

[0393] Even in the case where there is correlation with more various scales, the geometric shape decoding device 200 having the aforementioned configuration can reduce the correlation with each scale. That is, the correlation of multiple resolutions can be reduced while suppressing an increase in the amount of calculation.

[0394] <Flow of geometric shape decoding processing>

[0395] Reference Fig.32 The flowchart describes an example of the flow of the geometry decoding process to be performed by the geometry decoding device 200 in this case.

[0396] When the geometry decoding process starts, in step S651, the decoding unit 211-1 performs entropy decoding and inverse quantization on the first resolution bitstream to generate (reconstruct) a CNN feature vector grid of the first resolution. That is, the decoding unit 211-1 decodes the encoded data to generate a feature vector of the first resolution having a lower spatial correlation than the spatial correlation of the parameter vector of the coordinate-based network representing the geometry of the 3D data.

[0397] In step S652, the parameter vector mesh generation unit 212-1 performs a parameter vector mesh generation process to generate a first transformation vector mesh from the CNN feature vector mesh of the first resolution generated in step S651 using 3D-CNN_D1_1.

[0398] In step S653 , the generation unit 221 selects the lowest resolution from the unprocessed resolutions as the processing target resolution (Kth resolution).

[0399] In step S654, the decoding unit 211-K performs entropy decoding and inverse quantization on the K-th resolution bitstream to generate (reconstruct) a CNN feature vector grid of the K-th resolution. That is, the decoding unit 211-K decodes the encoded data to generate a feature vector of the K-th resolution having a lower spatial correlation than the spatial correlation of the parameter vector of the coordinate-based network representing the geometry of the 3D data.

[0400] In step S655, the adding unit 551 adds the CNN feature vector grid of the Kth resolution generated in step S654 to the (K-1)th transformation vector grid for each element to generate a (K-1)th addition result vector grid.

[0401] In step S656 , the parameter vector mesh generation unit 212 -K performs a parameter vector mesh generation process to generate (reconstruct) a K-th transformation vector mesh from the (K−1)-th addition result vector mesh generated in step S655 using 3D-CNN_D1_K.

[0402] In step S657, the generation unit 221 determines whether all resolutions have been processed. When there is an unprocessed resolution, the geometry decoding process returns to step S653. Then, various types of processing in and after step S653 are performed. That is, various types of processing from step S653 to step S657 are performed for each resolution in order from low resolution to high resolution. Then, when it is determined in step S657 that all resolutions have been processed, the geometry decoding process proceeds to step S658.

[0403] In step S658, the geometry generation unit 213 uses the Nth transformation vector mesh generated as described above as a parameter mesh, and uses the parameter vector mesh and the coordinate-based network to generate (reconstruct) a geometry. For example, the geometry generation unit 213 sets the query point as described above, obtains the occupancy probability of each query point using the position information about the query point as an input to the coordinate-based network and the parameter vector as a condition vector of the coordinate-based network, and generates the geometry of the 3D data (point cloud) using the position information about one of the query points whose occupancy probability is high.

[0404] When the processing in step S658 is completed, the geometry decoding processing is completed.

[0405] Even in the case where there is correlation with more various scales, the geometric shape decoding device 200 can reduce the correlation with each scale by performing various types of processing as described above. That is, the correlation of multiple resolutions can be reduced while suppressing an increase in the amount of calculation.

[0406] <Method 1-3-3>

[0407] Note that when applying methods 1-3, if Fig. 22 As shown in the bottom row of the table, some computational layers in CNN can be shared between multiple resolutions. For example, 3D-CNN_E2_1, ..., and 3D-CNN_E2_N used by the CNN feature vector grid generation unit 112-1, ..., and CNN feature vector grid generation unit 112-N of the geometry encoding device 100 can share some computational layers.

[0408] Fig.33 This example is shown in Fig.33 In the exemplary case shown, 3D-CNN_E2 is used to generate a first resolution CNN feature vector grid, ..., and a third resolution CNN feature vector grid according to a parameter vector grid. In this case, the 3D-CNN_E2 that generates the first resolution CNN feature vector grid, the 3D-CNN_E2 that generates the second resolution CNN feature vector grid, and the 3D-CNN_E2 that generates the third resolution CNN feature vector grid share some calculation layers (layers [1], ..., and layers [L]). Therefore, the first resolution CNN feature vector grid, ..., and the third resolution CNN feature vector grid are generated by calculation via the shared calculation layer (layer [1], ..., and layer [L]).

[0409] Therefore, the geometric shape encoding device 100 is able to suppress the increase in computational costs.

[0410] <8. Implementation Method (Point Cloud Coding Device)>

[0411] The above-mentioned present technology is applicable to any device. For example, the present technology is also applicable to a point cloud encoding device for encoding point cloud data.

[0412] Fig.34 : is a block diagram showing an example of the configuration of a point cloud encoding device as one aspect of an information processing device to which the present technology is applied. Fig.34 The point cloud encoding device 700 shown is a device for encoding point cloud data. The point cloud encoding device 700 applies the above-mentioned present technology to encode point cloud data.

[0413] Notice, Fig.34 The main processing unit, data flow, etc. are shown; therefore, the point cloud encoding device 700 is not limited to Fig.34 That is, the point cloud encoding device 700 may include Fig.34 The processing units not shown in block form and may involve Fig.34 Processing or data flow not indicated by arrows etc.

[0414] like Fig.34 As shown, the point cloud encoding device 700 includes a geometry encoding unit 701, a geometry decoding unit 702, a point cloud generating unit 703, an attribute encoding unit 704 and a bit stream generating unit 705.

[0415] The geometry encoding unit 701 performs processing related to encoding of geometry data. For example, the geometry encoding unit 701 acquires geometry data of point cloud data to be input to the point cloud encoding device 700. The geometry encoding unit 701 encodes the geometry data to generate encoded data. The geometry encoding unit 701 supplies the generated geometry encoded data to the geometry decoding unit 702 and the bit stream generation unit 705.

[0416] The geometry decoding unit 702 performs processing related to decoding of geometry coded data. For example, the geometry decoding unit 702 acquires the geometry coded data supplied from the geometry coding unit 701. The geometry decoding unit 702 decodes the coded data by a decoding method having a corresponding relationship with the encoding method applied to the geometry coding unit 701 to generate (reconstruct) geometry data. The geometry decoding unit 702 supplies the generated geometry data to the point cloud generation unit 703.

[0417] The point cloud generation unit 703 performs processing related to the generation of point cloud data. For example, the point cloud generation unit 703 acquires attribute data of the point cloud data to be input to the point cloud encoding device 700. In addition, the point cloud generation unit 703 acquires the geometric shape data supplied from the geometric shape decoding unit 702.

[0418] The geometry data may be changed by processing such as encoding or decoding (eg, points may be increased or decreased or may be moved). That is, the geometry data supplied from the geometry decoding unit 702 may be different from the geometry data before being encoded by the geometry encoding unit 701 .

[0419] Therefore, the point cloud generation unit 703 performs a process of matching the attribute data with the geometric shape data (decoding result) (hereinafter, this process is also referred to as a recoloring process). That is, the point cloud generation unit 703 updates the attribute data according to the update of the geometric shape data. The point cloud generation unit 703 supplies the updated attribute data (attribute data having a corresponding relationship with the geometric shape data (decoding result)) to the attribute encoding unit 704.

[0420] The attribute encoding unit 704 performs processing related to encoding of attributes. For example, the attribute encoding unit 704 obtains the attribute data supplied from the point cloud generation unit 703. In addition, the attribute encoding unit 704 encodes the attribute data by an arbitrary method to generate encoded data of the attribute data. The encoding method is arbitrary. The attribute encoding unit 704 supplies the generated encoded data of the attribute data to the bit stream generation unit 705.

[0421] The bitstream generation unit 705 performs processing related to the generation of a bitstream. For example, the bitstream generation unit 705 obtains the encoded data of the geometry supplied from the geometry encoding unit 701. In addition, the bitstream generation unit 705 obtains the encoded data of the attribute data supplied from the attribute encoding unit 704. The bitstream generation unit 705 generates a bitstream including the encoded data. The bitstream generation unit 705 outputs the generated bitstream to the outside of the point cloud encoding device 700.

[0422] In the point cloud encoding device 700, the geometric shape encoding unit 701 can apply the above-mentioned present technology to encode the geometric shape data. For example, Figure 5 The illustrated geometry encoding unit 701 and the geometry encoding device 100 may have similar configurations and perform similar processing. Fig.14 The illustrated geometry encoding unit 701 and the geometry encoding device 100 may have similar configurations and perform similar processing. Fig.23 The illustrated geometry encoding unit 701 and the geometry encoding device 100 may have similar configurations and perform similar processing. Fig.28 The illustrated geometry encoding unit 701 and geometry encoding device 100 may have similar configurations and perform similar processing.

[0423] Therefore, the point cloud encoding device 700 can use a coordinate-based network to encode the geometric shape of the point cloud. The point cloud encoding device 700 can also improve the encoding efficiency by reducing spatial correlation. The point cloud encoding device 700 can also apply the aforementioned various methods to obtain the beneficial effects of various methods.

[0424] Alternatively, the geometry decoding unit 702 may apply the aforementioned present technology to decode the coded data of the geometry. Fig.10 The illustrated geometry decoding unit 702 and the geometry decoding device 200 may have similar configurations and perform similar processing. Fig.17 The illustrated geometry decoding unit 702 and the geometry decoding device 200 may have similar configurations and perform similar processing. Fig.26 The illustrated geometry decoding unit 702 and the geometry decoding device 200 may have similar configurations and perform similar processing. Fig.31 The illustrated geometry decoding unit 702 and the geometry decoding device 200 may have similar configurations and perform similar processing.

[0425] Therefore, the point cloud encoding device 700 can generate parameter vectors at a higher speed (that is, the processing time for generating parameter vectors can be reduced). In particular, even when the number of blocks increases, the point cloud encoding device 700 can suppress the increase in processing time for generating parameter vectors. The point cloud encoding device 700 can also apply the aforementioned various methods, thereby obtaining the beneficial effects of the various methods.

[0426] Note that each of these processing units (geometry encoding unit 701, ..., and bitstream generation unit 705) of the point cloud encoding device 700 has an arbitrary configuration. For example, each processing unit may be composed of a logic circuit that implements the above-mentioned processing. Alternatively, each processing unit may include, for example, a CPU, ROM, RAM, etc., and use them to execute a program to implement the aforementioned processing. Of course, each processing unit may have these two configurations, part of the aforementioned processing may be implemented by a logic circuit, and the rest may be implemented by executing a program. The structures of each processing unit may be independent of each other. For example, some of the processing units may implement part of the aforementioned processing by a logic circuit. Others of the processing units may implement the aforementioned processing by executing a program. Others of the processing units may implement the aforementioned processing by executing a logic circuit and a program.

[0427] <Point cloud coding process>

[0428] The point cloud encoding device 700 encodes point cloud data by performing point cloud encoding processing. Fig.35 The flowchart describes an example of the process of the point cloud encoding process.

[0429] When the point cloud encoding process starts, in step S701 , the geometry encoding unit 701 performs geometry encoding processing to encode geometry data and generate geometry encoding data.

[0430] In step S702 , the geometry decoding unit 702 performs a geometry decoding process to decode the geometry encoding data generated in step S701 , and generates (reconstructs) geometry data.

[0431] In step S703 , the point cloud generation unit 703 performs a recoloring process so that the attribute data corresponds to the geometric shape data generated in step S702 .

[0432] In step S704 , the attribute encoding unit 704 encodes the attribute data subjected to the recoloring process in step S703 .

[0433] In step S705 , the bit stream generation unit 705 generates a bit stream including the encoded data of the geometric shape data generated in step S701 and the encoded data of the attribute data generated in step S704 .

[0434] When the processing in step S705 ends, the point cloud encoding processing ends.

[0435] In step S701 of the point cloud encoding process, the geometric shape encoding unit 701 may perform the geometric shape encoding process to which the present technology is applied. Figure 8 The geometric shape encoding unit 701 can also perform the geometric shape encoding process according to the flow chart described in FIG. Fig.13 The geometric shape encoding unit 701 can also perform the geometric shape encoding process according to the flow chart described in FIG. Fig.16 The geometric shape encoding process is performed by the process described in the flowchart of FIG. 1. Similarly, in the geometric shape encoding process, the geometric shape encoding unit 701 can also apply the reference Fig.19 The CNN feature vector grid encoding process described in the flowchart of FIG. 701 can also be based on the reference Fig.25 The geometric shape encoding unit 701 can also perform the geometric shape encoding process according to the flow chart described in FIG. Fig.30 The flowchart describes the process to perform geometry encoding processing.

[0436] Therefore, the point cloud encoding device 700 can use a coordinate-based network to encode the geometric shape of the point cloud. The point cloud encoding device 700 can also improve the encoding efficiency by reducing spatial correlation. The point cloud encoding device 700 can also apply the aforementioned various methods to obtain the beneficial effects of various methods.

[0437] In addition, in step S702 of the point cloud encoding process, the geometric shape decoding unit 702 may perform the geometric shape decoding process to which the present technology is applied. Fig.11 The geometric shape decoding unit 702 can also perform the geometric shape decoding process according to the flow chart described in FIG. Fig.18 The geometric shape decoding process is performed by the process described in the flowchart of FIG. 1. Similarly, in the geometric shape decoding process, the geometric shape decoding unit 702 can also apply the reference Fig. 20 The CNN feature vector grid decoding process described in the flowchart of FIG. The geometric shape decoding unit 702 can also refer to Fig. 27 The geometric shape decoding unit 702 can also perform the geometric shape decoding process according to the flow chart described in FIG. Fig.32 The flowchart describes the process to perform the geometry decoding process.

[0438] Therefore, the point cloud encoding device 700 can generate parameter vectors at a higher speed (that is, the processing time for generating parameter vectors can be reduced). In particular, even when the number of blocks increases, the point cloud encoding device 700 can suppress the increase in processing time for generating parameter vectors. The point cloud encoding device 700 can also apply the aforementioned various methods, thereby obtaining the beneficial effects of the various methods.

[0439] <9. Implementation Method (Point Cloud Decoding Device)>

[0440] Fig.36 : is a block diagram showing an example of the configuration of a point cloud decoding device as one aspect of an information processing device to which the present technology is applied. Fig.36 The illustrated point cloud decoding device 800 is a decoding device having a corresponding relationship with the point cloud encoding device 700, and decodes a bit stream of a point cloud to generate point cloud data. The point cloud decoding device 800 applies the aforementioned present technology to decode a bit stream of a point cloud.

[0441] Notice, Fig.36 The main processing unit, data flow, etc. are shown; therefore, the point cloud decoding device 800 is not limited to Fig.36 That is, the point cloud decoding device 800 may include Fig.36 The processing units not shown in block form and may involve Fig.36 Processing or data flow not indicated by arrows etc.

[0442] like Fig.36 As shown, the point cloud decoding device 800 includes an extraction unit 801, a geometry decoding unit 802, an attribute decoding unit 803 and a point cloud generating unit 804.

[0443] The extraction unit 801 extracts geometry coded data from the bit stream, and supplies the geometry coded data to the geometry decoding unit 802. In addition, the extraction unit 801 extracts attribute coded data from the bit stream, and supplies the attribute coded data to the attribute decoding unit 803.

[0444] The geometry decoding unit 802 performs processing related to decoding of geometry coded data. For example, the geometry decoding unit 802 acquires the geometry coded data supplied from the extraction unit 801, and decodes the geometry coded data to generate (reconstruct) geometry data. The geometry decoding unit 802 supplies the generated geometry data to the attribute decoding unit 803 and the point cloud generation unit 804.

[0445] The attribute decoding unit 803 performs processing related to decoding of the attribute coded data. For example, the attribute decoding unit 803 acquires the attribute coded data supplied from the extraction unit 801. The attribute decoding unit 803 also acquires the geometry data supplied from the geometry decoding unit 802. The attribute decoding unit 803 decodes the attribute coded data using the geometry data to generate (reconstruct) the attribute data.

[0446] The attribute decoding unit 803 communicates with the attribute encoding unit 704 ( Fig.34 The attribute decoding unit 803 supplies the generated attribute data to the point cloud generation unit 804.

[0447] The point cloud generation unit 804 performs processing related to the generation of the point cloud. For example, the point cloud generation unit 804 acquires the geometry data supplied from the geometry decoding unit 802. The point cloud generation unit 804 also acquires the attribute data supplied from the attribute decoding unit 803. Then, the point cloud generation unit 804 makes the geometry data correspond to the attribute data to generate the point cloud data. The point cloud generation unit 804 outputs the generated point cloud data to the outside of the point cloud decoding device 800.

[0448] In the point cloud decoding device 800, the geometric shape decoding unit 802 can apply the above-mentioned present technology to decode the geometric shape encoded data. For example, Fig.10 The illustrated geometry decoding unit 802 and the geometry decoding device 200 may have similar configurations and perform similar processing. Fig.17 The illustrated geometry decoding unit 802 and the geometry decoding device 200 may have similar configurations and perform similar processing. Fig.26 The illustrated geometry decoding unit 802 and the geometry decoding device 200 may have similar configurations and perform similar processing. Fig.31 The illustrated geometry decoding unit 802 and the geometry decoding device 200 may have similar configurations and perform similar processing.

[0449] Therefore, the point cloud decoding device 800 can generate parameter vectors at a higher speed (that is, the processing time for generating parameter vectors can be reduced). In particular, even when the number of blocks increases, the point cloud decoding device 800 can suppress the increase in processing time for generating parameter vectors. The point cloud decoding device 800 can also apply the aforementioned various methods, thereby obtaining the beneficial effects of the various methods.

[0450] Note that each of these processing units (extraction unit 801, ..., and point cloud generation unit 804) of the point cloud decoding device 800 has an arbitrary configuration. For example, each processing unit may be composed of a logic circuit that implements the above-mentioned processing. Alternatively, each processing unit may include, for example, a CPU, ROM, RAM, etc., and use them to execute a program to implement the aforementioned processing. Of course, each processing unit may have these two configurations, part of the aforementioned processing may be implemented by a logic circuit, and the rest may be implemented by executing a program. The structures of each processing unit may also be independent of each other. For example, some of the processing units may implement part of the aforementioned processing by a logic circuit. Others of the processing units may implement the aforementioned processing by executing a program. Others of the processing units may implement the aforementioned processing by executing a logic circuit and a program.

[0451] <Point cloud decoding process>

[0452] The point cloud decoding device 800 decodes the bit stream by performing point cloud decoding processing. Fig.37 The flowchart describes an example of the flow of the point cloud decoding process.

[0453] When the point cloud decoding process starts, in step S801, the extraction unit 801 extracts geometry encoding data and attribute encoding data from the bit stream.

[0454] In step S802 , the geometry decoding unit 802 performs a geometry decoding process to decode the geometry encoding data and generate (reconstruct) geometry data.

[0455] In step S803, the attribute decoding unit 803 decodes the attribute coded data to generate (reconstruct) attribute data.

[0456] In step S804 , the point cloud generating unit 804 makes the geometric shape data generated in step S802 correspond to the attribute data generated in step S803 to generate point cloud data.

[0457] When the processing in step S804 ends, the point cloud decoding processing ends.

[0458] In step S802 of the point cloud decoding process, the geometric shape decoding unit 802 may perform the geometric shape decoding process to which the present technology is applied. Fig.11 The geometric shape decoding unit 802 can also perform the geometric shape decoding process according to the flow chart described in FIG. Fig.18 The geometric shape decoding process is performed by the process described in the flowchart of FIG. 802. Similarly, in the geometric shape decoding process, the geometric shape decoding unit 802 can also apply the reference Fig. 20 The CNN feature vector grid decoding process described in the flowchart of FIG. The geometric shape decoding unit 802 can also refer to Fig. 27 The geometric shape decoding unit 802 can also perform the geometric shape decoding process according to the flow chart described in FIG. Fig.32 The flowchart describes the process to perform the geometry decoding process.

[0459] Therefore, the point cloud decoding device 800 can generate parameter vectors at a higher speed (that is, the processing time for generating parameter vectors can be reduced). In particular, even when the number of blocks increases, the point cloud decoding device 800 can suppress the increase in processing time for generating parameter vectors. The point cloud decoding device 800 can also apply the aforementioned various methods, thereby obtaining the beneficial effects of the various methods.

[0460] <10. Additional Notes>

[0461] <Computer>

[0462] The aforementioned series of processes may be performed by hardware, or may be performed by software. When a series of processes are performed by software, the program constituting the software is installed in a computer. Here, examples of computers include computers incorporated in dedicated hardware, general-purpose personal computers capable of performing various functions by various programs installed therein, and the like.

[0463] Fig.38 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes by a program.

[0464] exist Fig.38 In the illustrated computer 900 , a central processing unit (CPU) 901 , a read only memory (ROM) 902 , and a random access memory (RAM) 903 are connected to one another via a bus 904 .

[0465] The input / output interface 910 is connected to the bus 904. An input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected to the input / output interface 910.

[0466] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a nonvolatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0467] The computer configured as described above performs the above-described series of processing in a manner in which the CPU 901 loads, for example, a program stored in the storage unit 913 onto the RAM 903 via the input / output interface 910 and the bus 904 and executes the program. In addition, the RAM 903 appropriately stores data and the like necessary for the CPU 901 to perform various types of processing.

[0468] The program to be executed by the computer can be applied together with the program recorded in the removable medium 921 serving as a package medium, etc. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 with the removable medium 921 attached to the drive 915 .

[0469] In addition, the program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received at the communication unit 914 and then installed in the storage unit 913.

[0470] Alternatively, the program may also be installed in the ROM 902 or the storage unit 913 in advance.

[0471] <Goal of applying this technology>

[0472] The present technology is applicable to any configuration. For example, the present technology can be applied to various electronic devices.

[0473] For example, the present technology may also be implemented as a partial configuration of a device such as a processor used as a system large-scale integration (LSI) (e.g., a video processor), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a collection of units to which other functions are added (e.g., a video collection).

[0474] For example, the present technology is also applicable to a network system including multiple devices. For example, the present technology can be implemented as cloud computing, in which multiple devices perform processing in a shared and collaborative manner via a network. For example, the present technology can be implemented in a cloud service that provides services related to images (moving images) to any terminal such as a computer, an audio-visual (AV) device, a mobile information processing terminal, or an Internet of Things (IoT) device.

[0475] Note that the term "system" as used herein refers to a collection of multiple components (e.g., devices, modules (components)), regardless of whether all the components are accommodated in a single housing. Therefore, the term "system" refers to multiple devices that are respectively accommodated in separate housings and connected to each other via a network, as well as one device including multiple modules accommodated in one housing.

[0476] <Fields and applications of this technology>

[0477] Systems, devices, processing units, etc. to which the present technology is applied can be used in any field such as transportation, medical treatment, crime prevention, agriculture, animal husbandry, mining, beauty, factories, household appliances, weather, and natural monitoring. Its application is also arbitrary.

[0478] <Others>

[0479] Note that the term "flag" as used herein refers to information used to identify multiple states, and not only refers to information to be used to identify the two states of true (1) and false (0), but also refers to information that can identify three or more states. Therefore, the value that the "flag" can take can be, for example, a binary value of 1 or 0, or can be a ternary value or more. That is, the number of bits that constitute the "flag" is arbitrary. For example, the number of bits can be one or more than one. In addition, it is assumed herein that the identification information (including the flag) is included in the bit stream not only in the form of identification information, but also in the form of difference information about the identification information and the reference information. Therefore, the term "flag" or "identification information" as used herein refers not only to the information, but also to difference information with the reference information.

[0480] In addition, various types of information (e.g., metadata) about the coded data (bitstream) can be sent or recorded in any form as long as the sending or recording form is associated with the coded data. As used herein, the terms "associate, associates, associating, associated, association" mean, for example: when processing one data item, another data item can be used (can be linked). That is, mutually associated data items can be regarded as a single data item, or can also be regarded as separate data items. For example, information associated with the coded data (image) can be transmitted through a transmission path different from the transmission path used for the coded data (image). Alternatively, for example, information associated with the coded data (image) can be recorded in a recording medium different from the recording medium recording the coded data (image) (or in a single recording medium, in a recording area different from the recording area recording the coded data (image)). Note that a portion of the data, rather than the entire data, can be "associated". For example, an image and information corresponding to the image can be associated with each other in arbitrary units such as multiple frames, a single frame, or a portion of a frame.

[0481] Note that the terms "combine," "multiplex," "add," "merge," "include," "store," "place," "load," "insert," and the like as used herein refer to the merging of multiple things into one thing, for example, merging encoded data and metadata into a single data item, and to a method of the aforementioned term "association."

[0482] In addition, the embodiments of the present technology are not limited to the aforementioned embodiments, and various modifications may be made without departing from the scope of the present technology.

[0483] For example, a configuration described as a single device (or a single processing unit) may be divided and configured as a plurality of devices (or a plurality of processing units). Conversely, the configuration described above as a plurality of devices (or a plurality of processing units) may be integrated and configured as a single device (or a single processing unit). Alternatively, configurations other than those described above may also be added to the configuration of each device (or each processing unit). In addition, a portion of the configuration of one device (or a processing unit) may be included in the configuration of another device (or another processing unit), as long as the configuration and operation as the entire system are substantially the same.

[0484] In addition, for example, the aforementioned program may be executed by any device. In this case, the device only needs to have necessary functions (eg, functional blocks) and obtain necessary information.

[0485] In addition, for example, the steps of a single flowchart may be performed by a single device, or may be performed by a plurality of devices in a shared manner. Furthermore, when a single step includes multiple types of processing, the multiple types of processing may be performed by a single device, or may be performed by a plurality of devices in a shared manner. In other words, the multiple types of processing in a single step may also be performed as multiple types of processing of multiple steps. Conversely, the multiple types of processing described as multiple steps may be performed together as a single step.

[0486] In addition, for example, the program to be executed by the computer may be executed in a manner that multiple types of processing describing the steps of the program are performed in a time series manner according to the order described herein, or the program to be executed by the computer may be executed in parallel or individually at a desired timing such as the timing of calling the program. That is, unless a contradiction occurs, multiple types of processing of each step may be performed in an order different from the above order. In addition, multiple types of processing describing the steps of a program may be performed in parallel or in combination with multiple types of processing of another program.

[0487] In addition, for example, multiple technologies related to the present technology may be implemented independently and individually unless a contradiction occurs. Of course, any of the multiple technologies related to the present technology may be implemented in combination. For example, part or all of the present technology described in one embodiment may be implemented in combination with part or all of the present technology described in another embodiment. In addition, any one or all of the multiple technologies related to the present technology may be implemented in combination with another technology not described above.

[0488] Note that the present technology can also adopt the following configurations.

[0489] (1) An information processing device comprising:

[0490] a generating unit that generates, based on a parameter vector of a coordinate-based network representing a geometric shape of 3D data, a feature vector representing a spatial correlation lower than that of the parameter vector; and

[0491] An encoding unit is used to encode the feature vector.

[0492] (2) The information processing device according to (1), wherein:

[0493] The generating unit generates the feature vector using a 3D-CNN (3D-Convolutional Neural Network) having a three-dimensional convolutional layer.

[0494] (3) The information processing device according to (2), wherein:

[0495] The generating unit generates a parameter vector maintained in a mesh shape according to the geometric shape, and generates the feature vector using the 3D-CNN with the parameter vector as input.

[0496] (4) The information processing device according to (3), wherein:

[0497] The generating unit generates the parameter vector using the 3D-CNN with the geometric shape as input.

[0498] (5) The information processing device according to (3), wherein:

[0499] The generating unit generates the parameter vector by optimizing the Lagrangian function.

[0500] (6) The information processing device according to (1), wherein:

[0501] The generating unit generates the feature vector using a 3D-CNN (3D-Convolutional Neural Network) having a sparse three-dimensional convolutional layer.

[0502] (7) The information processing device according to (6), wherein:

[0503] The encoding unit also encodes a mask grid used in the 3D-CNN.

[0504] (8) The information processing device according to (6) or (7), wherein:

[0505] The encoding unit derives a prediction value of the feature vector according to the mask grid used in the 3D-CNN, derives a prediction residual as a difference between the feature vector and the prediction value, and encodes the prediction residual.

[0506] (9) The information processing device according to any one of (6) to (8), wherein:

[0507] The encoding unit obtains parameters of an entropy model of the feature vector according to a mask grid used in the 3D-CNN, and performs entropy encoding on the feature vector using the parameters.

[0508] (10) An information processing method comprising:

[0509] generating, based on a parameter vector of a coordinate-based network representing the geometry of the 3D data, a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector; and

[0510] The feature vector is encoded.

[0511] (11) An information processing device comprising:

[0512] a decoding unit that decodes the encoded data to generate a feature vector representing a spatial correlation lower than that of a parameter vector of a coordinate-based network representing a geometry of the 3D data; and

[0513] A generating unit is used to generate the geometric shape according to the feature vector.

[0514] (12) The information processing device according to (11), wherein:

[0515] The generation unit generates the parameter vector according to the feature vector, and generates the geometric shape using a coordinate-based network to which the parameter vector is applied.

[0516] (13) The information processing device according to (12), wherein:

[0517] The generating unit generates the parameter vector by using a 3D-CNN (3D-Convolutional Neural Network) having a three-dimensional deconvolution layer, taking the feature vector as input.

[0518] (14) The information processing device according to (12) or (13), wherein:

[0519] The generation unit takes the position information of the query point as input, derives the occupancy probability of the query point using a coordinate-based network to which the parameter vector is applied, and outputs information on the query point with a high occupancy probability.

[0520] (15) The information processing device according to (14), wherein:

[0521] The generating unit outputs the occupancy probability value of the query point with high occupancy probability.

[0522] (16) The information processing device according to (14) or (15), wherein:

[0523] The generating unit sets the query point and uses the set query point as an input.

[0524] (17) The information processing device according to any one of (14) to (16), wherein:

[0525] The generation unit also uses time information as input.

[0526] (18) The information processing device according to any one of (14) to (17), wherein:

[0527] The generation unit also uses the gaze direction as input.

[0528] (19) The information processing device according to any one of (12) to (18), wherein:

[0529] The generating unit generates the parameter vector by using a 3D-CNN (3D-Convolutional Neural Network) with a sparse 3D deconvolution layer, taking the feature vector as input.

[0530] (20) An information processing method comprising:

[0531] decoding the encoded data to generate a feature vector representing a spatial correlation lower than a spatial correlation of a parameter vector of a coordinate-based network representing the geometry of the 3D data; and

[0532] The geometric shape is generated according to the feature vector.

[0533] (21) An information processing device comprising:

[0534] a generating unit that generates, based on a parameter vector of a coordinate-based network representing a geometric shape of 3D data, a feature vector representing a spatial correlation lower than that of the parameter vector; and

[0535] An encoding unit is used to encode the feature vector.

[0536] (22) The information processing device according to (21), wherein:

[0537] The generating unit generates a first feature vector as a feature vector of a first resolution and a second feature vector as a feature vector of a second resolution,

[0538] The encoding unit encodes the first feature vector and the second feature vector, and

[0539] The second resolution is higher than the first resolution.

[0540] (23) The information processing device according to (22), wherein:

[0541] The generating unit generates the first feature vector using a first 3D-CNN (3D-Convolutional Neural Network), and generates the second feature vector using a second 3D-CNN different from the first 3D-CNN, and

[0542] The three-dimensional convolutional layer of the first 3D-CNN is different from the three-dimensional convolutional layer of the second 3D-CNN.

[0543] (24) The information processing device according to (23), wherein:

[0544] The first 3D-CNN and the second 3D-CNN share at least one computing layer.

[0545] (25) An information processing method comprising:

[0546] generating, based on a parameter vector of a coordinate-based network representing the geometry of the 3D data, a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector; and

[0547] The feature vector is encoded.

[0548] (31) An information processing device comprising:

[0549] a decoding unit that decodes the encoded data to generate a feature vector representing a spatial correlation lower than that of a parameter vector of a coordinate-based network representing a geometry of the 3D data; and

[0550] A generating unit is used to generate the geometric shape according to the feature vector.

[0551] (32) The information processing device according to (31), wherein:

[0552] The decoding unit decodes the first encoded data to generate a first feature vector as a feature vector of a first resolution,

[0553] The decoding unit decodes second encoded data different from the first encoded data to generate a second feature vector as a feature vector of a second resolution,

[0554] The generating unit generates a transformation vector according to the first feature vector,

[0555] The generating unit adds the second eigenvector to the transformation vector for each element to generate an addition result vector,

[0556] The generating unit generates the geometric shape according to the addition result vector, and

[0557] The second resolution is higher than the first resolution.

[0558] (33) The information processing device according to (32), wherein:

[0559] The generation unit generates the parameter vector according to the addition result vector, and generates the geometric shape using a coordinate-based network to which the parameter vector is applied.

[0560] (34) The information processing device according to (33), wherein:

[0561] The generating unit uses the first feature vector as input and uses a first 3D-CNN (3D-convolutional neural network) to generate the transformation vector,

[0562] The generating unit uses the addition result vector as input and uses a second 3D-CNN different from the first 3D-CNN to generate the parameter vector, and

[0563] The three-dimensional convolutional layer of the first 3D-CNN is different from the three-dimensional convolutional layer of the second 3D-CNN.

[0564] (35) An information processing method comprising:

[0565] decoding the encoded data to generate a feature vector representing a spatial correlation lower than a spatial correlation of a parameter vector of a coordinate-based network representing the geometry of the 3D data; and

[0566] The geometric shape is generated according to the feature vector.

[0567] Reference numerals list

[0568] 100 Geometric shape encoding device

[0569] 111 Parameter Vector Mesh Generation Unit

[0570] 112 CNN feature vector grid generation unit

[0571] 113 Coding Unit

[0572] 121 Generation Unit

[0573] 200 Geometry decoding device

[0574] 211 Decoding Unit

[0575] 212 Parameter vector mesh generation unit

[0576] 213 Geometry Generation Unit

[0577] 221 Generation Unit

[0578] 301 CNN feature vector grid encoding unit

[0579] 302 Mask grid coding unit

[0580] 351 CNN feature vector grid decoding unit

[0581] 352 Mask grid decoding unit

[0582] 551 Addition Unit

[0583] 700 Point Cloud Encoding Device

[0584] 701 Geometry Coding Unit

[0585] 702 Geometry Decoding Unit

[0586] 703 Point Cloud Generation Unit

[0587] 704 Attribute decoding unit

[0588] 705 Bitstream Generation Unit

[0589] 800 Point Cloud Decoding Unit

[0590] 801 Extraction Unit

[0591] 802 Geometry Decoding Unit

[0592] 803 Attribute decoding unit

[0593] 804 Point cloud generation unit

[0594] 900 Computer< / lvac>

Claims

1. An information processing device, comprising: a generating unit that generates, based on a parameter vector of a coordinate-based network representing a geometric shape of 3D data, a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector; as well as An encoding unit is used to encode the feature vector.

2. The information processing device according to claim 1, wherein: The generating unit generates the feature vector using a 3D-CNN (3D-Convolutional Neural Network) having a three-dimensional convolutional layer.

3. The information processing device according to claim 2, wherein: The generating unit generates a parameter vector maintained in a mesh shape according to the geometric shape, and generates the feature vector using the 3D-CNN with the parameter vector as input.

4. The information processing device according to claim 3, wherein: The generating unit generates the parameter vector using the 3D-CNN with the geometric shape as input.

5. The information processing device according to claim 3, wherein: The generating unit generates the parameter vector by optimizing the Lagrangian function.

6. The information processing device according to claim 1, wherein: The generating unit generates the feature vector using a 3D-CNN (3D-Convolutional Neural Network) having a sparse three-dimensional convolutional layer.

7. The information processing device according to claim 6, wherein: The encoding unit also encodes a mask grid used in the 3D-CNN.

8. The information processing device according to claim 6, wherein: The encoding unit derives a prediction value of the feature vector according to the mask grid used in the 3D-CNN, derives a prediction residual as a difference between the feature vector and the prediction value, and encodes the prediction residual.

9. The information processing device according to claim 1, wherein: The generating unit generates a first feature vector as a feature vector of a first resolution and a second feature vector as a feature vector of a second resolution, The encoding unit encodes the first feature vector and the second feature vector respectively, and The second resolution is higher than the first resolution.

10. An information processing method, comprising: generating, based on a parameter vector of a coordinate-based network representing the geometry of the 3D data, a feature vector representing a spatial correlation lower than the spatial correlation of the parameter vector; as well as The feature vector is encoded.

11. An information processing device, comprising: a decoding unit that decodes the encoded data to generate a feature vector representing a spatial correlation lower than a spatial correlation of a parameter vector of a coordinate-based network representing a geometry of the 3D data; as well as A generating unit is used to generate the geometric shape according to the feature vector.

12. The information processing device according to claim 11, wherein: The generation unit generates the parameter vector according to the feature vector, and generates the geometric shape using a coordinate-based network to which the parameter vector is applied.

13. The information processing device according to claim 12, wherein: The generating unit generates the parameter vector by using a 3D-CNN (3D-Convolutional Neural Network) having a three-dimensional deconvolution layer, taking the feature vector as input.

14. The information processing device according to claim 12, wherein: The generation unit takes the position information of the query point as input, derives the occupancy probability of the query point using a coordinate-based network to which the parameter vector is applied, and outputs information on the query point with a high occupancy probability.

15. The information processing device according to claim 14, wherein: The generating unit outputs the occupancy probability value of the query point with high occupancy probability.

16. The information processing device according to claim 14, wherein: The generating unit sets a query point and uses the set query point as an input.

17. The information processing device according to claim 14, wherein: The generation unit also uses time information as input.

18. The information processing device according to claim 14, wherein: The generation unit also uses the gaze direction as input.

19. The information processing device according to claim 11, wherein: The decoding unit decodes the first encoded data to generate a first feature vector as a feature vector of a first resolution, The decoding unit decodes second encoded data different from the first encoded data to generate a second feature vector as a feature vector of a second resolution, The generating unit generates a transformation vector according to the first feature vector, The generating unit adds the second eigenvector to the transformation vector for each element to generate an addition result vector, The generating unit generates the geometric shape according to the addition result vector, and The second resolution is higher than the first resolution.

20. An information processing method, comprising: decoding the encoded data to generate a feature vector representing a spatial correlation lower than a spatial correlation of a parameter vector of a coordinate-based network representing the geometry of the 3D data; as well as The geometric shape is generated according to the feature vector.

Images