Information processing apparatus and method

Abstract

The present disclosure relates to an information processing device and method capable of suppressing a decrease in encoding efficiency. According to the present invention, importance coefficient vectors for controlling contributions to prediction are used to combine a plurality of context vectors corresponding to nodes to be processed of 3D data having a tree structure, thereby generating a composite vector. A prediction probability vector indicating a probability value of a potential occupancy state of each child node of the node to be processed is derived on the basis of the composite vector, and the prediction probability vector is used to encode information indicating the occupancy state of the child nodes of the node to be processed. The present disclosure can be applied, for example, to an information processing device, an electronic device, an information processing method, and a program.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and method, and more particularly to an information processing apparatus and method capable of reducing a decrease in coding efficiency. Background Art

[0002] Conventionally, as a method for encoding geometric information of 3D data, there has been a method of encoding an octree by using a neural network (see, for example, Non-Patent Document 1). In this method, a context vector is derived based on the occupancy state of a nearby region, a prediction probability vector is derived based on the context vector, and the occupancy state is encoded by using the prediction probability vector. At this time, the context vector is derived not only for the processing target frame but also for nearby frames, and the context vector is applied to the derivation of the prediction probability vector. Therefore, geometric information can be encoded by using intra-frame correlation and inter-frame correlation such as intra-frame prediction and inter-frame prediction of 2D encoding.

[0003] Citation List

[0004] Non-Patent Document

[0005] Non-Patent Document 1: Zizheng Que, Guo Lu, Dong Xu, “VoxelContext-Net: An Octree-based Framework for Point Cloud Compression”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 6042-6051. Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] However, in the method described in Non-Patent Document 1, the contribution degree of each context vector to prediction has been determined during learning. Therefore, it is possible that it is difficult to obtain an optimal prediction probability vector for an encoding target during inference, and there is a probability of reducing the coding efficiency.

[0008] The present disclosure has been made in view of such a situation, and an object thereof is to be able to suppress a decrease in coding efficiency.

[0009] Solutions to the Problems

[0010] An information processing apparatus according to one aspect of the present technology is an information processing apparatus including: a vector synthesis unit configured to generate a synthesis vector by combining a plurality of context vectors corresponding to a processing target node of 3D data having a tree structure by using an importance coefficient vector for controlling the contribution degree to prediction; a prediction probability vector derivation unit configured to derive a prediction probability vector indicating probability values of occupancy states that each child node of the processing target node can take based on the synthesis vector; and an occupancy state encoding unit configured to encode information indicating the occupancy state of the child node of the processing target node by using the prediction probability vector, wherein the context vectors among the plurality of context vectors correspond to the occupancy state of a nearby region in the spatial direction of the processing target node in the processing target frame, or correspond to the occupancy state of a nearby region in the spatial direction of a node corresponding to the processing target node in a nearby frame in the time direction, the plurality of context vectors include: a context vector corresponding to the occupancy state of a nearby region in the processing target layer of the processing target frame; a context vector corresponding to the occupancy state of a nearby region in the processing target layer of a nearby frame; and a context vector corresponding to the occupancy state of a nearby region in a layer lower than the processing target layer of a nearby frame, and the prediction is a prediction of the occupancy state of the child node of the processing target node by using intra-frame correlation and inter-frame correlation.

[0011] An information processing method according to one aspect of the present technology is an information processing method including: generating a synthesis vector by combining a plurality of context vectors corresponding to a processing target node of 3D data having a tree structure by using an importance coefficient vector for controlling the contribution degree to prediction; deriving a prediction probability vector indicating probability values of occupancy states that each child node of the processing target node can take based on the synthesis vector; and encoding information indicating the occupancy state of the child node of the processing target node by using the prediction probability vector, wherein the context vectors among the plurality of context vectors correspond to the occupancy state of a nearby region in the spatial direction of the processing target node in the processing target frame, or correspond to the occupancy state of a nearby region in the spatial direction of a node corresponding to the processing target node in a nearby frame in the time direction, the plurality of context vectors include: a context vector corresponding to the occupancy state of a nearby region in the processing target layer of the processing target frame; a context vector corresponding to the occupancy state of a nearby region in the processing target layer of a nearby frame; and a context vector corresponding to the occupancy state of a nearby region in a layer lower than the processing target layer of a nearby frame, and the prediction is a prediction of the occupancy state of the child node of the processing target node by using intra-frame correlation and inter-frame correlation.

[0012] An information processing apparatus according to another aspect of the present technology is an information processing apparatus including: a vector synthesis unit configured to generate a synthesis vector by combining a plurality of context vectors corresponding to a processing target node of 3D data having a tree structure by using an importance coefficient vector for controlling a contribution degree to prediction; a prediction probability vector derivation unit configured to derive a prediction probability vector indicating probability values of occupancy states that each child node of the processing target node can take based on the synthesis vector; and an occupancy state decoding unit configured to decode a bit stream by using the prediction probability vector to generate information indicating the occupancy states of the child nodes of the processing target node, wherein the context vectors among the plurality of context vectors correspond to the occupancy states of a nearby region in the spatial direction of the processing target node in the processing target frame, or correspond to the occupancy states of a nearby region in the spatial direction of a node corresponding to the processing target node in a nearby frame in the time direction, the plurality of context vectors include: a context vector corresponding to the occupancy state of a nearby region in the processing target layer of the processing target frame; a context vector corresponding to the occupancy state of a nearby region in the processing target layer of a nearby frame; and a context vector corresponding to the occupancy state of a nearby region in a layer lower than the processing target layer of a nearby frame, and the prediction is a prediction of the occupancy state of the child nodes of the processing target node performed by using intra-frame correlation and inter-frame correlation.

[0013] An information processing method according to another aspect of the present technology is an information processing method including: generating a synthesis vector by combining a plurality of context vectors corresponding to a processing target node of 3D data having a tree structure by using an importance coefficient vector for controlling a contribution degree to prediction; deriving a prediction probability vector indicating probability values of occupancy states that each child node of the processing target node can take based on the synthesis vector; and decoding a bit stream by using the prediction probability vector to generate information indicating the occupancy states of the child nodes of the processing target node, wherein the context vectors among the plurality of context vectors correspond to the occupancy states of a nearby region in the spatial direction of the processing target node in the processing target frame, or correspond to the occupancy states of a nearby region in the spatial direction of a node corresponding to the processing target node in a nearby frame in the time direction, the plurality of context vectors include: a context vector corresponding to the occupancy state of a nearby region in the processing target layer of the processing target frame; a context vector corresponding to the occupancy state of a nearby region in the processing target layer of a nearby frame; and a context vector corresponding to the occupancy state of a nearby region in a layer lower than the processing target layer of a nearby frame, and the prediction is a prediction of the occupancy state of the child nodes of the processing target node performed by using intra-frame correlation and inter-frame correlation.

[0014] In an information processing apparatus and method according to one aspect of the present technology, a plurality of context vectors corresponding to a processing target node of 3D data having a tree structure are combined by using an importance coefficient vector for controlling the contribution degree to prediction to generate a synthetic vector. A prediction probability vector indicating probability values of occupancy states that each child node of the processing target node can take is derived based on the synthetic vector. Information indicating the occupancy state of the child node of the processing target node is encoded by using the prediction probability vector.

[0015] In an information processing apparatus and method according to another aspect of the present technology, a plurality of context vectors corresponding to a processing target node of 3D data having a tree structure are combined by using an importance coefficient vector for controlling the contribution degree to prediction to generate a synthetic vector. A prediction probability vector indicating probability values of occupancy states that each child node of the processing target node can take is derived based on the synthetic vector. The prediction probability vector is used to decode a bit stream to generate information indicating the occupancy state of the child node of the processing target node. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a diagram showing an example of the processing order of an octree.

[0017] Figure 2 is a diagram showing an example of the state of prediction of the occupancy state.

[0018] Figure 3 is a diagram showing an example of the derivation state of the prediction probability vector.

[0019] Figure 4 is a table showing an example of a geometric information encoding method.

[0020] Figure 5 is a block diagram showing an example of the main configuration of a geometric information encoding apparatus.

[0021] Figure 6 is a block diagram showing an example of the main configuration of an octree encoding unit.

[0022] Figure 7 is a flowchart showing an example of the flow of encoding processing.

[0023] Figure 8 is a flowchart showing an example of the flow of octree encoding processing.

[0024] Figure 9 is subsequent to Figure 8 and is a flowchart showing an example of the flow of octree encoding processing.

[0025] Figure 10 is a block diagram showing an example of the main configuration of a geometric information decoding apparatus.

[0026] Figure 11 It is a block diagram showing a main configuration example of an octree decoding unit.

[0027] Figure 12 It is a flowchart for explaining an example of the process of decoding processing.

[0028] Figure 13 It is a flowchart for explaining an example of the process of octree decoding processing.

[0029] Figure 14 It is a block diagram showing a main configuration example of an octree encoding unit.

[0030] Figure 15 It is a flowchart for explaining an example of the process of octree encoding processing.

[0031] Figure 16 It is after Figure 15 and is a flowchart for explaining an example of the process of octree encoding processing.

[0032] Figure 17 It is a block diagram showing a main configuration example of an octree decoding unit.

[0033] Figure 18 It is a flowchart for explaining an example of the process of octree decoding processing.

[0034] Figure 19 It is a block diagram showing a main configuration example of an octree encoding unit.

[0035] Figure 20 It is a block diagram showing a main configuration example of a probability vector generation unit.

[0036] Figure 21 It is a flowchart for explaining an example of the process of octree encoding processing.

[0037] Figure 22 It is after Figure 21 and is a flowchart for explaining an example of the process of octree encoding processing.

[0038] Figure 23 It is a flowchart for explaining an example of the process of probability vector generation processing.

[0039] Figure 24 It is a block diagram showing a main configuration example of an octree decoding unit.

[0040] Figure 25 It is a flowchart for explaining an example of the process of octree decoding processing.

[0041] Figure 26 It is a view showing an example of a macroblock.

[0042] Figure 27 It is a diagram showing an example of the area near the child.

[0043] Figure 28 It is a block diagram showing an example of the main configuration of a computer. Detailed implementation

[0044] The ways for implementing the present disclosure (hereinafter referred to as embodiments) are described below. Note that the description will be carried out in the following order.

[0045] 1. Documents, etc. supporting technical content and technical terms

[0046] 2. Geometric information encoding

[0047] 3. Importance coefficient vector

[0048] 4. Query vector

[0049] 5. Division of the nearby area

[0050] 6. Addition of metadata

[0051] 7. Appendix

[0052] <1. Documents, etc. supporting technical content and technical terms>

[0053] The scope disclosed in this technology, in addition to the content disclosed in the embodiments, also includes the content described in the following non-patent documents, etc. known at the time of application, and the content of other documents mentioned in the following non-patent documents, etc. <()

[0054] Non-patent document 1: (as described above)

[0055] That is to say, the content described in the above non-patent documents, the content of other documents mentioned in the above non-patent documents, etc. are also the basis for determining the support requirements.

[0056] <2. Geometric information encoding>

[0057] <Point cloud>

[0058] Conventionally, as 3D data representing the three-dimensional structure of a three-dimensional object (an object with a three-dimensional shape), there has been a point cloud that represents the object as a set of a large number of points. The data of the point cloud (also referred to as point cloud data) includes the geometric information (position information) and attributes (attribute information) of each point that makes up the point cloud. The geometric information indicates the position of the point in three-dimensional space. The attribute indicates the attribute of the point. The attribute can include any information. For example, color information, reflectance information, normal information, etc. about each point can be included in the attribute. As described above, the point cloud has a relatively simple data structure, and by using a sufficiently large number of points, any three-dimensional object can be represented with sufficient accuracy.

[0059] <Voxel representation>

[0060] However, since such a point cloud has a relatively large data volume, it has been necessary to compress the data volume through encoding or the like. For example, as the position accuracy of the geometric information of each point increases, the data volume increases. Therefore, a method of representing geometric information by using voxels has been considered. A voxel is a region obtained by dividing a three-dimensional space region including an object. The position of each point in the point cloud is located at a predetermined position (e.g., the center) within such a voxel. In other words, it indicates whether there is a point in each voxel. By doing so, the geometric information of each point can be quantified in units of voxels. Therefore, an increase in the data volume of the geometric information can be suppressed. Note that in this specification, such a representation method of geometric information using voxels is also referred to as voxel representation.

[0061] A voxel can be divided into multiple voxels. That is, by recursively and repeatedly dividing the voxel, the size of each voxel can be further reduced. The smaller the size of the voxel, the higher the resolution. That is, the position of each point can be represented more accurately. In other words, the effect of reducing the data volume of the geometric information by the above quantization is suppressed.

[0062] Note that in such voxel representation, only the voxels in which there are points are divided. The voxels in which there are points are divided into eight voxels (2×2×2). Among the eight voxels, only the voxels in which there are points are further divided into eight. In this way, the voxels in which there are points are recursively divided until the smallest unit is obtained. In this way, a hierarchical structure is formed.

[0063] <Octree representation>

[0064] As described above, the voxel representation indicates the presence or absence of points in each voxel. That is, using the voxels of each layer as nodes, for each segmented voxel, 0 and 1 are used to represent the presence or absence of points, so that the geometric information can be represented in a tree structure. For example, when the voxel is segmented into eight for each layer as described above, the geometric information can be represented as an octree. In this specification, such a representation method of geometric information using an octree is also called octree representation. Such a bit pattern of each node is arranged and encoded in a predetermined order. By using the octree representation, scalable decoding of geometric information can be achieved. That is, only the necessary information can be decoded to obtain the geometric information (resolution) of any layer.

[0065] In addition, as described above, in the voxel representation, the segmentation of voxels where points do not exist can be omitted, so that in the octree representation, nodes where points do not exist can also be omitted. Therefore, an increase in the data volume of geometric information can be suppressed.

[0066] <voxelcontext-net>

[0067] Meanwhile, as such an encoding method of an octree, Non-Patent Document 1 discloses a technique called VoxelContext-Net that uses context. In this method, a context vector is derived based on the occupancy state of a nearby region, a prediction probability vector indicating the occupancy state of the child nodes of the node to be processed is derived based on this context vector, and the occupancy state is encoded using this prediction probability vector. At this time, the context vector is derived not only for the frame to be processed but also for nearby frames, and the context vector is applied to the derivation of the prediction probability vector. That is, not only the octree at the processing object time t (the octree corresponding to the point cloud at time t) is used, but also the octrees at times t - 1, t + 1, etc. are used. That is, octrees of multiple frames (multiple times) are used for encoding and decoding. That is, like in intra-frame prediction and inter-frame prediction in 2D encoding, intra-frame correlation and inter-frame correlation can be used to predict the occupancy state of the child nodes of the node to be processed, and the prediction result can be used to encode or decode geometric information.

[0068] Note that in this specification, such octrees of multiple frames (multiple times) are also referred to as an octree sequence. In addition, the frame at the processing object time t is also referred to as the processing object frame. In addition, the frames at times close to the processing object frame are also referred to as nearby frames. In particular, among the nearby frames, the frames adjacent to the processing object frame (that is, the frame at time t - 1 or the frame at time t + 1) are also referred to as adjacent frames. In addition, the nearby region indicates a region that is spatially close to the processing object node in the processing object frame, or a region that is spatially close to the node corresponding to the processing object node in the nearby frame (for example, the node at the same position as the processing object node). For example, in the case of the processing object frame, the nearby region indicates the peripheral region in the spatial direction of the processing object node or the node group located within this region (the region of the processing object frame within a predetermined range based on the processing object node). In addition, in the case of the nearby frame, the nearby region indicates the region corresponding to the nearby region in the processing object frame of the nearby frame (the region having a predetermined positional relationship with the nearby region in the processing object frame) or the node group located within this region.

[0069] Next, the processing order of each node of the octree will be described. For example, the layer of depth k of the octree at time t (also referred to as LoD) is represented as Octree_(t,k)∈{0,1}^(2^k×2^k×2^k). In addition, the point cloud PC_t at time t is quantized by 2^K bits, and the depth of the octree is K. That is, k = 0,..., K. Note that in this specification, x_y indicates that the subscript of x is y (x y )). In addition, x^y indicates that the superscript of x is y (x y )).

[0070] The intermediate nodes (= nodes other than leaf nodes) in all octrees constituting the octree sequence are sequentially visited and processed. Figure 1 As shown, the order of access (processing order) of each intermediate node is the order of time → LoD. That is, after sequentially processing all nodes of the octree at each time of the processing target LoD and processing all nodes at all times, the processing target is moved to the lower LoD.

[0071] Figure 1 An example is shown for the case where T=3 (time t=1, 2, 3) and K=2 (LoD=0, 1, 2). Each square in the figure indicates a voxel for each LoD. When LoD=0, the number of voxels is 1. When LoD=1, the voxel is divided into 2×2, and the number of voxels is 4. In the case of LoD=2, each voxel of LoD=1 is further divided into 2×2, and the number of voxels is 16. Note that although for ease of description, Figure 1 The voxel structure is shown in a plane, but the voxels are actually arranged in a three-dimensional shape. Therefore, in reality, the number of voxels with LoD = 1 is 8 (= 2 × 2 × 2), and the number of voxels with LoD = 2 is 64 (= 4 × 4 × 4). In the figure, gray squares indicate voxels that include points (voxels occupied by points), and white squares indicate voxels that do not include points (voxels not occupied by points). In addition, the numbers in the squares indicate the access order (processing order). That is, only voxels occupied by points are processed.

[0072] Note that the order in which nodes in the same LoD are visited at the same time can be arbitrary. Figure 1 The access order shown is an example, and for example, the access order "4" and "5" can be interchanged, or the access order "6" and "7" can be interchanged. However, the access order needs to match between the decoder and the encoder.

[0073] In this method, for the intermediate node (processing object node) being accessed, the occupancy states of its eight child nodes are encoded. The occupancy states of the eight child nodes are represented by 8 bits and there are a total of 2^8 = 256 patterns. For encoding, the encoder predicts the occupancy states of the child nodes of the processing object node based on the occupancy states of the nearby region. That is, according to the occupancy state of each node in the nearby region, a probability vector indicating the occupancy states of the child nodes of the processing object node (normalized in such a way that all elements among the 256 elements are non-negative and their sum is 1) is predicted. Each element of the prediction result (also referred to as the prediction probability vector) corresponds to an occupancy state. Using the prediction probability vector serving as an entropy model, the actual occupancy state is entropy-encoded. As described above, when the encoding of the occupancy states of the child nodes is completed for all intermediate nodes in all octrees, a bitstream is output.

[0074] The prediction (probability vector) of the occupancy states of the child nodes of the processing object node can be performed based on the occupancy states of the nearby region in the processing object frame or nearby frames (e.g., neighboring frames) or both. For example, as Figure 2 shown, the prediction (probability vector) of the occupancy states of the child nodes of the processing object node can be made based on the occupancy states of the nearby region of the processing object frame of the processing object LoD, the occupancy states of the nearby region of the neighboring frame (the immediately previous frame) of the processing object LoD, the occupancy states of the nearby region of the neighboring frame (the immediately following frame) of the processing object LoD, and the occupancy states of the nearby region of the neighboring frame (the immediately previous frame) of the lower-level LoD.

[0075] For example, the occupancy state {V_(k,i)}^t of the nearby region (9×9×9) (also referred to as the nearby voxels) of the processing object node n_i in Octree_(t,k) can be represented by the following expression (1). In addition, the occupancy state {V_(k,i)}^(t - 1) of the nearby region (9×9×9) of the processing object node n_i in Octree_(t - 1,k) can be represented by the following expression (2). In addition, the occupancy state {V_(k,i)}^(t + 1) of the nearby region (9×9×9) of the processing object node n_i in Octree_(t + 1,k) can be represented by the following expression (3). In addition, the occupancy state {V_(k + 1,i)}^(t - 1) of the nearby region (10×10×10) of the processing object node n_i in Octree_(t - 1,k + 1) can be represented by the following expression (4).

[0076] [Mathematical formula 1]

[0077]

[0078] [Mathematical formula 2]

[0079]

[0080] [Mathematical formula 3]

[0081]

[0082] [Mathematical formula 4]

[0083]

[0084]

[0085] Note that the nearby area can have any size. As in the example of Figure 2 , the nearby area of the LoD to be processed can be an area including 9×9×9 voxels, and the nearby area in the lower-level LoD can be an area including 10×10×10 voxels. Of course, the size of the nearby area of the LoD to be processed can also be a size other than 9×9×9, and the size of the nearby area in the lower-level LoD can also be a size other than 10×10×10. For example, as in the example indicated by the square of Figure 2 , the nearby area of the LoD to be processed can be an area including 5×5×5 voxels, and the nearby area in the lower-level LoD can be an area including 6×6×6 voxels.

[0086] By inputting the nearby voxel V into a predetermined 3D convolutional neural network (3DCNN), a context vector f is obtained. A neural network is a composite function of multiple linear transformations and non-linear transformations. Linear transformations and non-linear transformations are alternately performed on the input vector x to derive the output vector y. In the linear transformation, matrix multiplication and addition of a bias vector to a vector are performed. In the non-linear transformation, a non-linear function is applied to each element of the vector. The weight matrices and bias vectors of all linear transformations are learnable parameters. By stacking such linear transformation and non-linear transformation layers in multiple layers and optimizing the parameters, a complex transformation (function) can be approximated as x->y. Note that this neural network is also called a fully connected network or a multi-layer perceptron. A 3DCNN is a fully connected neural network in which the linear layer is a 3D convolutional layer (or a 3D deconvolutional layer), and includes a 3D convolutional layer and a pooling layer (downsampling operation, such as average pooling for calculating the average, etc.). The 3D convolutional layer performs 3D convolution operations. The 3D convolution operation is obtained by extending the input / output and filter of the two-dimensional convolution operation (2D convolution operation) to three dimensions (3D), and is an operation of convolving a filter of a predetermined size (three dimensions) for each piece of data in a three-dimensional region. The filter coefficients correspond to learnable weights.

[0087] For example, as Figure 3 As shown, by inputting the above four neighboring voxels \(\{V_{(k,i)}\}^t\), \(\{V_{(k,i)}\}^{t - 1}\), \(\{V_{(k,i)}\}^{t + 1}\), and \(\{V_{(k + 1,i)}\}^{t - 1}\) into independent 3DCNNs (3DCNN 11-1 to 3DCNN 11-4), the corresponding context vectors \(f\) are obtained. For example, assuming that the context vector \(\{f_{(k,i)}\}^t\) is obtained by inputting the neighboring voxel \(\{V_{(k,i)}\}^t\) into 3DCNN 11-1 (\(\{(3DCNN_0)^0\}()\)), this operation can be expressed as the following expression (5). Additionally, assuming that the context vector \(\{f_{(k,i)}\}^{t - 1}\) is obtained by inputting the neighboring voxel \(\{V_{(k,i)}\}^{t - 1}\) into 3DCNN 11-2 (\(\{(3DCNN_0)^{-1}\}()\)), this operation can be expressed as the following expression (6). Additionally, assuming that the context vector \(\{f_{(k,i)}\}^{t + 1}\) is obtained by inputting the neighboring voxel \(\{V_{(k,i)}\}^{t + 1}\) into 3DCNN 11-3 (\(\{(3DCNN_0)^{+1}\}()\)), this operation can be expressed as the following expression (7). Additionally, assuming that the context vector \(\{f_{(k + 1,i)}\}^{t - 1}\) is obtained by inputting the neighboring voxel \(\{V_{(k + 1,i)}\}^{t - 1}\) into 3DCNN 11-4 (\(\{(3DCNN_{+1})^{-1}\}()\)), this operation can be expressed as the following expression (8).

[0088] [Equation 5]

[0089]

[0090] [Equation 6]

[0091]

[0092] [Equation 7]

[0093]

[0094] [Equation 8]

[0095]

[0096] Next, the 4 obtained context vectors are concatenated in the dimension direction by the vector concatenation unit 12 to obtain the concatenated context vector \(f_i\). The concatenated context vector \(f_i\) can be expressed as the following expression (9).

[0097] [Equation 9]

[0098]

[0099] Next, input the connection context vector \(f_i\) into the multi-layer perceptron (MLP) 13. MLP 13 is a multi-layer perceptron, and based on the input connection context vector \(f_i\), a 256-dimensional prediction probability vector \(p_i\) is derived. The prediction probability vector \(p_i\) can be expressed as the following expression (10). The input dimension of MLP 13 is equal to the dimension of the connection context vector, and the output dimension is 256. That is, the prediction probability vector \(p_i\) is derived as in the following expression (11). Note that the activation function of the final layer of MLP 13 is the softmax function to perform normalization so that each element of the prediction probability vector \(p_i\) is non-negative and the sum is 1. The softmax function (also known as the normalized exponential function) is a function that transforms and outputs the result in such a way that the sum of multiple output values is 1.0 (= 100%). The range of each output value is from 0.0 to 1.0.

[0100] [Equation 10]

[0101] p i ∈[0, 1] 256

[0102] ……(10)

[0103] [Equation 11]

[0104] p i =MLP(f i )

[0105] ……(11)

[0106] The decoder accesses and processes all intermediate nodes in all octrees in the same order as the encoder. However, at the beginning of decoding, only the occupancy states of LoD = 0 are known ( Figure 1 the nodes with access orders "1", "2", and "3" in

[0107] ). The occupancy states of deeper LoD = 1, 2,... are reconstructed during decoding. Similar to the case of the above encoder, the decoder predicts the probability vector corresponding to the occupancy state of the child nodes of the intermediate node being accessed, and decodes the bitstream by using the prediction probability vector as the entropy model to generate (recover) the occupancy state of the child nodes (one of 256 patterns). The encoder and decoder only use the context known at the time of decoding for prediction. Therefore, the context applied to the prediction by the encoder and decoder is the same as each other, and thus, the obtained prediction probability vectors are also the same as each other. That is, the occupancy states of the child nodes of the processing object node are also the same as each other. That is, lossless compression can be achieved.

[0108] The decoder adds the occupancy states of the child nodes obtained by entropy-decoding the bitstream using such a prediction probability vector as the child nodes of the intermediate node being accessed to the octree. The decoder constructs a sequence of octrees by repeating such processing.

[0109] As described above, in VoxelContext-Net, a prediction probability vector is derived by inputting the vector obtained by concatenating context vectors in the dimensional direction into the MLP. The weight parameters of the MLP are optimized using learning data in the learning phase and fixed before the subsequent inference phase. Since the prediction probability vector is derived based on this weight parameter, it can be said that in the learning phase, it is learned which context vectors contribute to the prediction to what extent in order to be optimal for the learning data, and stored and fixed in the form of the values of the weight parameters of the MLP. That is, it has become difficult to control the prediction probability vector to minimize the bit size of the inference data.

[0110] Therefore, for example, when the number of parameters of the MLP is insufficient, when the number of learning data blocks is insufficient, when the characteristics of the learning data and the inference data deviate, etc., the prediction probability vector obtained by the MLP may no longer be optimal for the inference data. As a result, the coding efficiency may have decreased.

[0111] <3. Importance Coefficient Vector>

[0112] <Method 1>

[0113] Therefore, as Figure 4 shown at the top of the table in

[0114] A prediction probability vector indicating the probability values of the occupancy states that each child node of the processing target node can take is derived based on the composite vector; and an occupancy state encoding unit configured to encode information indicating the occupancy state of the child nodes of the processing target node by using the prediction probability vector.

[0115] For example, in an information processing method (also referred to as the first information processing method) executed by a first information processing device, a plurality of context vectors corresponding to a processing target node of 3D data having a tree structure are combined by using an importance coefficient vector for controlling the contribution degree to prediction to generate a composite vector; a prediction probability vector indicating probability values of occupancy states that each child node of the processing target node can take is derived based on the composite vector; and information indicating the occupancy state of the child node of the processing target node is encoded by using the prediction probability vector.

[0116] For example, an information processing device (also referred to as the second information processing device) for decoding a bitstream obtained by encoding geometric information of 3D data includes: a vector synthesis unit configured to combine a plurality of context vectors corresponding to a processing target node of 3D data having a tree structure by using an importance coefficient vector for controlling the contribution degree to prediction to generate a composite vector; a prediction probability vector derivation unit configured to derive a prediction probability vector indicating probability values of occupancy states that each child node of the processing target node can take based on the composite vector; and an occupancy state decoding unit configured to decode the bitstream by using the prediction probability vector to generate information indicating the occupancy state of the child node of the processing target node.

[0117] For example, in an information processing method (also referred to as the second information processing method) executed by a second information processing device, a plurality of context vectors corresponding to a processing target node of 3D data having a tree structure are combined by using an importance coefficient vector for controlling the contribution degree to prediction to generate a composite vector; a prediction probability vector indicating probability values of occupancy states that each child node of the processing target node can take is derived based on the composite vector; and the bitstream is decoded by using the prediction probability vector to generate information indicating the occupancy state of the child node of the processing target node.

[0118] Note that the above context vectors correspond to the occupancy state of a nearby region in the spatial direction of the processing target node in the processing target frame, or correspond to the occupancy state of a nearby region in the spatial direction of a node corresponding to the processing target node in a nearby frame in the time direction. In addition, the above plurality of context vectors may include: a context vector corresponding to the occupancy state of a nearby region in the processing target layer of the processing target frame; a context vector corresponding to the occupancy state of a nearby region in the processing target layer of a nearby frame; and a context vector corresponding to the occupancy state of a nearby region in a layer lower than the processing target layer of a nearby frame. In addition, the above prediction is a prediction of the occupancy state of a child node of the processing target node by using intra-frame correlation and inter-frame correlation.

[0119] By doing so, the contribution degree of each context vector to the prediction can be controlled by using the importance coefficient vector. Therefore, during inference, the contribution degree of each context vector can be adapted to the 3D data to be encoded. As a result, a decrease in the encoding efficiency can be suppressed.

[0120] Note that the synthesized vector can be a weighted sum obtained by weighting each of the multiple context vectors by each element of the importance coefficient vector. For example, in the first information processing device and the second information processing device, the vector synthesis unit can derive a weighted sum by weighting each of the multiple context vectors by each element of the importance coefficient vector, and use the weighted sum as the synthesized vector. By doing so, by controlling the value of each element of the importance coefficient vector, the contribution degree of each context vector to the prediction can be controlled.

[0121] Furthermore, the prediction probability vector can be derived by inputting the synthesized vector to a multi-layer perceptron. For example, in the first information processing device and the second information processing device, the prediction probability vector derivation unit can derive the prediction probability vector by using a multi-layer perceptron that uses the synthesized vector as an input.

[0122] Furthermore, the context vector can be derived. For example, the first information processing device and the second information processing device may further include a context vector derivation unit configured to derive a context vector based on the occupancy state of a nearby area.

[0123] In this case, mutually different 3DCNNs can be used to derive a context vector corresponding to the processing object frame of the processing object LoD, a context vector corresponding to the frame immediately before the processing object frame of the processing object LoD, a context vector corresponding to the frame immediately after the processing object frame of the processing object LoD, and a context vector corresponding to the frame immediately before the processing object frame of the LoD one level lower than the processing object LoD. For example, in the first information processing device and the second information processing device, the context vector derivation unit can derive a context vector corresponding to the occupancy state of a nearby area in the processing object layer of the processing object frame, a context vector corresponding to the occupancy state of a nearby area in the processing object layer of the frame immediately before the processing object frame, a context vector corresponding to the occupancy state of a nearby area in the processing object layer of the frame immediately after the processing object frame, and a context vector corresponding to the occupancy state of a nearby area in the layer lower than the processing object layer of the frame immediately before the processing object frame by using mutually different neural networks.

[0124] In this case, the dimension of the importance coefficient vector can be fixed to four. For example, in the first information processing device and the second information processing device, the dimension of the importance coefficient vector can be four.

[0125] In addition, in this case, the occupancy state of the nearby area can be set. For example, the first information processing device and the second information processing device may further include a nearby area occupancy state setting unit configured to set: the occupancy state of the nearby area in the processing object layer of the processing object frame; the occupancy state of the nearby area in the processing object layer of the frame immediately before the processing object frame; the occupancy state of the nearby area in the processing object layer of the frame immediately after the processing object frame; and the occupancy state of the nearby area in the layer lower than the processing object layer of the frame immediately before the processing object frame. Then, the context vector derivation unit can derive each context vector based on the occupancy state of each nearby area set by the nearby area occupancy state setting unit.

[0126] Note that in the encoder, any of the above-mentioned setting methods of the importance coefficient vector can be adopted. For example, multiple candidates of the importance coefficient vector can be prepared, the code amount in the case of applying each candidate can be estimated, and the candidate for minimizing the code amount can be selected. Note that the decoder applies the candidate set by the encoder. For example, the first information processing device may further include: a code amount estimation unit configured to estimate the code amount in the case of applying the prediction probability vector derived by the prediction probability vector derivation unit; and a selection unit configured to select the importance coefficient vector to be applied from multiple candidates based on the code amount estimated by the code amount estimation unit. Then, the vector synthesis unit can generate a synthesis vector corresponding to each of the multiple candidates of the importance coefficient vector. Then, the prediction probability vector derivation unit can derive a prediction probability vector corresponding to each of the multiple candidates. Then, the code amount estimation unit can estimate the code amount corresponding to each of the multiple candidates. Then, the selection unit can select, from the code amounts respectively corresponding to each of the multiple candidates, the candidate corresponding to the minimum code amount as the importance coefficient vector to be applied to the encoding of the information indicating the occupancy state of the child node of the processing object node. Then, the occupancy state encoding unit can encode the information indicating the occupancy state of the child node of the processing object node by using the candidate selected by the selection unit. By doing so, it is easier to optimize the importance coefficient vector for the data to be encoded during inference. Therefore, a decrease in encoding efficiency can be suppressed.

[0127] Note that the importance coefficient vector can be transmitted from the encoder to the decoder, as Figure 4 as shown at the top of the table in

[0128] <Method 1-1>

[0129] In this case, as shown in the second row from the top of the table in Figure 4 the importance coefficient vector can be encoded by vector quantization (Method 1-1). That is, the index corresponding to the importance coefficient vector can be entropy encoded.

[0130] For example, in the first information processing device, the importance coefficient vector encoding unit can perform entropy encoding on the index indicating the importance coefficient vector. In addition, in the second information processing device, the importance coefficient vector decoding unit can perform entropy decoding on the bitstream to generate (recover) the index indicating the importance coefficient vector.

[0131] <Geometric Information Encoding Device>

[0132] Figure 5 is a block diagram showing an example of the configuration of a geometric information encoding device as one mode of the information processing device to which the present technology is applied. Figure 5 The geometric information encoding device 100 shown encodes the geometric information of a point cloud (3D data). The geometric information encoding device 100 encodes the geometric information by applying the above Method 1 or Method 1-1.

[0133] Note that in Figure 5 the main processing unit, data stream, etc. are shown, and Figure 5 what is shown is not necessarily all. That is, in the geometric information encoding device 100, there may be Figure 5 processing units not shown in a box in Figure 5 or there may be processing or data streams not shown by an arrow, etc. in

[0134] As Figure 5 shown, the geometric information encoding device 100 includes a quantization unit 111, an octree construction unit 112, and an octree encoding unit 113.

[0135] The quantization unit 111 acquires the geometric information of the point cloud (point cloud sequence {Pc_t|t = 1,...,T}) at each time and performs quantization. That is, the quantization unit 111 converts the representation method of the input geometric information into a voxel representation. The quantization unit 111 supplies the quantized geometric information (voxel data) at each time to the octree construction unit 112.

[0136] The octree construction unit 112 converts the representation method of the voxel data at each time into an octree representation and constructs an octree at each time (that is, octree sequence {Octree_t|t = 1,...,T}). The octree construction unit 112 supplies the octree sequence to the octree encoding unit 113.

[0137] The octree encoding unit 113 encodes the octree sequence (octree at each time) to generate a bitstream. The octree encoding unit 113 outputs the generated bitstream to the outside of the geometric information encoding device 100. This bitstream can be provided to a geometric information decoding device that decodes the geometric information bitstream via any transmission path or any recording medium.

[0138] <Octree Encoding Unit>

[0139] Figure 7 is a block diagram showing Figure 5 an example of the configuration of the octree encoding unit 113. The octree encoding unit 113 encodes the octree sequence by applying the above method 1 or method 1-1.

[0140] Note that in Figure 6 , the main processing unit, data stream, etc. are shown, and Figure 6 what is shown is not necessarily all. That is, in the octree encoding unit 113, there may be Figure 6 processing units not shown in a box in Figure 6 , or there may be processing or data streams not shown by an arrow, etc. in

[0141] As Figure 6 shown, the octree encoding unit 113 includes a codebook storage unit 131, a nearby voxel setting unit 132, a frame memory 133, a context vector derivation unit 134, a vector synthesis unit 135, an MLP 136, a bit size approximation derivation unit 137, a codeword selection unit 138, an importance coefficient vector encoding unit 139, and an occupancy state encoding unit 140.

[0142] The codebook storage unit 131 includes a storage medium and stores a codebook C. The codebook storage unit 131 supplies the stored codebook C to the vector synthesis unit 135 and the codeword selection unit 138. The codebook C is a set (candidate) of importance coefficient vectors and can be expressed by the following expression (12). Note that each importance coefficient vector α^c (c = 0, ..., |C| - 1) included in the codebook C is also referred to as a codeword. It is assumed that the size C of the codebook is set as a hyperparameter of the model before learning. In addition, it is also assumed that all codewords in the codebook and the |C|-element probability table for entropy encoding the indices of the codewords are optimized during learning, for example, as part of the parameters of the model.

[0143] [Equation 12]

[0144] C = {α 0 , …, α |C|-1}

[0145] ……(12)

[0146] The nearby voxel setting unit 132 acquires an octree (octree sequence) at each time. The nearby voxel setting unit 132 supplies the acquired octree to the frame memory 133 for storage. In addition, the nearby voxel setting unit 132 also reads from the frame memory 133 the octree sequence (octrees of the processing object frame and nearby frames) required for setting nearby voxels. The nearby voxel setting unit 132 sets nearby voxels for the processing object frame, nearby frames, etc. based on the read octree sequence. That is, the nearby voxel setting unit 132 can also be referred to as a nearby voxel setting unit or a nearby area occupancy state setting unit.

[0147] As described above, the nearby voxels indicate the occupancy state of the nearby area of the processing object node. For example, the nearby voxel setting unit 132 sets the nearby voxels {V_(k,i)}^t in the processing object frame of the processing object LoD, the nearby voxels {V_(k,i)}^(t - 1) in the frame immediately before the processing object frame of the processing object LoD, the nearby voxels {V_(k,i)}^(t + 1) in the frame immediately after the processing object frame of the processing object LoD, and the nearby voxels {V_(k + 1,i)}^(t - 1) in the frame immediately before the processing object frame of the LoD one level lower than the processing object LoD.

[0148] The nearby voxel setting unit 132 supplies the set nearby voxels to the context vector derivation unit 134. For example, the nearby voxel setting unit 132 supplies the nearby voxels {V_(k,i)}^t to the 3D CNN 151. In addition, the nearby voxel setting unit 132 supplies the nearby voxels {V_(k,i)}^(t - 1) to the 3D CNN 152. In addition, the nearby voxel setting unit 132 supplies the nearby voxels {V_(k,i)}^(t + 1) to the 3D CNN 153. In addition, the nearby voxel setting unit 132 supplies the nearby voxels {V_(k + 1,i)}^(t - 1) to the 3D CNN 154.

[0149] The frame memory 133 includes a storage medium and stores the octree supplied from the nearby voxel setting unit 132. In addition, the frame memory 133 also supplies the octree (or octree sequence) requested by the nearby voxel setting unit 132 to the nearby voxel setting unit 132.

[0150] The context vector derivation unit 134 derives a context vector based on the nearby voxels supplied from the nearby voxel setting unit 132 (that is, the occupancy state of the nearby area), and supplies the context vector to the vector synthesis unit 135. For example, this context vector corresponds to the occupancy state of the nearby area in the spatial direction of the processing object node in the processing object frame, or corresponds to the occupancy state of the nearby area in the spatial direction of the node corresponding to the processing object node in the nearby frame in the time direction.

[0151] For example, the context vector derivation unit 134 derives context vectors corresponding to each of a plurality of nearby voxels by using neural networks different from each other. The plurality of context vectors include: a context vector corresponding to the occupancy state of a nearby area in the processing object layer of the processing object frame; a context vector corresponding to the occupancy state of a nearby area in the processing object layer of a nearby frame; and a context vector corresponding to the occupancy state of a nearby area in a layer lower than the processing object layer of a nearby frame. For example, the context vector derivation unit 134 includes 3DCNNs 151 to 154. The 3DCNN 151 is {(3DCNN_0)^0}() corresponding to the processing object frame of the processing object LoD, generates a context vector {f_(k,i)}^t corresponding to the processing object frame of the processing object LoD based on the input nearby voxels {V_(k,i)}^t, and supplies the context vector {f_(k,i)}^t to the vector synthesis unit 135. The 3DCNN 152 is {(3DCNN_0)^-1}() corresponding to the frame immediately before the processing object frame of the processing object LoD, generates a context vector {f_(k,i)}^(t-1) corresponding to the frame immediately before the processing object frame of the processing object LoD based on the input nearby voxels {V_(k,i)}^(t-1), and supplies the context vector {f_(k,i)}^(t-1) to the vector synthesis unit 135. The 3DCNN 153 is {(3DCNN_0)^+1}() corresponding to the frame immediately after the processing object frame of the processing object LoD, generates a context vector {f_(k,i)}^(t+1) corresponding to the frame immediately after the processing object frame of the processing object LoD based on the input nearby voxels {V_(k,i)}^(t+1), and supplies the context vector {f_(k,i)}^(t+1) to the vector synthesis unit 135. The 3DCNN 154 is {(3DCNN_+1)^-1}() corresponding to the frame immediately before the processing object frame of the LoD one level lower than the processing object LoD, generates a context vector {f_(k+1,i)}^(t-1) corresponding to the frame immediately before the processing object frame of the LoD one level lower than the processing object LoD based on the input nearby voxels {V_(k+1,i)}^(t-1), and supplies the context vector {f_(k+1,i)}^(t-1) to the vector synthesis unit 135.

[0152] That is to say, the context vector derivation unit 134 can derive a context vector corresponding to the occupancy state of a nearby area in the processing object layer of the processing object frame, a context vector corresponding to the occupancy state of a nearby area in the processing object layer of the frame immediately before the processing object frame, a context vector corresponding to the occupancy state of a nearby area in the processing object layer of the frame immediately after the processing object frame, and a context vector corresponding to the occupancy state of a nearby area in a layer lower than the processing object layer of the frame immediately before the processing object frame, by using mutually different neural networks.

[0153] Note that in this specification, for the sake of simplicity of description, the context vector {f_(k,i)}^t is also referred to as x_0. Similarly, the context vector {f_(k,i)}^(t - 1) is also referred to as x_1. Similarly, the context vector {f_(k,i)}^(t + 1) is also referred to as x_2. Similarly, the context vector {f_(k + 1,i)}^(t - 1) is also referred to as x_3.

[0154] The vector synthesis unit 135 acquires a plurality of context vectors supplied from the context vector derivation unit 134. For example, the vector synthesis unit 135 acquires the context vector {f_(k,i)}^t supplied from the 3D CNN 151. In addition, the vector synthesis unit 135 acquires the context vector {f_(k,i)}^(t - 1) supplied from the 3D CNN 152. In addition, the vector synthesis unit 135 acquires the context vector {f_(k,i)}^(t + 1) supplied from the 3D CNN 153. In addition, the vector synthesis unit 135 acquires the context vector {f_(k + 1,i)}^(t - 1) supplied from the 3D CNN 154. In addition, the vector synthesis unit 135 reads and acquires the codebook C (candidates of the importance coefficient vector α) from the codebook storage unit 131.

[0155] The vector synthesis unit 135 combines the plurality of acquired context vectors by using the acquired importance coefficient vector to generate a synthesized vector u_i, and supplies the synthesized vector u_i to the MLP 136. That is to say, the vector synthesis unit 135 combines a plurality of context vectors corresponding to the processing object nodes of the 3D data having a tree structure by using the importance coefficient vector for controlling the contribution degree of the context vector to the prediction of the occupancy state of the child nodes of the processing object node, by utilizing the intra-frame correlation and the inter-frame correlation, to generate a synthesized vector.

[0156] For example, the vector synthesis unit 135 can derive a weighted sum by weighting each element of the importance coefficient vector to each of the plurality of context vectors, and use the weighted sum as the synthesized vector. For example, as Figure 6 As shown, when the context vector derivation unit 134 obtains four context vectors ({f_(k,i)}^t, {f_(k,i)}^(t - 1), {f_(k,i)}^(t + 1), and {f_(k + 1,i)}^(t - 1)) from the context vector, the vector synthesis unit 135 obtains a synthesized vector by deriving a weighted sum of the context vectors using the importance coefficient vector α_i = [α_(i,0), α_(i,1), α_(i,2), α_(i,3)] ∈ [0,1]^4 (where normalization is performed to obtain Σ_j[{α_(i,j)} = 1 (a subscript i is added because there are differences for each node n_i)).

[0157] As described above, the vector synthesis unit 135 can generate the synthesized vector u_i by the following expression (13).

[0158] [Equation 13]

[0159] u i = ∑ j α i，j x j

[0160] ……(13)

[0161] In this way, the contribution of each context vector to the prediction can be explicitly controlled by using the value of the importance coefficient vector α_i. That is, an explicit prediction mode can be controlled in which the feature amount of which frame is emphasized more. For example, when intra - picture prediction should be emphasized more, the vector synthesis unit 135 can set the value of the importance coefficient corresponding to the context vector corresponding to the frame to be processed to be large and set the value of the importance coefficient corresponding to the context vector corresponding to the nearby frame to be small. Conversely, when inter - picture prediction should be emphasized more, the vector synthesis unit 135 can set the value of the importance coefficient corresponding to the context vector corresponding to the nearby frame to be large and set the value of the importance coefficient corresponding to the context vector corresponding to the frame to be processed to be small.

[0162] Note that the vector synthesis unit 135 can generate the synthesized vector (u_i)^c for each of the multiple candidates of the importance coefficient vector included in the codebook C by the above method and supply the synthesized vector (u_i)^c to the MLP 136. For example, the vector synthesis unit 135 can generate the synthesized vector (u_i)^c for each candidate of the importance coefficient vector by the following expression (14).

[0163] [Equation 14]

[0164]

[0165] The MLP 136 derives a prediction probability vector p_i that indicates the probability values of the occupancy states that each child node of the processing target node can take, based on the synthesized vector u_i supplied from the vector synthesis unit 135. That is to say, the MLP 136 can also be referred to as a prediction probability vector derivation unit. Note that the MLP 136 can be a multi-layer perceptron that uses the synthesized vector u_i supplied from the vector synthesis unit 135 as input to derive the prediction probability vector p_i. That is to say, the MLP 136 can derive the prediction probability vector by using a multi-layer perceptron with the synthesized vector as input. The MLP 136 supplies the derived prediction probability vector p_i to the bit size approximation derivation unit 137.

[0166] Note that the MLP 136 can derive the prediction probability vector (p_i)^c for each of the multiple candidates of the importance coefficient vectors included in the codebook C by the above method, and supply the prediction probability vector (p_i)^c to the bit size approximation derivation unit 137. For example, the MLP 136 can generate the prediction probability vector (p_i)^c for each candidate of the importance coefficient vector by the following expression (15). Note that in expression (15), MLP' is used to distinguish from Figure 3 the MLP 13 (clearly indicating that the neural networks are different). The input dimension of MLP' is equal to the dimension of the context vector, the output dimension is 256, and the softmax function is used as the activation function of the final layer.

[0167] [Mathematical formula 15]

[0168]

[0169] The bit size approximation derivation unit 137 obtains the prediction probability vector p_i supplied from the MLP 136, estimates the code amount (total compressed bit size) in the case where the prediction probability vector p_i is applied, and derives an estimated value of the code amount (also referred to as the bit size approximation R_i). That is to say, the bit size approximation derivation unit 137 can also be referred to as a code amount estimation unit. Note that the bit size approximation derivation unit 137 can estimate the code amount corresponding to each of the multiple candidates of the importance coefficient vectors included in the codebook C. For example, the bit size approximation derivation unit 137 can derive the bit size approximation R_i(c) for each candidate of the importance coefficient vector by the following expression (16).

[0170] R_i(c) = -log2({element value of pic corresponding to the true occupancy state}) - log2({element value of the c-th element of the probability table for importance coefficient compression})... (16)

[0171] The bit size approximation derivation unit 137 supplies the bit size approximation value \(R_i(c)\) corresponding to each candidate of the importance coefficient vector derived in this way and the prediction probability vector \((p_i)^c\) used for estimation to the codeword selection unit 138.

[0172] The codeword selection unit 138 obtains the bit size approximation value \(R_i(c)\) corresponding to each candidate of the importance coefficient vector and the prediction probability vector \((p_i)^c\) supplied from the bit size approximation derivation unit 137. In addition, the codeword selection unit 138 reads and obtains the codebook \(C(\text{importance coefficient vector}\alpha^c)\) from the codebook storage unit 131. Based on the bit size approximation value \(R_i(c)\), the codeword selection unit 138 selects the importance coefficient vector corresponding to the prediction probability vector to be applied to the encoding of the occupancy state from among multiple candidates. That is, the codeword selection unit 138 can also be referred to as a selection unit that selects the importance coefficient vector to be applied from among multiple candidates based on the code amount estimated by the code amount estimation unit.

[0173] For example, the codeword selection unit 138 can select a candidate for minimizing the code amount. That is, the codeword selection unit 138 can select, from among the code amounts corresponding to each of the multiple candidates, the candidate corresponding to the minimum code amount as the importance coefficient vector to be applied to the encoding of the information indicating the occupancy state of the child node of the processing target node. For example, the codeword selection unit 138 can calculate the index \((c_i)^*\) of the candidate of the importance coefficient vector to minimize the bit size approximation value \(R_i(c)\) by performing the operation of the following expression (17).

[0174] [Mathematical formula 16]

[0175]

[0176] The codeword selection unit 138 supplies the index \((c_i)^*\) corresponding to the selected candidate of the importance coefficient vector to the importance coefficient vector encoding unit 139. In addition, the codeword selection unit 138 supplies the prediction probability vector \((p_i)^{(c_i)^*}\) corresponding to the selected index \((c_i)^*\) to the occupancy state encoding unit 140.

[0177] The importance coefficient vector encoding unit 139 performs entropy encoding on the index \((c_i)^*\) supplied from the codeword selection unit 138 to generate its bitstream (also referred to as the importance coefficient bitstream). That is to say, it can also be said that the importance coefficient vector encoding unit 139 encodes the selected importance coefficient vector. The importance coefficient vector encoding unit 139 outputs the generated importance coefficient bitstream to the outside of the octree encoding unit 113 (outside the geometric information encoding device 100). This importance coefficient bitstream can be provided to the geometric information decoding device that decodes the geometric information bitstream via any transmission path or any recording medium.

[0178] The occupancy state encoding unit 140 acquires the prediction probability vector \((p_i)^{(c_i)^*}\) supplied from the codeword selection unit 138. In addition, the occupancy state encoding unit 140 acquires the information indicating the occupancy state of the child nodes of the processing target node (that is, the actual occupancy state) supplied from the octree construction unit 112. The occupancy state encoding unit 140 encodes the information indicating the occupancy state of the child nodes of the processing target node by using the prediction probability vector to generate its bitstream (occupancy state bitstream). For example, the occupancy state encoding unit 140 can perform entropy encoding on the information indicating the occupancy state of the child nodes of the processing target node by using the prediction probability vector \((p_i)^{(c_i)^*}\) corresponding to the candidate of the importance coefficient vector selected by the codeword selection unit 138 as the entropy model to generate the occupancy state bitstream. The occupancy state encoding unit 140 outputs the generated occupancy state bitstream to the outside of the octree encoding unit 113 (outside the geometric information encoding device 100). This occupancy state bitstream can be provided to the geometric information decoding device that decodes the geometric information bitstream via any transmission path or any recording medium.

[0179] Note that the geometric information encoding device 100 (octree encoding unit 113) can output the importance coefficient bitstream and the occupancy state bitstream together as one bitstream. That is to say, the importance coefficient bitstream and the occupancy state bitstream can be provided to the geometric information decoding device as one bitstream.

[0180] With the above configuration, the geometric information encoding device 100 can control the contribution rate of each context vector to the prediction by using the importance coefficient vector, and can suppress the reduction of the encoding efficiency. In addition, since the geometric information encoding device 100 can perform continuous prediction mode control instead of discrete prediction mode selection in a 2D moving image codec or the like, the reduction of the encoding efficiency can be suppressed.

[0181] <Flow of encoding process>

[0182] The geometric information encoding device 100 encodes the geometric information as described above by performing an encoding process. An example of the flow of this encoding process will be described with reference to the flowchart in Figure 7 below.

[0183] When the encoding process starts, the quantization unit 111 quantizes the geometric information of the point cloud in step S101 and converts the geometric information into a voxel representation. The quantization unit 111 performs such processing at each time (each frame) and generates voxel data at each time (each frame).

[0184] In step S102, the octree construction unit 112 converts the voxel data into an octree representation. That is, the octree construction unit 112 constructs an octree. The octree construction unit 112 performs such processing at each time (each frame) to generate an octree sequence.

[0185] In step S103, the octree encoding unit 113 performs an octree encoding process and encodes the octree to generate a bitstream (importance coefficient bitstream and occupancy status bitstream). At this time, the octree encoding unit 113 performs encoding using not only the octree of the frame to be processed but also the octrees of nearby frames. In addition, the octree encoding unit 113 performs such processing at each time (each frame) to encode the octree sequence.

[0186] When the processing of step S113 ends, the encoding process ends.

[0187] <Flow of Octree Encoding Process>

[0188] Next, an example of the flow of the octree encoding process performed in step S103 of Figure 8 will be described with reference to the flowcharts in Figure 9 and Figure 7 below.

[0189] When the octree encoding process starts, in step S131 of Figure 8 below, the importance coefficient vector encoding unit 139 and the occupancy status encoding unit 140 initialize the occupancy status bitstream and the importance coefficient bitstream.

[0190] In step S132, the nearby voxel setting unit 132 sets nearby voxels as described above. In step S133, the context vector derivation unit 134 derives a context vector corresponding to each nearby voxel as described above.

[0191] In step S134, as described above, the vector synthesis unit 135 combines context vectors by using the importance coefficient vector (candidates of the processing target) to generate a synthesized vector. For example, the vector synthesis unit 135 combines context vectors by using weighted operations of the importance coefficient vector.

[0192] In step S135, the MLP 136 inputs the synthesized vector to a multi-layer perceptron (MLP') to derive a prediction probability vector. That is, the MLP 136 generates a prediction probability vector corresponding to the candidates of the processing target of the importance coefficient.

[0193] In step S136, the bit size approximation derivation unit 137 derives a bit size approximation in the case where the prediction probability vector is applied. That is, the bit size approximation derivation unit 137 derives a bit size approximation corresponding to the candidates of the processing target of the importance coefficient.

[0194] In step S137, the codeword selection unit 138 determines whether processing has been performed on all codewords included in the codebook, and performs each of the processes of steps S134 to S137 for each codeword until it is determined that processing has been performed on all codewords. Then, when it is determined in step S137 that processing has been performed on all codewords, the process proceeds to Figure 9 。

[0195] In Figure 9 step S151, the codeword selection unit 138 selects a codeword (candidate of the importance coefficient vector) that minimizes the bit size approximation derived in step S136.

[0196] In step S152, the importance coefficient vector encoding unit 139 performs entropy encoding on the index of the selected codeword and adds the encoded data to the importance coefficient bit stream.

[0197] In step S153, the occupancy state encoding unit 140 performs entropy encoding on the information indicating the occupancy state of the child nodes of the processing target node by using the prediction probability vector corresponding to the selected codeword as an entropy model, and adds the encoded data to the occupancy state bit stream.

[0198] In step S154, the occupancy state encoding unit 140 determines whether all nodes have been processed, and for each node, performs Figure 8 each of the processes of steps S132 to S137 of Figure 9 and each of the processes of steps S151 to S154 of Figure 9 When it is determined in step S154 that processing has been performed on all nodes, the processing proceeds to step S155.

[0199] In step S155, the occupancy state encoding unit 140 determines whether all frames have been processed, and for each node of each frame, performs Figure 8 each process of steps S132 to S137 of Figure 9 and each process of steps S151 to S155 of Figure 9 until it is determined that all nodes of all frames of the processing target LoD have been processed. Then, when it is determined in

[0200] step S155 that processing has been performed on all frames, the processing proceeds to step S156. Figure 8 In step S156, the occupancy state encoding unit 140 determines whether all layers have been processed, and for each node of each frame of each LoD, performs Figure 9 ]>each process of steps S132 to S137 of Figure 9 and each process of steps S151 to S156 of

[0201] until it is determined that all nodes of all frames of all LoDs have been processed. Then, when it is determined in

[0202] step S156 that processing has been performed on all layers, the processing proceeds to step S157. Figure 7

[0203] By performing each process as described above, the geometric information encoding device 10 can control the contribution rate of each context vector to prediction by using the importance coefficient vector, and can suppress a decrease in coding efficiency. In addition, since the geometric information encoding device 100 can perform continuous prediction mode control instead of discrete prediction mode selection in a 2D moving image codec or the like, a decrease in coding efficiency can be suppressed.

[0204] <Geometric Information Decoding Device>

[0205] Figure 10 is a block diagram showing an example of the configuration of a geometric information decoding device as one mode of an information processing device to which the present technology is applied. Figure 10 ​The geometric information decoding device 200 shown decodes a bitstream obtained by encoding the geometric information of a point cloud (3D data). The geometric information decoding device 200 applies the above-described method 1 or method 1-1 to decode the bitstream to generate (recover) geometric information.

[0206] Note that in Figure 10 the main processing unit, data streams, etc. are shown, and Figure 10 what is shown is not necessarily all. That is, in the geometric information decoding device 200, there may be Figure 10 processing units not shown in a box in Figure 10 or there may be processing or data streams not shown by an arrow, etc. in

[0207] As Figure 10 shown, the geometric information decoding device 200 includes an octree decoding unit 211 and a point cloud construction unit 212.

[0208] The octree decoding unit 211 acquires the bitstream of the geometric information of the point cloud input to the geometric information decoding device 200. For example, the octree decoding unit 211 acquires the importance coefficient bitstream and the occupancy state bitstream generated by the geometric information encoding device 100 or the like. The octree decoding unit 211 decodes the acquired bitstreams (importance coefficient bitstream and occupancy state bitstream), constructs an octree, and supplies the octree to the point cloud construction unit 212. The octree decoding unit 211 performs such processing for each time (each frame) and supplies an octree sequence to the point cloud construction unit 212.

[0209] The point cloud construction unit 212 constructs a point cloud (point cloud sequence) based on the octree (octree sequence). The point cloud construction unit 212 outputs the constructed point cloud (point cloud sequence) as decoded geometric information to the outside of the geometric information decoding device 200. For example, this geometric information can be used for rendering in a later stage.

[0210] <Octree Decoding Unit>

[0211] Figure 11 is a block diagram showing an example of the configuration of the octree decoding unit 211 in Figure 10 The octree decoding unit 211 applies the above-described method 1 or method 1-1 to decode the bitstream to generate (recover) an octree sequence.

[0212] Note that in Figure 11 the main processing unit, data streams, etc. are shown, and Figure 11 what is shown is not necessarily all. That is, in the octree decoding unit 211, there may be Figure 11 processing units not shown in a box in Figure 11 Processes or data flows not shown by arrows or the like.

[0213] As Figure 11 As shown, the octree decoding unit 211 includes a frame memory 231, a nearby voxel setting unit 232, a context vector derivation unit 233, a significance coefficient vector decoding unit 234, a codebook storage unit 235, a vector synthesis unit 236, an MLP 237, and an occupancy state decoding unit 238.

[0214] The frame memory 231 includes a storage medium and constructs an octree by acquiring and storing information supplied from the occupancy state decoding unit 238 and indicating the occupancy states of the child nodes of the node to be processed. That is, the frame memory 231 stores the generated (restored) octree. Note that since decoding is performed for each frame, the frame memory 231 can store the octree for each frame. That is, it can be said that the frame memory 231 stores a sequence of generated (restored) octrees. In addition, the frame memory 231 also supplies the octree (or octree sequence) requested by the nearby voxel setting unit 232 to the nearby voxel setting unit 232.

[0215] The nearby voxel setting unit 232 performs similar processing to the nearby voxel setting unit 132. For example, the nearby voxel setting unit 232 also reads from the frame memory 231 the octree sequence (octrees of the frame to be processed and nearby frames) required for setting nearby voxels. The nearby voxel setting unit 232 sets the nearby voxels for the frame to be processed, nearby frames, etc. based on the read octree sequence. That is, the nearby voxel setting unit 232 can also be referred to as a nearby voxel setting unit or a nearby area occupancy state setting unit.

[0216] For example, the nearby voxel setting unit 232 sets the nearby voxels {V_(k,i)}^t in the frame to be processed of the processing object LoD, the nearby voxels {V_(k,i)}^(t - 1) in the frame immediately before the frame to be processed of the processing object LoD, the nearby voxels {V_(k,i)}^(t + 1) in the frame immediately after the frame to be processed of the processing object LoD, and the nearby voxels {V_(k + 1,i)}^(t - 1) in the frame immediately before the frame to be processed of the LoD one level lower than the processing object LoD.

[0217] The nearby voxel setting unit 232 supplies the set nearby voxels to the context vector derivation unit 233. For example, the nearby voxel setting unit 232 supplies the nearby voxels {V_(k,i)}^t to the 3DCNN 251. In addition, the nearby voxel setting unit 232 supplies the nearby voxels {V_(k,i)}^(t - 1) to the 3DCNN 252. In addition, the nearby voxel setting unit 232 supplies the nearby voxels {V_(k,i)}^(t + 1) to the 3DCNN 253. In addition, the nearby voxel setting unit 232 supplies the nearby voxels {V_(k + 1,i)}^(t - 1) to the 3DCNN 254.

[0218] The context vector derivation unit 233 performs a process similar to that of the context vector derivation unit 134. For example, the context vector derivation unit 233 derives a context vector based on the nearby voxels supplied from the nearby voxel setting unit 232 (that is, the occupancy state of the nearby area), and supplies the context vector to the vector synthesis unit 236.

[0219] For example, the context vector derivation unit 233 derives context vectors corresponding to each of a plurality of nearby voxels by using neural networks different from each other. For example, the context vector derivation unit 233 includes 3DCNNs 251 to 254. The 3DCNN 251 is {(3DCNN_0)^0}() corresponding to the processing object frame of the processing object LoD, generates a context vector {f_(k,i)}^t corresponding to the processing object frame of the processing object LoD based on the input nearby voxels {V_(k,i)}^t, and supplies the context vector to the vector synthesis unit 236. The 3DCNN 252 is {(3DCNN_0)^-1}() corresponding to the frame immediately before the processing object frame of the processing object LoD, generates a context vector {f_(k,i)}^(t-1) corresponding to the frame immediately before the processing object frame of the processing object LoD based on the input nearby voxels {V_(k,i)}^(t-1), and supplies the context vector to the vector synthesis unit 236. The 3DCNN 253 is {(3DCNN_0)^+1}() corresponding to the frame immediately after the processing object frame of the processing object LoD, generates a context vector {f_(k,i)}^(t+1) corresponding to the frame immediately after the processing object frame of the processing object LoD based on the input nearby voxels {V_(k,i)}^(t+1), and supplies the context vector to the vector synthesis unit 236. The 3DCNN 254 is {(3DCNN_+1)^-1}() corresponding to the frame immediately before the processing object frame of the LoD one level lower than the processing object LoD, generates a context vector {f_(k+1,i)}^(t-1) corresponding to the frame immediately before the processing object frame of the LoD one level lower than the processing object LoD based on the input nearby voxels {V_(k+1,i)}^(t-1), and supplies the context vector to the vector synthesis unit 236.

[0220] That is to say, the context vector derivation unit 233 can derive a context vector corresponding to the occupancy state of a nearby area in the processing object layer of the processing object frame, a context vector corresponding to the occupancy state of a nearby area in the processing object layer of the frame immediately before the processing object frame, a context vector corresponding to the occupancy state of a nearby area in the processing object layer of the frame immediately after the processing object frame, and a context vector corresponding to the occupancy state of a nearby area in the layer lower than the processing object layer of the frame immediately before the processing object frame by using neural networks different from each other.

[0221] The importance coefficient vector decoding unit 234 obtains the importance coefficient bitstream input to the geometric information decoding device 200. The importance coefficient vector decoding unit 234 decodes the importance coefficient bitstream to generate an importance coefficient vector. For example, the importance coefficient vector decoding unit 234 performs entropy decoding on the importance coefficient bitstream to generate (recover) an index \((c_i)^*\) indicating the importance coefficient vector. The importance coefficient vector decoding unit 234 supplies the generated (recovered) index \((c_i)^*\) to the codebook storage unit 235.

[0222] Similar to the codebook storage unit 131, the codebook storage unit 235 includes a storage medium and stores the codebook C. The codebook storage unit 235 supplies the importance coefficient vector \(\alpha^{(c_i)^*}\) (that is, the importance coefficient vector applied during encoding) included in the stored codebook C and corresponding to the index \((c_i)^*\) supplied from the importance coefficient vector decoding unit 234 to the vector synthesis unit 236. Therefore, the vector synthesis unit 236 can combine vectors by applying the candidate of the importance coefficient vector applied during encoding.

[0223] The vector synthesis unit 236 performs a process similar to that of the vector synthesis unit 135 to combine context vectors. For example, the vector synthesis unit 236 obtains a plurality of context vectors supplied from the context vector derivation unit 233. For example, the vector synthesis unit 236 obtains the context vector \(\{f_{(k,i)}\}^t\) supplied from the 3D CNN 251. In addition, the vector synthesis unit 236 obtains the context vector \(\{f_{(k,i)}\}^{t - 1}\) supplied from the 3D CNN 252. In addition, the vector synthesis unit 236 obtains the context vector \(\{f_{(k,i)}\}^{t + 1}\) supplied from the 3D CNN 253. In addition, the vector synthesis unit 236 obtains the context vector \(\{f_{(k + 1,i)}\}^{t - 1}\) supplied from the 3D CNN 254. In addition, the vector synthesis unit 236 obtains the codebook C (importance coefficient vector \(\alpha^{(c_i)^*}\)) supplied from the codebook storage unit 235.

[0224] The vector synthesis unit 236 combines the obtained plurality of context vectors by using the obtained importance coefficient vector, generates a synthesized vector \((u_i)^{(c_i)^*}\), and supplies the synthesized vector \((u_i)^{(c_i)^*}\) to the MLP 237. That is, the vector synthesis unit 236 combines a plurality of context vectors corresponding to the processing object node of the 3D data having a tree structure by using the importance coefficient vector for controlling the contribution degree of the context vector to the occupancy state prediction of the child node of the processing object node by utilizing the intra-frame correlation and the inter-frame correlation to generate a synthesized vector.

[0225] For example, the vector synthesis unit 236 can derive a weighted sum by weighting each of the plurality of context vectors with each element of the importance coefficient vector, and use the weighted sum as the synthesized vector. For example, as Figure 11 shown, when four context vectors ({f_(k,i)}^t, {f_(k,i)}^(t-1), {f_(k,i)}^(t+1), and {f_(k+1,i)}^(t-1)) are obtained from the context vector derivation unit 233, the vector synthesis unit 236 obtains a synthesized vector by deriving a weighted sum of the context vectors using the importance coefficient vector α_i)^{(c_i)^*} = [{α_(i,0)}^{(c_i)^*}, {α_(i,1)}^{(c_i)^*}, {α_(i,2)}^{(c_i)^*}, {α_(i,3)}^{(c_i)^*}] ∈ [0,1]^4 (where normalization is performed to obtain Σ_j[{α_(i,j)}^{(c_i)^*}] = 1). In this way, the dimension of the importance coefficient vector can be four. That is, the vector synthesis unit 236 can generate the synthesized vector (u_i)^{(c_i)^*} by the following expression (18).

[0226] [Equation 17]

[0227]

[0228] In this way, the contribution of each context vector to the prediction can be explicitly controlled by using the value of the importance coefficient vector (α_i)^{(c_i)^*}. That is, an explicit prediction mode can be controlled in which the feature amount of which frame is emphasized more. Further, at the time of decoding, the importance coefficient vector (α_i)^{(c_i)^*} applied at the time of encoding can be used. Therefore, a decrease in coding efficiency can be suppressed.

[0229] The MLP 237 performs a similar process to the MLP 136 to derive the predicted probability vector \((p_i)^{(c_i)^*}\). That is, the MLP 237 derives the predicted probability vector \((p_i)^{(c_i)^*}\) indicating the probability values of the occupancy states that each child node of the processing target node can take, based on the synthesized vector \((u_i)^{(c_i)^*}\) supplied from the vector synthesis unit 236. That is, the MLP 237 can also be referred to as a predicted probability vector derivation unit. Note that the MLP 237 can be a multi-layer perceptron that uses the synthesized vector \((u_i)^{(c_i)^*}\) supplied from the vector synthesis unit 236 as input to derive the predicted probability vector \((p_i)^{(c_i)^*}\). That is, the MLP 237 can derive the predicted probability vector by using a multi-layer perceptron with the synthesized vector as input. The MLP 237 supplies the derived predicted probability vector \((p_i)^{(c_i)^*}\) to the occupancy state decoding unit 238.

[0230] For example, the MLP 237 can generate the predicted probability vector \((p_i)^{(c_i)^*}\) corresponding to the importance coefficient vector \((α_i)^{(c_i)^*}\) applied during encoding through the following expression (19). Note that in expression (18), MLP' is used to distinguish from Figure 3 the MLP 13 (clearly indicating the difference in neural networks). The input dimension of MLP' is equal to the dimension of the context vector, the output dimension is 256, and the softmax function is used as the activation function of the final layer.

[0231] [Mathematical formula 18]

[0232]

[0233] The occupancy state decoding unit 238 decodes the bitstream by using the prediction probability vector to generate information indicating the occupancy state of the child nodes of the node to be processed. For example, the occupancy state decoding unit 238 obtains the prediction probability vector (p_i)^{(c_i)^*} supplied from the MLP 237. In addition, the occupancy state decoding unit 238 obtains the occupancy state bitstream input to the geometric information decoding device 200. The occupancy state decoding unit 238 performs entropy decoding on the occupancy state bitstream by using the prediction probability vector (p_i)^{(c_i)^*} as an entropy model to generate (recover) information indicating the occupancy state of the child nodes of the node to be processed. The occupancy state decoding unit 238 supplies the generated (recovered) information indicating the occupancy state of the child nodes of the node to be processed to the frame memory 231 for storage. In addition, the occupancy state decoding unit 238 supplies the generated (recovered) information indicating the occupancy state of the child nodes of the node to be processed to the point cloud construction unit 212 as part of the generated (recovered) octree ( Figure 10 ). Note that the occupancy state decoding unit 238 can output (supply to the point cloud construction unit 212 or the frame memory 231) the octree (or octree sequence) after the octree is completed.

[0234] With the above configuration, the geometric information decoding device 200 can control the contribution rate of each context vector to the prediction by using the importance coefficient vector, and can suppress the reduction of the coding efficiency. In addition, since the geometric information decoding device 200 can perform continuous prediction mode control instead of discrete prediction mode selection in a 2D moving image codec or the like, the reduction of the coding efficiency can be suppressed.

[0235] <Flow of decoding process>

[0236] The geometric information decoding device 200 decodes the bitstream of the geometric information as described above by performing a decoding process. An example of the flow of this decoding process will be described with reference to the flowchart of Figure 12 .

[0237] When the decoding process starts, the octree decoding unit 211 performs an octree decoding process on the octree bitstream in step S201. The octree decoding unit 211 performs such a process at each time (each frame) to generate an octree (that is, an octree sequence) at each time (each frame).

[0238] In step S202, the point cloud construction unit 212 constructs a point cloud by using the octree obtained by decoding the bitstream. The point cloud construction unit 212 performs such a process at each time (each frame) to generate a point cloud (that is, a point cloud sequence) at each time (each frame).

[0239] When the processing in step S202 ends, the decoding process ends.

[0240] <Flow of Octree Decoding Process>

[0241] Next, an example of the flow of the octree decoding process executed in step S201 of Figure 13 will be described with reference to the flowchart of Figure 12 the steps S231 to S238.

[0242] When the octree decoding process starts, in step S231, the neighboring voxel setting unit 232 sets neighboring voxels as described above. In step S232, the context vector derivation unit 233 derives context vectors corresponding to each neighboring voxel as described above.

[0243] In step S233, the importance coefficient vector decoding unit 234 decodes the importance coefficient bitstream to generate an index (c_i)^* of the codeword applied at the time of encoding.

[0244] In step S234, the vector synthesis unit 236 performs the processing as described above, and combines the context vectors by using the importance coefficient vector (α_i)^{(c_i)^*} corresponding to the index (c_i)^* to generate a combined vector (u_i)^{(c_i)^*}. For example, the vector synthesis unit 236 combines the context vectors by using a weighted operation of the importance coefficient vector (α_i)^{(c_i)^*}.

[0245] In step S235, the MLP 237 inputs the combined vector (u_i)^{(c_i)^*} to the multi-layer perceptron (MLP') to derive a predicted probability vector (p_i)^{(c_i)^*}.

[0246] In step S236, the occupancy state decoding unit 238 decodes the occupancy state bitstream by using the predicted probability vector (p_i)^{(c_i)^*} as an entropy model to generate information indicating the occupancy state of the child nodes of the processing target node (recovery).

[0247] In step S237, the occupancy state decoding unit 238 determines whether all nodes have been processed, and performs each of the processes of steps S231 to S237 for each node until it is determined that all nodes of the processing target frame of the processing target LoD have been processed. Then, when it is determined in step S237 that the processing has been performed on all nodes, the process proceeds to step S238.

[0248] In step S238, the occupancy state decoding unit 238 determines whether all frames have been processed, and performs each process of steps S231 to S238 for each node of each frame until it is determined that all nodes of all frames of the processing target LoD have been processed. Then, when it is determined in step S238 that the processing for all frames has been performed, the processing proceeds to step S239.

[0249] In step S239, the occupancy state decoding unit 238 determines whether all layers have been processed, and performs each process of steps S231 to S239 for each node of each frame of each LoD until it is determined that all nodes of all frames of all LoD have been processed. Then, when it is determined in step S239 that the processing for all layers has been performed, the processing proceeds to step S240.

[0250] In step S240, the occupancy state decoding unit 238 outputs the octree. When the processing of step S240 ends, the octree decoding process ends, and the processing returns to Figure 12 .

[0251] By performing each of the above-described processes, the geometric information decoding device 200 can control the contribution rate of each context vector to prediction by using the importance coefficient vector, and can suppress a decrease in coding efficiency. In addition, since the geometric information decoding device 200 can perform continuous prediction mode control instead of discrete prediction mode selection in a 2D moving image codec or the like, a decrease in coding efficiency can be suppressed.

[0252] <4. Query Vector>

[0253] <Method 1-2>

[0254] When Method 1 is applied, as shown in the third row from the top of the table in Figure 4 the importance coefficient vector (Method 1-2) can be derived by using the attention calculation of the query vector and the context vector. For example, in the first information processing device and the second information processing device, the vector synthesis unit can generate a synthesis vector by performing scaled dot product attention calculation using a query vector having the same dimension as the dimension of the context vector.

[0255] A query vector q_i (with the same dimension as the context vector) can be introduced, and an importance coefficient vector α_i can be derived through scaled dot - product attention calculation between the query vector q_i and each context vector, as shown in the following expression (20). However, in expression (20), F_i is a matrix with all context vectors in a row (e.g., {f_(k,i)}^t). Thus, the number of rows is equal to the number of context vectors, and the number of columns is equal to the dimension of the context vector. Furthermore, d_k is the dimension of the context vector (= the dimension of the query vector). Since the output of the softmax function is a 1×{number of context vectors} matrix, i.e., a row vector, the output is transposed to be output as a column vector.

[0256] [Equation 19]

[0257]

[0258] By applying such a query vector, the number of context vectors can be made variable while the dimension of the query vector is fixed. That is, the number of reference frames can be changed without changing the dimension of the query vector. For example, it is possible to change the number of reference frames at a certain mid - point in the sequence, such as applying only t - 1 frames to suppress the computational cost in one frame and applying t - 2, t - 1, t, t + 1, and t + 2 frames to prioritize compression performance in other frames. Since such a change can be made without changing the dimension of the query vector, the change can be implemented more easily.

[0259] Note that the context vector can be derived for each layer through a neural network (a common 3DCNN in the LOD). For example, the first information processing device and the second information processing device may further include a context vector derivation unit configured to derive context vectors for each layer through a neural network. That is, the 3D - CNN can be completely common in the time direction. That is, two 3DCNNs of (3DCNN_0)^* and (3DCNN_+1)^* can be used.

[0260] By combining expressions (20) and (13), they can be represented as the following expressions (21) to (23). Q is a 1×d_k matrix. Furthermore, the matrix K in expression (22) does not indicate the depth of the octree.

[0261] [Equation 20]

[0262]

[0263] [Equation 21]

[0264] K = V = F i

[0265] ……(22)

[0266] [Mathematical formula 22]

[0267] u i = Attention(Q, K, V) T

[0268] ……(23)

[0269] <Method 1-2-1>

[0270] Note that, in this case, for example, a query vector can be transmitted instead of the importance coefficient vector. For example, as shown in the fourth line from the top in Figure 4 The query vector can be encoded by vector quantization (Method 1-2-1). For example, the first information processing device may further include a query vector encoding unit configured to encode the query vector. In this case, the query vector encoding unit may perform entropy encoding on the index indicating the query vector. In addition, the second information processing device may further include a query vector decoding unit configured to decode the bitstream to generate the query vector. In this case, the query vector decoding unit may perform entropy decoding on the bitstream to generate the index indicating the query vector.

[0271] <Octree encoding unit>

[0272] Figure 14 The main configuration example of the octree encoding unit 113 in this case is shown. In the example of this drawing, the octree encoding unit 113 includes a codebook storage unit 331 instead of Figure 6 the codebook storage unit 131. In addition, the octree encoding unit 113 includes a context vector derivation unit 334 instead of the context vector derivation unit 134. In addition, the octree encoding unit 113 includes a vector synthesis unit 335 instead of the vector synthesis unit 135. In addition, the octree encoding unit 113 includes a bit size approximation value derivation unit 337 instead of the bit size approximation value derivation unit 137. In addition, the octree encoding unit 113 includes a query vector encoding unit 339 instead of the importance coefficient vector encoding unit 139.

[0273] Similar to the case of the codebook storage unit 131, the codebook storage unit 331 includes a storage medium and stores the codebook C. The codebook storage unit 331 supplies the stored codebook C to the vector synthesis unit 335 and the codeword selection unit 138. However, in this case, the codebook C is a set of candidate query vectors q^c. That is, the codebook storage unit 331 stores and provides the codebook C = {q^0,..., q^(|C|-1)} of query vectors.

[0274] The context vector derivation unit 334 derives context vectors corresponding to each of a plurality of neighboring voxels by using a neural network for each layer, and supplies the context vectors to the vector synthesis unit 335. For example, the context vector derivation unit 334 may include 3DCNNs 351 and 352, which are neural networks for respective layers.

[0275] The 3DCNN 351 is {(3DCNN_0)^*}() corresponding to the LoD to be processed. The 3DCNN 351 receives inputs of neighboring voxels (e.g., {V_(k,i)}^t, {V_(k,i)}^(t - 1), {V_(k,i)}^(t + 1), etc.) of any frame of the LoD to be processed, generates context vectors corresponding to the neighboring voxels, and supplies the context vectors to the vector synthesis unit 135. For example, the 3DCNN 351 generates a context vector {f_(k,i)}^t corresponding to the processing object frame of the LoD to be processed based on the input neighboring voxels {V_(k,i)}^t, and supplies the context vector to the vector synthesis unit 135. Further, the 3DCNN 351 generates a context vector {f_(k,i)}^(t - 1) corresponding to the frame immediately before the processing object frame of the LoD to be processed based on the input neighboring voxels {V_(k,i)}^(t - 1), and supplies the context vector to the vector synthesis unit 135. Further, the 3DCNN 351 generates a context vector {f_(k,i)}^(t + 1) corresponding to the frame immediately after the processing object frame of the LoD to be processed based on the input neighboring voxels {V_(k,i)}^(t + 1), and supplies the context vector to the vector synthesis unit 135.

[0276] The 3DCNN 352 is the {(3DCNN_+1)^*}() corresponding to the LoD one level lower than the LoD of the object to be processed. The 3DCNN 352 receives the input of the neighboring voxels of any frame at the LoD one level lower than the LoD of the object to be processed (e.g., {V_(k+1,i)}^(t-1), {V_(k+1,i)}^(t-2), etc.), generates a context vector corresponding to the neighboring voxels, and supplies the context vector to the vector synthesis unit 135. For example, the 3DCNN 352 generates a context vector {f_(k+1,i)}^(t-1) corresponding to the frame immediately before the object frame at the LoD one level lower than the LoD of the object to be processed based on the input neighboring voxel {V_(k+1,i)}^(t-1), and supplies the context vector to the vector synthesis unit 135. In addition, the 3DCNN 352 generates a context vector {f_(k+1,i)}^(t-2) corresponding to the frame two frames before the object frame at the LoD one level lower than the LoD of the object to be processed based on the input neighboring voxel {V_(k+1,i)}^(t-2), and supplies the context vector to the vector synthesis unit 135.

[0277] That is, since these 3DCNNs are applied jointly in the time direction, the octree encoding unit 113 can make the number of reference frames variable.

[0278] The vector synthesis unit 335 obtains a plurality of context vectors supplied from the context vector derivation unit 334. For example, the vector synthesis unit 335 obtains the context vectors corresponding to the LoD of the object to be processed (e.g., {f_(k,i)}^t, {f_(k,i)}^(t-1), {f_(k,i)}^(t+1), etc.) supplied from the 3DCNN 351. In addition, the vector synthesis unit 335 obtains the context vectors corresponding to the LoD one level lower than the LoD of the object to be processed (e.g., {f_(k+1,i)}^(t-1), {f_(k+1,i)}^(t-2), etc.) supplied from the 3DCNN 352. In addition, the vector synthesis unit 335 reads and obtains the codebook C (query vector (q_i)^c) from the codebook storage unit 331.

[0279] The vector synthesis unit 335 performs the above-mentioned scaled dot product attention calculation by using the obtained query vector (q_i)^c (a query vector having the same dimension as the context vector) to combine the plurality of obtained context vectors to generate a synthetic vector. For example, the vector synthesis unit 335 derives the synthetic vector (u_i)^c as in the following expression (24). Note that Q and K satisfy the following expressions (25) and (26). The vector synthesis unit 335 supplies the generated synthetic vector to the MLP 136.

[0280] [Mathematical formula 23]

[0281]

[0282] [Mathematical formula 24]

[0283]

[0284] [Mathematical formula 25]

[0285] K = V = F i

[0286] ……(26)

[0287] The bit size approximation derivation unit 337 obtains the predicted probability vector p_i supplied from the MLP 136, and estimates the amount of code (total bit size after compression) in the case where the predicted probability vector p_i is applied (derives the bit size approximation R_i(c)). That is to say, the bit size approximation derivation unit 337 can also be referred to as a code amount estimation unit. Note that the bit size approximation derivation unit 337 can estimate the code amount corresponding to each of the multiple candidates of the query vector included in the codebook C. For example, the bit size approximation derivation unit 337 can derive the bit size approximation R_i(c) through the following expression (27).

[0288] R_i(c) = -log2({the value of the element of (p_i)^c corresponding to the true occupancy state}) - log2({the c-th element of the probability table for query vector compression})……(27)

[0289] The bit size approximation derivation unit 337 supplies the bit size approximation R_i(c) corresponding to each candidate of the importance coefficient vector derived in this way and the predicted probability vector (p_i)^c for estimation to the codeword selection unit 138.

[0290] The query vector encoding unit 339 encodes the index (c_i)^* of the query vector supplied from the codeword selection unit 138 to generate its bit stream (also referred to as the query vector bit stream). That is to say, it can also be said that the query vector encoding unit 339 encodes the query vector selected by the codeword selection unit 138. The query vector encoding unit 339 outputs the generated query vector bit stream to the outside of the octree encoding unit 113 (outside the geometric information encoding device 100). This query vector bit stream can be provided to the geometric information decoding device that decodes the geometric information bit stream via any transmission path or any recording medium.

[0291] The other processing unit performs processing similar to that in the case of Figure 6 Through such a configuration, the geometric information encoding device 100 can easily make the number of reference frames variable.

[0292] <Octree encoding processing flow>

[0293] With reference to Figure 15 and Figure 16 The flowchart describes an example of the octree encoding processing flow in this case. In this case, when the octree encoding processing starts, the query vector encoding unit 339 and the occupancy state encoding unit 140 initialize the occupancy state bitstream and the query vector bitstream in Figure 15 step S331 of

[0294] In step S332, the nearby voxel setting unit 132 sets nearby voxels similarly to the case of step S132( Figure 8 ). In step S333, the context vector derivation unit 334 derives the context vector corresponding to each nearby voxel as described above.

[0295] In step S334, as described above, the vector synthesis unit 335 combines the context vectors by using the scaled dot-product attention calculation of the query vector to generate a synthesized vector.

[0296] In step S335, the MLP 136 inputs the synthesized vector to the multi-layer perceptron (MLP') to derive a predicted probability vector. That is, the MLP 136 generates a predicted probability vector corresponding to the processing object candidate of the query vector.

[0297] In step S336, the bit size approximation derivation unit 337 derives the bit size approximation by using the above expression (27). That is, the bit size approximation derivation unit 337 derives the bit size approximation corresponding to the candidate of the processing object of the query vector.

[0298] In step S337, the codeword selection unit 138 determines whether processing has been performed on all codewords included in the codebook, and performs each process of steps S334 to S337 for each codeword until it is determined that processing has been performed on all codewords. Then, when it is determined in step S337 that processing has been performed on all codewords, the process proceeds to Figure 16 .

[0299] In Figure 16 step S351 of

[0300] the codeword selection unit 138 selects the codeword (candidate of the importance coefficient vector) that minimizes the bit size approximation derived in step S336.

[0301] In step S353, the occupancy status encoding unit 140 performs entropy encoding on the information indicating the occupancy status of the child nodes of the processing target node by using the prediction probability vector corresponding to the selected codeword as an entropy model, and adds the encoded data to the occupancy status bitstream.

[0302] In step S354, the occupancy status encoding unit 140 determines whether all nodes have been processed, and performs each process of Figure 15 steps S332 to S337 for each node and Figure 16 each process of steps S351 to S354 until it is determined that all nodes of the processing target frame of the processing target LoD have been processed. Then, when it is determined in Figure 16 step S354 that the processing has been performed on all nodes, the process proceeds to step S355.

[0303] In step S355, the occupancy status encoding unit 140 determines whether all frames have been processed, and performs each process of Figure 15 steps S332 to S337 for each node of each frame and Figure 16 each process of steps S351 to S355 until it is determined that all nodes of all frames of the processing target LoD have been processed. Then, when it is determined in Figure 16 step S355 that the processing has been performed on all frames, the process proceeds to step S356.

[0304] In step S356, the occupancy status encoding unit 140 determines whether all layers have been processed, and performs each process of Figure 15 steps S332 to S337 for each node of each frame of each LoD and Figure 16 each process of steps S351 to S356 until it is determined that all nodes of all frames of all LoDs have been processed. Then, when it is determined in Figure 16 step S356 that the processing has been performed on all layers, the process proceeds to step S357.

[0305] In step S357, the query vector encoding unit 339 outputs a query vector bitstream. In addition, the occupancy status encoding unit 140 outputs an occupancy status bitstream.

[0306] When the processing of step S357 ends, the octree encoding process ends, and the process returns to Figure 7 .

[0307] By performing each process in this way, the geometric information encoding device 100 can easily make the number of reference frames variable.

[0308] <Octree Decoding Unit>

[0309] Figure 17 FIG. shows a main configuration example of the octree decoding unit 211 in this case. In the case of the example in this figure, the octree decoding unit 211 includes a context vector derivation unit 433 instead of Figure 11 the context vector derivation unit 233 in. In addition, the octree decoding unit 211 includes a query vector decoding unit 434 instead of the importance coefficient vector decoding unit 234. In addition, the octree decoding unit 211 includes a codebook storage unit 435 instead of the codebook storage unit 235. In addition, the octree decoding unit 211 includes a vector synthesis unit 436 instead of the vector synthesis unit 236.

[0310] The context vector derivation unit 433 derives a context vector corresponding to each of a plurality of neighboring voxels by using a neural network for each layer, and supplies the context vector to the vector synthesis unit 436. For example, the context vector derivation unit 433 may include 3DCNNs 451 and 452, which are neural networks for respective layers.

[0311] The 3DCNN 451 is {(3DCNN_0)^*}() corresponding to the LoD of the object to be processed. The 3DCNN 451 receives inputs of neighboring voxels (e.g., {V_(k,i)}^t, {V_(k,i)}^(t - 1), {V_(k,i)}^(t + 1), etc.) of any frame of the LoD of the object to be processed, generates a context vector corresponding to the neighboring voxels, and supplies the context vector to the vector synthesis unit 436. For example, the 3DCNN 451 generates a context vector {f_(k,i)}^t corresponding to the processing object frame of the LoD of the object to be processed based on the input neighboring voxel {V_(k,i)}^t, and supplies the context vector to the vector synthesis unit 436. In addition, the 3DCNN 451 generates a context vector {f_(k,i)}^(t - 1) corresponding to the frame immediately before the processing object frame of the LoD of the object to be processed based on the input neighboring voxel {V_(k,i)}^(t - 1), and supplies the context vector to the vector synthesis unit 436. In addition, the 3DCNN 451 generates a context vector {f_(k,i)}^(t + 1) corresponding to the frame immediately after the processing object frame of the LoD of the object to be processed based on the input neighboring voxel {V_(k,i)}^(t + 1), and supplies the context vector to the vector synthesis unit 436.

[0312] The 3DCNN 452 is the {(3DCNN_+1)^*}() corresponding to the LoD one level lower than the LoD of the object to be processed. The 3DCNN 452 receives the input of the neighboring voxels (e.g., {V_(k+1,i)}^(t-1), {V_(k+1,i)}^(t-2), etc.) of any frame at the LoD one level lower than the LoD of the object to be processed, generates a context vector corresponding to the neighboring voxels, and supplies the context vector to the vector synthesis unit 436. For example, the 3DCNN 452 generates a context vector {f_(k+1,i)}^(t-1) corresponding to the frame immediately before the frame to be processed at the LoD one level lower than the LoD of the object to be processed based on the input neighboring voxel {V_(k+1,i)}^(t-1), and supplies the context vector to the vector synthesis unit 436. In addition, the 3DCNN 452 generates a context vector {f_(k+1,i)}^(t-2) corresponding to the frame two frames before the frame to be processed at the LoD one level lower than the LoD of the object to be processed based on the input neighboring voxel {V_(k+1,i)}^(t-2), and supplies the context vector to the vector synthesis unit 436.

[0313] That is, since these 3DCNNs are applied jointly in the time direction, the octree decoding unit 211 can make the number of reference frames variable.

[0314] The query vector decoding unit 434 acquires the query vector bitstream supplied from outside the octree decoding unit 211 (the geometric information decoding device 200), and performs entropy decoding on the query vector bitstream to generate (restore) the index (c_i)^* corresponding to the query vector applied at the time of encoding. That is, it can also be said that the query vector decoding unit 434 decodes the query vector bitstream to generate a query vector. The query vector decoding unit 434 supplies the generated index (c_i)^* to the codebook storage unit 435.

[0315] Similar to the case of the codebook storage unit 131, the codebook storage unit 435 includes a storage medium and stores the codebook C. The codebook storage unit 331 supplies the stored codebook C to the vector synthesis unit 335 and the codeword selection unit 138. However, in this case, the codebook C is a set of candidate query vectors q^c. That is, the codebook storage unit 435 stores the codebook C = {q^0,..., q^(|C|-1)} of the query vectors, and provides the query vector q^{(c_i)^*} corresponding to the index (c_i)^* supplied from the query vector decoding unit 434 to the vector synthesis unit 436.

[0316] The vector synthesis unit 436 obtains a plurality of context vectors supplied from the context vector derivation unit 433. For example, the vector synthesis unit 436 obtains context vectors (e.g., {f_(k,i)}^t, {f_(k,i)}^(t - 1), {f_(k,i)}^(t + 1), etc.) supplied from the 3D CNN 451 and corresponding to the LoD of the processing object. In addition, the vector synthesis unit 436 obtains context vectors (e.g., {f_(k + 1,i)}^(t - 1), {f_(k + 1,i)}^(t - 2), etc.) supplied from the 3D CNN 452 and corresponding to the LoD one level lower than the LoD of the processing object. In addition, the vector synthesis unit 436 reads and obtains the codebook C (query vector (q_i)^c) from the codebook storage unit 435.

[0317] The vector synthesis unit 436 performs the above-mentioned scaled dot product attention calculation by using the obtained query vector q^{(c_i)^*} to combine the plurality of obtained context vectors to generate a synthesized vector. For example, the vector synthesis unit 436 derives the synthesized vector (u_i)^{(c_i)^*} as in the above expression (24). Note that Q and K satisfy the above expressions (25) and (26). The vector synthesis unit 436 supplies the generated synthesized vector to the MLP 237.

[0318] Other processing units perform processing similar to that in Figure 11 the case of. With such a configuration, the geometric information decoding device 200 can easily make the number of reference frames variable.

[0319] <Flow of octree decoding process>

[0320] Reference will be made to Figure 18 the flowchart in to describe an example of the flow of the octree decoding process in this case. In this case, when the octree encoding process starts, the nearby voxel setting unit 232 sets nearby voxels in step S431 in a similar manner to the case of step S231 ( Figure 13 ). In step S432, the context vector derivation unit 433 derives context vectors corresponding to each nearby voxel by using a neural network for each layer as described above.

[0321] In step S433, the query vector decoding unit 434 decodes the query vector bitstream as described above to generate (recover) the index (c_i)^* corresponding to the codeword (query vector) applied during encoding.

[0322] In step S434, the vector synthesis unit 436 performs the scaled dot - product attention calculation as described above by using the query vector q^{(c_i)^*} corresponding to the index (c_i)^* to combine the context vectors to generate the composite vector (u_i)^{(c_i)^*}.

[0323] In step S435, similar to the case of step S235( Figure 13 ), the MLP 237 inputs the composite vector (u_i)^{(c_i)^*} into the multi - layer perceptron (MLP') to derive the predicted probability vector (p_i)^{(c_i)^*}.

[0324] In step S436, similar to the case of step S236( Figure 13 ), the occupancy state decoding unit 238 decodes the occupancy state bitstream by using the predicted probability vector (p_i)^{(c_i)^*} as an entropy model to generate information indicating the occupancy state of the child nodes of the processing object node (recovery).

[0325] In step S437, similar to the case of step S237( Figure 13 ), the occupancy state decoding unit 238 determines whether all nodes have been processed, and performs each process of steps S431 to S437 for each node until it is determined that all nodes of the processing object frame of the processing object LoD have been processed. Then, when it is determined in step S437 that the processing has been performed for all nodes, the process proceeds to step S438.

[0326] In step S438, similar to the case of step S238( Figure 13 ), the occupancy state decoding unit 238 determines whether all frames have been processed, and performs each process of steps S431 to S438 for each node of each frame until it is determined that all nodes of all frames of the processing object LoD have been processed. Then, when it is determined in step S438 that the processing has been performed for all frames, the process proceeds to step S439.

[0327] In step S439, similar to the case of step S239( Figure 13 ), the occupancy state decoding unit 238 determines whether all layers have been processed, and performs each process of steps S431 to S439 for each node of each frame of each LoD until it is determined that all nodes of all frames of all LoDs have been processed. Then, when it is determined in step S439 that the processing has been performed for all layers, the process proceeds to step S440.

[0328] In step S440, similar to the case of step S240( Figure 13 ) Similarly, in the case of (), the occupancy state decoding unit 238 outputs an octree. When the process of step S440 ends, the octree decoding process ends, and the process returns to Figure 12 .

[0329] By performing each process in this way, the geometric information decoding device 200 can easily make the number of reference frames variable.

[0330] <Method 1-2-1-1>

[0331] For example, as shown in the fifth row from the top of the table of Figure 4 , context vectors can be used to derive the parameters of the entropy model of the query vector. In this way, by dynamically changing the entropy model of the query vector according to the set of context vectors, a decrease in coding efficiency can be suppressed.

[0332] For example, the first information processing device may further include an entropy model derivation unit configured to derive the entropy model of the query vector, and the query vector encoding unit may apply the derived entropy model to perform entropy encoding on the index. In addition, for example, the second information processing device may further include an entropy model derivation unit configured to derive the entropy model of the query vector, and the query vector decoding unit may apply the derived entropy model to perform entropy decoding on the bitstream to generate an index.

[0333] <Octree Encoding Unit>

[0334] Figure 19 Shows a main configuration example of the octree encoding unit 113 in this case. In the example of this drawing, the octree encoding unit 113 includes a probability vector generation unit 511 in addition to the configuration shown in Figure 14 . Further, the octree encoding unit 113 includes a vector synthesis unit 535 instead of the vector synthesis unit 335 ( Figure 14 ). Further, the octree encoding unit 113 includes a query vector encoding unit 539 instead of the query vector encoding unit 339 ( Figure 14 ).

[0335] Compared with the vector synthesis unit 335 ( Figure 14 ) is similar. The vector synthesis unit 535 combines multiple context vectors by using a query vector to generate a synthesized vector, and supplies the synthesized vector to the MLP 136. In addition, the vector synthesis unit 535 generates a matrix F_i in which a set of context vectors is arranged (the number of rows is equal to the number of context vectors, and the number of columns is equal to the dimension of the context vectors), and supplies the matrix F_i to the probability vector generation unit 511. The probability vector generation unit 511 derives a probability vector (p_i)^(entropy_model) based on the matrix F_i in which the set of context vectors is arranged, and supplies the probability vector (p_i)^(entropy_model) as an entropy model to the bit size approximation derivation unit 337 and the query vector encoding unit 539. That is to say, the probability vector generation unit 511 can also be referred to as an entropy model derivation unit, which derives an entropy model of the query vector based on a group of context vectors. The query vector encoding unit 539 applies the probability vector (p_i)^(entropy_model) as an entropy model, and performs entropy encoding on the index (c_i)^* to generate a query vector bit stream. Note that in this case, the bit size approximation derivation unit 337 can estimate the amount of code corresponding to each of a plurality of candidates of the query vector included in the codebook C by using the probability vector supplied from the probability vector generation unit 511.

[0336] By doing so, the geometric information encoding device 100 can dynamically change the entropy model of the query vector according to the set of context vectors, and can suppress a decrease in encoding efficiency.

[0337] Note that the method by which the probability vector generation unit 511 derives the entropy model of the query vector can be any method. For example, the probability vector generation unit 511 can derive the entropy model by calculating the average value of the context vectors and inputting the average value to a neural network. In this case, as Figure 20 shown, the probability vector generation unit 511 includes an element average value calculation unit 551 and an MLP 552. The element average value calculation unit 551 obtains the matrix F_i in which the set of context vectors is arranged from the vector synthesis unit 535 (the number of rows is equal to the number of context vectors, and the number of columns is equal to the dimension of the context vectors). The element average value calculation unit 551 calculates the average value of all elements for the column vectors of each column of the matrix F_i. The element average value calculation unit 551 sets a vector having the obtained average values of all columns as new elements to (f_i)^(aggregated). Note that the dimension of (f_i)^(aggregated) is equal to the dimension of the context vectors. The element average value calculation unit 551 supplies (f_i)^(aggregated) to the MLP 552.

[0338] The MLP 552 is a multi-layer perceptron that receives \((f_i)^{(aggregated)}\) supplied from the element average calculation unit 551 as input, derives the prediction probability vector \((p_i)^{(entropy_model)}\) of the query vector, and supplies the prediction probability vector \((p_i)^{(entropy_model)}\) to the query vector encoding unit 539. That is, the MLP 552 derives the prediction probability vector \((p_i)^{(entropy_model)}\) of the query vector based on \((f_i)^{(aggregated)}\). That is, the MLP 552 can also be referred to as a query vector prediction probability vector derivation unit.

[0339] For example, the MLP 552 can calculate the probability vector \((p_i)^{(entropy_model)}\) in the \(|C|\) dimension by inputting \((f_i)^{(aggregated)}\) to a newly prepared multi-layer perceptron (MLP″) as shown in the following expression (28). Note that in expression (28), MLP″ is used to distinguish from Figure 3 the MLP 13 and MLP′ of (clearly indicating different neural networks). The input dimension of MLP″ is equal to the dimension of the context vector, the output dimension is 256, and the softmax function is used as the activation function of the final layer.

[0340] [Mathematical formula 26]

[0341]

[0342] The query vector encoding unit 539 encodes the index \((c_i)^*\) by using the probability vector \((p_i)^{(entropy_model)}\) as an entropy model to generate a query vector bitstream. That is, it can also be said that the query vector encoding unit 339 encodes the query vector selected by the codeword selection unit 138. The query vector encoding unit 539 outputs the generated query vector bitstream to the outside of the octree encoding unit 113 (outside the geometric information encoding device 100). This query vector bitstream can be provided to a geometric information decoding device that decodes the geometric information bitstream via any transmission path or any recording medium.

[0343] Other processing units perform processing similar to that in Figure 14 the case of. By doing so, the geometric information encoding device 100 can dynamically change the entropy model of the query vector according to the context vector set and can suppress the reduction of the encoding efficiency.

[0344] <Process flow of octree encoding>

[0345] Will be referred to Figure 21 and Figure 22 The flowchart describes an example of the process of octree encoding processing in this case. In this case, when the octree encoding processing starts, the query vector encoding unit 539 and the occupancy state encoding unit 140 initialize the occupancy state bitstream and the query vector bitstream in Figure 21 step S531.

[0346] In step S532, the nearby voxel setting unit 132 sets nearby voxels in a similar manner to the case of step S332 ( Figure 15 ). In step S533, similar to the case of step S333 ( Figure 15 ), the context vector derivation unit 334 derives context vectors corresponding to each nearby voxel.

[0347] In step S534, the vector synthesis unit 535 generates a matrix F_i arranged with a set of context vectors. The probability vector generation unit 511 performs probability vector generation processing using the matrix F_i to generate a probability vector.

[0348] In step S535, similar to the case of step S334 ( Figure 15 ), the vector synthesis unit 535 combines context vectors by using scaled dot-product attention calculation of the query vector to generate a synthesized vector.

[0349] In step S536, similar to the case of step S335 ( Figure 15 ), the MLP 136 inputs the synthesized vector generated in step S535 to a multi-layer perceptron (MLP') to derive a predicted probability vector. That is, the MLP 136 generates a predicted probability vector corresponding to the processing object candidate of the query vector.

[0350] In step S537, similar to the case of step S336 ( Figure 15 ), the bit size approximation derivation unit 337 derives a bit size approximation by using the above expression (27). That is, the bit size approximation derivation unit 337 derives a bit size approximation corresponding to the candidate of the processing object of the query vector. [[ID=********]] [[ID=********]]

[0351] In step S538, similar to the case of step S337 ( Figure 15 ), the codeword selection unit 138 determines whether processing has been performed on all codewords included in the codebook, and performs each process of steps S535 to S538 for each codeword until it is determined that processing has been performed on all codewords. Then, when it is determined in step S538 that processing has been performed on all codewords, the process proceeds to Figure 22 .

[0352] In Figure 22 In step S551, similar to the case of step S351 ( Figure 16 ), the codeword selection unit 138 selects a codeword (candidate of the importance coefficient vector) that minimizes the bit size approximation value derived in step S537 ( Figure 21 ).

[0353] In step S552, the query vector encoding unit 539 encodes the index of the selected codeword (that is, the index of the query vector) by using the probability vector generated in step S534 of Figure 21 , and adds the encoded data to the query vector bitstream.

[0354] In step S553, similar to the case of step S353 ( Figure 16 ), the occupancy state encoding unit 140 performs entropy encoding on the information indicating the occupancy state of the child nodes of the processing target node by using the prediction probability vector corresponding to the selected codeword as an entropy model, and adds the encoded data to the occupancy state bitstream.

[0355] In step S554, similar to the case of step S354 ( Figure 16 ), the occupancy state encoding unit 140 determines whether all nodes have been processed, and performs each process of steps S532 to S538 of Figure 21 and each process of steps S551 to S554 of Figure 22 for each node until it is determined that all nodes of the processing target frame of the processing target LoD have been processed. Then, when it is determined in step S554 of Figure 22 that the processing has been performed on all nodes, the process proceeds to step S555.

[0356] In step S555, similar to the case of step S355 ( Figure 16 ), the occupancy state encoding unit 140 determines whether all frames have been processed, and performs each process of steps S532 to S538 of Figure 21 and each process of steps S551 to S555 of Figure 22 for each node of each frame until it is determined that all nodes of all frames of the processing target LoD have been processed. Then, when it is determined in step S555 of Figure 22 that the processing has been performed on all frames, the process proceeds to step S556.

[0357] In step S556, similar to the case of step S356 ( Figure 16 ), the occupancy state encoding unit 140 determines whether all layers have been processed, and performs each process of steps S532 to S538 of Figure 21 for each node of each frame of each LoD and Figure 22 each process of steps S551 to S556 until it is determined that all nodes of all frames of all LoD have been processed. Then, when it is determined in Figure 22 step S556 that processing has been performed for all layers, the processing proceeds to step S557.

[0358] In step S557, similar to the case of step S357 ( Figure 16 ), the query vector encoding unit 339 outputs a query vector bitstream. In addition, similar to the case of step S357 ( Figure 16 ), the occupancy state encoding unit 140 outputs an occupancy state bitstream.

[0359] When the processing of step S557 ends, the octree encoding process ends, and the processing returns to Figure 7 .

[0360] <Flow of Probability Vector Generation Process>

[0361] Next, an example of the flow of the probability vector generation process executed in step S552 of Figure 23 will be described with reference to the flowchart of Figure 22 . When the probability vector generation process starts, in step S571, the element average value calculation unit 551 calculates the average value of all elements for each column vector of the matrix F_i in which the context vector set is arranged. In step S572, the MLP 552 inputs the vector (f_i)^(aggregated) having the average value of all columns as an element to the multi-layer perceptron (MLP″) to derive the predicted probability vector (p_i)^(entropy_model) of the query vector. When the processing of step S572 ends, the probability vector generation process ends, and the processing returns to Figure 22 .

[0362] By performing each process in this way, the geometric information encoding device 100 can dynamically change the entropy model of the query vector according to the context vector set, and can suppress a decrease in encoding efficiency.

[0363] <Octree Decoding Unit>

[0364] Figure 24 shows a main configuration example of the octree decoding unit 211 in this case. In the example of this drawing, the octree decoding unit 211 includes a probability vector generation unit 611 in addition to the configuration shown in Figure 17 . In addition, the octree decoding unit 211 includes a vector synthesis unit 636 instead of the vector synthesis unit 436 ( Figure 17 )。In addition, the octree decoding unit 211 includes a query vector decoding unit 634 instead of the query vector decoding unit 434( ​ )。

[0365] Similar to the case of the vector synthesis unit 436( ​ ), the vector synthesis unit 636 combines a plurality of context vectors by using query vectors to generate a synthesized vector, and supplies the synthesized vector to the MLP 136. In addition, the vector synthesis unit 636 generates a matrix F_i in which a set of context vectors is arranged (the number of rows is equal to the number of context vectors, and the number of columns is equal to the dimension of the context vectors), and supplies the matrix F_i to the probability vector generation unit 611.

[0366] The probability vector generation unit 611 performs a process similar to that of the probability vector generation unit 511. That is, the probability vector generation unit 611 derives a probability vector (p_i)^(entropy_model) based on the matrix F_i in which a set of context vectors is arranged, and supplies the probability vector (p_i)^(entropy_model) as an entropy model to the query vector decoding unit 634. That is, the probability vector generation unit 611 may also be referred to as an entropy model derivation unit that derives an entropy model of a query vector based on a set of context vectors.

[0367] The query vector decoding unit 634 applies the probability vector (p_i)^(entropy_model) as an entropy model, and decodes the query vector bitstream to generate (recover) an index (c_i)^*.

[0368] By doing so, the geometric information decoding device 200 can dynamically change the entropy model of the query vector according to the set of context vectors, and can suppress a decrease in coding efficiency.

[0369] Note that the method for the probability vector generation unit 611 to derive the entropy model of the query vector can be any method. For example, the probability vector generation unit 611 can derive an entropy model by calculating the average value of the context vectors and inputting the average value to a neural network. In this case, the probability vector generation unit 611 can have, for example, a configuration similar to that of ​ the example shown, and perform a similar process.

[0370] By doing so, the geometric information decoding device 200 can dynamically change the entropy model of the query vector according to the set of context vectors, and can suppress a decrease in coding efficiency.

[0371] <Flow of Octree Decoding Process>

[0372] Reference will be made to ​ The flowchart in [ ] shows an example of the process of octree decoding processing in this case. In this case, when the octree decoding processing starts, the neighboring voxel setting unit 232 sets neighboring voxels in step S631 similarly to the case of step S431( ​ )). In step S632, similarly to the case of step S432( ​ ), the context vector derivation unit 433 derives context vectors corresponding to each neighboring voxel by using a neural network for each layer.

[0373] In step S633, the probability vector generation unit 611 performs probability vector generation processing to generate a probability vector. The probability vector generation processing in this case can have arbitrary content, and for example, it can be executed in a process similar to that of ​ the example.

[0374] In step S634, the query vector decoding unit 634 decodes the query vector bitstream by using the probability vector to generate (recover) an index (c_i)^* corresponding to the codeword (query vector) applied at the time of encoding.

[0375] In step S635, similarly to the case of step S434( ​ ), the vector synthesis unit 636 performs the scaled dot product attention calculation as described above by using the query vector q^{(c_i)^*} corresponding to the index (c_i)^* to combine the context vectors to generate a synthesized vector (u_i)^{(c_i)^*}. In addition, the vector synthesis unit 636 generates a matrix F_i in which the context vector set is arranged.

[0376] In step S636, similarly to the case of step S435( ​ ), the MLP 237 inputs the synthesized vector (u_i)^{(c_i)^*} to a multi-layer perceptron (MLP') to derive a predicted probability vector (p_i)^{(c_i)^*}.

[0377] In step S637, similarly to the case of step S436( ​ ), the occupancy state decoding unit 238 decodes the occupancy state bitstream by using the predicted probability vector (p_i)^{(c_i)^*} as an entropy model to generate (recover) information indicating the occupancy state of the child nodes of the processing target node.

[0378] In step S638, similarly to the case of step S437( ​ ) Similarly, the occupancy state decoding unit 238 determines whether all nodes have been processed, and performs each process of steps S631 to S638 for each node until it is determined that all nodes of the processing target frame of the processing target LoD have been processed. Then, when it is determined in step S638 that the processing has been performed on all nodes, the process proceeds to step S639.

[0379] In step S639, similar to the case of step S438 ( ​ ) the occupancy state decoding unit 238 determines whether all frames have been processed, and performs each process of steps S631 to S639 for each node of each frame until it is determined that all nodes of all frames of the processing target LoD have been processed. Then, when it is determined in step S639 that the processing has been performed on all frames, the process proceeds to step S640.

[0380] In step S640, similar to the case of step S439 ( ​ ) the occupancy state decoding unit 238 determines whether all layers have been processed, and performs each process of steps S631 to S640 for each node of each frame of each LoD until it is determined that all nodes of all frames of all LoDs have been processed. Then, when it is determined in step S640 that the processing has been performed on all layers, the process proceeds to step S641.

[0381] In step S641, similar to the case of step S440 ( ​ ) the occupancy state decoding unit 238 outputs the octree. When the process of step S641 ends, the octree decoding process ends, and the process returns to ​ .

[0382] By performing each process in this way, the geometric information decoding device 200 can dynamically change the entropy model of the query vector according to the context vector set, and can suppress a decrease in coding efficiency.

[0383] <Method 1-2-2>

[0384] For example, as shown in the sixth row from the top of the table of ​ , entropy encoding can be performed on each element of the query vector. For example, in the first information processing device, the query vector encoding unit can perform entropy encoding on each element of the query vector. In addition, in the second information processing device, the query vector decoding unit can perform entropy decoding on the bitstream to generate each element of the query vector.

[0385] For example, a probability table with the number of prepared pieces being {the dimension of the codeword} is used instead of a probability table with the number of elements |C|. Each probability table is used to perform entropy coding on the quantization values of each dimension of the codeword. For example, in the octree coding unit 113 in the example of ​ the query vector coding unit 339 has a probability table.

[0386] In this case, the estimated bit size of the query vector -log2({the c-th element of the probability table for query vector compression}) is an estimated value of the bit size after entropy coding each element of the query vector. As described with reference to ​ the codeword selection unit 138 selects the index (c_i)^* of the codeword that minimizes the bit size and supplies this index (c_i)^* to the query vector coding unit 339. Instead of entropy coding the index (c_i)^*, the query vector coding unit 339 quantizes the elements of each dimension of the corresponding codeword q^{(c_i)^*}, performs entropy coding on this element using the corresponding probability table, and writes this element into the bitstream. In the octree decoding unit 211 in the example of ​ the query vector decoding unit 434 also has a probability table. The query vector decoding unit 434 performs entropy decoding on the quantization value from the bitstream using the probability table and inverse quantizes this value.

[0387] <Method 1-2-3>

[0388] For example, as shown in the seventh row from the top of the table in ​ multi-head attention (multi-head attention calculation) can be applied to the attention calculation. In the multi-head attention calculation, the scaled dot-product attention calculation is performed multiple times, and the results are concatenated in the column direction and linearly transformed by the weight matrix W^O. As described above, by applying the multi-head attention calculation, the scaled dot-product attention calculation can be performed multiple times, so performance improvement can be expected compared to the case of applying a single scaled dot-product attention calculation. For example, in the first information processing device and the second information processing device, the vector synthesis unit can generate a synthesized vector by performing multi-head attention calculation using the query vector. In this case, for example, in the octree coding unit 113 in the example of ​ the vector synthesis unit 335 can generate a synthesized vector by performing multi-head attention calculation instead of performing a single scaled dot-product attention calculation. Further, in the octree decoding unit 211 in the example of ​ the vector synthesis unit 436 can generate a synthesized vector by performing multi-head attention calculation instead of performing a single scaled dot-product attention calculation.

[0389] <Method 1-2-4>

[0390] For example, as from ​ As shown in the eighth row from the top of the table in ​ shown, a macroblock can be provided in the vicinity area 710 with a predetermined size, and the query vector can be shared by the nodes in the macroblock. For example, in the first information processing device and the second information processing device, the vector synthesis unit can generate a synthesis vector by using a common query vector for each predetermined area (macroblock). In this case, for example, in ​ the octree encoding unit 113 in the example of ​ the vector synthesis unit 335 can generate a synthesis vector by using a common query vector for each macroblock. In addition, in

[0391] <Division of Vicinity Area>

[0392] <Methods 1 - 3>

[0393] For example, as shown in the ninth row from the top of the table in ​ the vicinity voxels can be divided in the spatial direction. For example, as ​ shown, the vicinity area 720 can be divided into sub-vicinity areas 721 to 724, and the context vectors can be derived individually. In addition, the lower-level vicinity area 730 can be divided into sub-vicinity areas 731 to 734, and the context vectors can be derived individually. For example, the context vectors can correspond to the occupancy states of the sub-vicinity areas formed in the vicinity area, and the vector synthesis unit can combine the context vectors of the respective sub-vicinity areas in the first information processing device and the second information processing device.

[0394] In this case, for example, in ​ the octree encoding unit 113 in the example of ​ the context vector derivation unit 134 can derive the context vectors of each sub-vicinity area, and the vector synthesis unit 135 can combine the context vectors of the respective sub-vicinity areas to generate a synthesis vector. In addition, in

[0395] In addition, in ​ In the octree coding unit 113 of the example, the context vector derivation unit 334 can derive the context vectors of each sub-nearby region, and the vector synthesis unit 335 can combine the context vectors of the respective sub-nearby regions to generate a synthesized vector. In addition, in ​ In the octree decoding unit 211 of the example, the context vector derivation unit 433 can derive the context vectors of each sub-nearby region, and the vector synthesis unit 436 can combine the context vectors of the respective sub-nearby regions to generate a synthesized vector.

[0396] In addition, in ​ In the octree coding unit 113 of the example, the context vector derivation unit 334 can derive the context vectors of each sub-nearby region, and the vector synthesis unit 535 can combine the context vectors of the respective sub-nearby regions to generate a synthesized vector. In addition, in ​ In the octree decoding unit 211 of the example, the context vector derivation unit 433 can derive the context vectors of each sub-nearby region, and the vector synthesis unit 636 can combine the context vectors of the respective sub-nearby regions to generate a synthesized vector.

[0397] By doing so, the correlation in each direction can be used. In a scene with motion, a time / space direction such as "which region to emphasize at which time" can be selected, and an improvement in coding efficiency is expected.

[0398] <Addition of Metadata>

[0399] <Methods 1-4>

[0400] For example, as shown at the bottom of the table in ​ metadata such as coordinate values, octree depth, and time can be added to the context vector. For example, the center coordinate value of the nearby voxels corresponding to the context vector, the depth k of the octree, and the time t can be concatenated to the context vector to obtain a new context vector.

[0401] In this case, for example, in ​ the octree coding unit 113 in the example, the context vector derivation unit 134 can add metadata to the context vector. In addition, in ​ the octree decoding unit 211 of the example, the context vector derivation unit 233 can add metadata to the context vector. In addition, in ​ the octree coding unit 113 in the example, the context vector derivation unit 334 can add metadata to the context vector. In addition, in ​ the octree decoding unit 211 of the example, the context vector derivation unit 433 can add metadata to the context vector. This similarly applies to ​ and ​ in the case of an example of

[0402] <7. Supplementary Explanation>

[0403] <Computer>

[0404] The above series of processes can be executed by hardware or software. When a series of processes are executed by software, a program configuring the software is installed in a computer. Here, examples of the computer include, for example, a computer built in dedicated hardware, a general-purpose personal computer capable of executing various functions by installing various programs, and the like.

[0405] ​ is a block diagram showing a configuration example of the hardware of a computer that executes the above series of processes according to a program.

[0406] In ​ in the computer 900 shown, a central processing unit (CPU) 901, a read-only memory (ROM) 902, and a random access memory (RAM) 903 are interconnected via a bus 904.

[0407] The bus 904 is also connected to an input / output interface 910. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0408] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like. The output unit 912 includes, for example, a display, a speaker, an output terminal, and the like. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, and the like. The communication unit 914 includes, for example, a network interface. The drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0409] In the computer configured as described above, for example, a program recorded in the storage unit 913 is loaded into the RAM 903 via the input / output interface 910 and the bus 904 by the CPU 901 and executed to perform the above series of processes. For example, the RAM 903 also appropriately stores data required for the CPU 901 to execute various processes.

[0410] A program executed by a computer can be applied by being recorded on, for example, a removable medium 921 as a packaged medium. In this case, by attaching the removable medium 921 to the drive 915, the program can be installed in the storage unit 913 via the input / output interface 910.

[0411] In addition, the program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0412] In addition, the program can be pre-installed in the ROM 902 and the storage unit 913.

[0413] <Object of Application of the Present Technology>

[0414] The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0415] In addition, for example, the present technology can also be implemented as a partial configuration of the following devices, for example, as a processor (e.g., a video processor) such as a system large-scale integration (LSI), a module (e.g., a video module) using multiple processors, etc., a unit (e.g., a video unit) using multiple modules, etc., or a set (e.g., a video set) obtained by further adding other functions to the unit.

[0416] In addition, for example, the present technology can also be applied to a network system including multiple devices. For example, the present technology can be implemented as cloud computing that is collaboratively shared and processed by multiple devices via a network. For example, the present technology can be implemented in a cloud service that provides services related to images (moving images) to any terminal such as a computer, an audio-visual (AV) device, a portable information processing terminal, or an Internet of Things (IoT) device.

[0417] Note that in this specification, a system means a collection of multiple components (devices, modules, etc.), and it is not important whether all components are in the same housing. Therefore, both multiple devices stored in different housings and connected via a network and a single device in which multiple modules are stored in one housing are systems.

[0418] <Fields and Applications to which the Present Technology Applies>

[0419] A system, device, processing unit, etc. to which the present technology is applied can be used, for example, in any field such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty care, factories, household appliances, weather, and natural surveillance. In addition, its application is also arbitrary.

[0420] As a technology (so-called intelligent construction) aimed at improving the productivity and safety of a construction site and solving the shortage of manpower, the present technology can be used, for example, to create a digital twin at a construction site through three-dimensional surveying, construction management using the digital twin, etc. For example, the present technology can be applied to three-dimensional surveying using sensors installed on drones or construction machinery and feedback based on three-dimensional data (e.g., point cloud) obtained through the surveying (e.g., construction progress management, earthwork volume management, etc.).

[0421] Such surveys are usually carried out multiple times at different times (e.g., every other day, etc.). That is, the point cloud data obtained through surveys at different times are accumulated. Therefore, as the number of surveys increases, the increase in the storage capacity and transmission cost of the point cloud data sequence may become a problem. There is redundancy between the point clouds at different times, such as having a similar structure. That is, there is temporal redundancy. By applying the present technology, it is expected that point cloud compression using the redundancy in the time direction, i.e., point cloud sequence compression, is effective.

[0422] <Other>

[0423] Note that in this specification, a "flag" is information for identifying multiple states, and includes not only information for identifying two states of true (1) and false (0), but also information capable of identifying three or more states. Therefore, the values that the "flag" can take can be, for example, binary such as 1 / 0 or ternary or more. That is, the number of bits forming the "flag" is an arbitrary number and can be one bit or multiple bits. In addition, it is assumed that the identification information (including the flag) includes not only its identification information in the bit stream, but also the differential information of the identification information with respect to a certain reference information in the bit stream, and thus, in this specification, the "flag" and the "identification information" include not only their information, but also the differential information with respect to the reference information.

[0424] In addition, various information (e.g., metadata) related to the encoded data (bit stream) can be transmitted or recorded in any form as long as it is associated with the encoded data. Here, the term "associated" means, for example, allowing the use (linking) of other data when processing one data. That is, the data associated with each other can be collected as one data, or can be separate data. For example, information associated with the encoded data (image) can be transmitted on a transmission path different from that of the encoded data (image). In addition, for example, information associated with the encoded data (image) can be recorded in a recording medium different from that of the encoded data (image) (or another recording area of the same recording medium). Note that this "association" can be not all of the data, but a part of the data. For example, an image and the information corresponding to the image can be associated with each other in any unit such as multiple frames, one frame, or a part within a frame.

[0425] Note that in this specification, terms such as "combine", "multiplex", "add", "merge", "include", "store", "put", "introduce", "insert" mean, for example, combining multiple objects into one object, such as combining the encoded data and the metadata into one data, and mean a method of the above-mentioned "association".

[0426] In addition, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the scope of the present technology.

[0427] For example, the configuration described as one device (or processing unit) can be divided and configured into multiple devices (or processing units). Conversely, the above configuration as multiple devices (or processing units) can be commonly configured as one device (or processing unit). In addition, it goes without saying that configurations other than the above can be added to the configuration of each device (or each processing unit). In addition, as long as the configuration and operation of the entire system are substantially the same, a part of the configuration of a certain device (or processing unit) can be included in the configuration of another device (or another processing unit).

[0428] In addition, for example, the above program can be executed in any device. In this case, the device only needs to have the necessary functions (function blocks, etc.) and obtain the necessary information.

[0429] In addition, for example, each step in a flowchart can be executed by one device, or can be executed by being shared by multiple devices. In addition, when multiple processing blocks are included in one step, the multiple processing blocks can be executed by one device, or can be shared and executed by multiple devices. In other words, the multiple processing blocks included in one step can also be executed as the processing blocks of multiple steps. Conversely, the processing described as multiple steps can also be executed commonly as one step. z

[0430] In addition, for example, in a program executed by a computer, the processing of the steps describing the program can be executed in chronological order as described in this specification, or can be executed in parallel or separately at the required timing, for example, when called. That is, as long as there is no contradiction, the processing blocks of each step can be executed in an order different from the above order. In addition, the processing in the steps describing the program can be executed in parallel with the processing of other programs, or can be executed in combination with the processing of other programs.

[0431] In addition, for example, as long as there is no contradiction, multiple technologies related to the present technology can be independently implemented as a single entity. It goes without saying that any multiple of the present technologies can be combined and implemented. For example, a part or all of the present technology described in any embodiment can be combined and implemented with a part or all of the present technology described in other embodiments. In addition, a part or all of any of the above-presented present technologies can be implemented together with other technologies not described above.

[0432] Note that the present technology can also provide the following configuration.

[0433] (1) An information processing device, comprising:

[0434] A vector synthesis unit configured to generate a synthesized vector by combining a plurality of context vectors corresponding to a processing target node of three-dimensional (3D) data having a tree structure by using an importance coefficient vector for controlling the degree of contribution to prediction;

[0435] A prediction probability vector derivation unit configured to derive a prediction probability vector indicating probability values of occupancy states that each child node of the processing target node can take based on the synthesized vector; and

[0436] The occupancy state encoding unit is configured to encode information indicating the occupancy state of the child nodes of the processing target node by using the prediction probability vector, where

[0437] The context vectors among the plurality of context vectors correspond to the occupancy states of a nearby region in the spatial direction of the processing target node in a processing target frame, or correspond to the occupancy states of a nearby region in the spatial direction of a node corresponding to the processing target node in a nearby frame in the temporal direction,

[0438] The plurality of context vectors include: context vectors corresponding to the occupancy states of the nearby region in a processing target layer of the processing target frame; context vectors corresponding to the occupancy states of the nearby region in the processing target layer of the nearby frame; and context vectors corresponding to the occupancy states of the nearby region in a layer lower than the processing target layer of the nearby frame, and

[0439] The prediction is a prediction of the occupancy state of the child nodes of the processing target node by using intra-frame correlation and inter-frame correlation.

[0440] (2) The information processing apparatus according to (1), wherein

[0441] The vector synthesis unit derives a weighted sum by weighting each of the plurality of context vectors by each element of the importance coefficient vector, and sets the weighted sum as the synthesized vector.

[0442] (3) The information processing apparatus according to (1) or (2), wherein

[0443] The prediction probability vector derivation unit derives a prediction probability vector by using a multi-layer perceptron that takes the synthesized vector as an input.

[0444] (4) The information processing apparatus according to any one of (1) to (3) further includes:

[0445] A context vector derivation unit configured to derive each of the context vectors based on the occupancy state of the nearby region.

[0446] (5) The information processing apparatus according to (4), wherein

[0447] The context vector derivation unit derives a context vector corresponding to the occupancy state of the nearby region in the processing target layer of the processing target frame, a context vector corresponding to the occupancy state of the nearby region in the processing target layer of the frame immediately before the processing target frame, a context vector corresponding to the occupancy state of the nearby region in the processing target layer of the frame immediately after the processing target frame, and a context vector corresponding to the occupancy state of the nearby region in a layer lower than the processing target layer of the frame immediately before the processing target frame, by using mutually different neural networks.

[0448] (6) The information processing apparatus according to (5), wherein

[0449] The dimension of the importance coefficient vector is four.

[0450] (7) The information processing apparatus according to any one of (1) to (6) further includes: [[ID=·16]]

[0451] A code amount estimation unit configured to estimate the code amount in the case where the prediction probability vector derived by the prediction probability vector derivation unit is applied; and

[0452] A selection unit configured to select, based on the code amount estimated by the code amount estimation unit, the importance coefficient vector to be applied from among a plurality of candidates, wherein

[0453] The vector synthesis unit generates the synthesis vector corresponding to each of the plurality of candidates of the importance coefficient vector,

[0454] The prediction probability vector derivation unit derives the prediction probability vector corresponding to each of the plurality of candidates,

[0455] The code amount estimation unit estimates the code amount corresponding to each of the plurality of candidates,

[0456] The selection unit selects, from among the candidates, the candidate corresponding to the smallest code amount among the code amounts respectively corresponding to the plurality of candidates, as the importance coefficient vector to be applied to the encoding of the information indicating the occupancy state of the child node of the processing target node, and

[0457] The occupancy status encoding unit encodes information indicating the occupancy status of the child nodes of the processing target node by using the candidate selected by the selection unit.

[0458] (8) The information processing apparatus according to any one of (1) to (7) further includes:

[0459] An importance coefficient vector encoding unit configured to encode the importance coefficient vector.

[0460] (9) The information processing apparatus according to (8), wherein

[0461] The importance coefficient vector encoding unit performs entropy encoding on an index indicating the importance coefficient vector.

[0462] (10) The information processing apparatus according to any one of (1) to (9), wherein

[0463] The vector synthesis unit generates the synthesis vector by performing scaled dot-product attention calculation using a query vector having the same dimension as each of the context vectors.

[0464] (11) The information processing apparatus according to (10) further includes:

[0465] A context vector derivation unit configured to derive each of the context vectors by using a neural network for each layer.

[0466] (12) The information processing apparatus according to (10) or (11) further includes:

[0467] A query vector encoding unit configured to encode the query vector.

[0468] (13) The information processing apparatus according to (12), wherein

[0469] The query vector encoding unit performs entropy encoding on an index indicating the query vector.

[0470] (14) The information processing apparatus according to (13) further includes:

[0471] An entropy model derivation unit configured to derive an entropy model of the query vector, wherein

[0472] The query vector encoding unit applies the derived entropy model to perform entropy encoding on the index.

[0473] (15) The information processing apparatus according to any one of (12) to (13), wherein,

[0474] The query vector encoding unit performs entropy encoding on each element of the query vector.

[0475] (16) The information processing apparatus according to any one of (10) to (15), wherein,

[0476] The vector synthesis unit generates the synthesis vector by performing multi-head attention calculation using the query vector.

[0477] (17) The information processing apparatus according to any one of (10) to (16), wherein,

[0478] The vector synthesis unit generates the synthesis vector by using the query vector common to each predetermined region.

[0479] (18) The information processing apparatus according to any one of (1) to (17), wherein,

[0480] Each of the context vectors corresponds to an occupancy state of a sub-near region formed in the near region, and

[0481] The vector synthesis unit combines the context vectors of the respective sub-near regions.

[0482] (19) The information processing apparatus according to any one of (1) to (18), wherein,

[0483] Each of the context vectors includes metadata.

[0484] (20) An information processing method, comprising:

[0485] Combining a plurality of context vectors corresponding to a processing object node of 3D data having a tree structure by using an importance coefficient vector for controlling a contribution degree to prediction to generate a synthesis vector;

[0486] Deriving a prediction probability vector indicating a probability value of an occupancy state that each child node of the processing object node can take based on the synthesis vector; and

[0487] Encoding information indicating an occupancy state of the child nodes of the processing object node by using the prediction probability vector, wherein,

[0488] The context vectors among the multiple context vectors correspond to the occupancy state of a nearby area in the spatial direction of the processing object node in the processing object frame, or correspond to the occupancy state of a nearby area in the spatial direction of the node corresponding to the processing object node in a nearby frame in the time direction.

[0489] The multiple context vectors include: a context vector corresponding to the occupancy state of the nearby area in the processing object layer of the processing object frame; a context vector corresponding to the occupancy state of the nearby area in the processing object layer of the nearby frame; and a context vector corresponding to the occupancy state of the nearby area in a layer lower than the processing object layer of the nearby frame, and

[0490] The prediction is a prediction of the occupancy state of the child nodes of the processing object node by using intra-frame correlation and inter-frame correlation.

[0491] (21) An information processing apparatus, comprising:

[0492] A vector synthesis unit configured to generate a synthesis vector by combining multiple context vectors corresponding to a processing object node of 3D data having a tree structure by using an importance coefficient vector for controlling the contribution degree to the prediction;

[0493] A prediction probability vector derivation unit configured to derive a prediction probability vector indicating the probability values of the occupancy states that each child node of the processing object node can take based on the synthesis vector; and

[0494] An occupancy state decoding unit configured to decode a bitstream by using the prediction probability vector to generate information indicating the occupancy state of the child nodes of the processing object node, where

[0495] The context vectors among the multiple context vectors correspond to the occupancy state of a nearby area in the spatial direction of the processing object node in the processing object frame, or correspond to the occupancy state of a nearby area in the spatial direction of the node corresponding to the processing object node in a nearby frame in the time direction.

[0496] The multiple context vectors include: a context vector corresponding to the occupancy state of the nearby area in the processing object layer of the processing object frame; a context vector corresponding to the occupancy state of the nearby area in the processing object layer of the nearby frame; and a context vector corresponding to the occupancy state of the nearby area in a layer lower than the processing object layer of the nearby frame, and

[0497] The prediction is a prediction of the occupancy state of the child nodes of the processing object node by using intra-frame correlation and inter-frame correlation.

[0498] (22) The information processing apparatus according to (21), wherein,

[0499] The vector synthesis unit derives a weighted sum by weighting each of the plurality of context vectors by each element of the importance coefficient vector, and sets the weighted sum as the synthesis vector.

[0500] (23) The information processing apparatus according to (21) or (22), wherein,

[0501] The prediction probability vector derivation unit derives a prediction probability vector by using a multi-layer perceptron that uses the synthesis vector as an input.

[0502] (24) The information processing apparatus according to any one of (21) to (23) further includes:

[0503] A context vector derivation unit configured to derive each of the context vectors based on the occupancy state of the nearby region.

[0504] (25) The information processing apparatus according to (24), wherein,

[0505] The context vector derivation unit derives a context vector corresponding to the occupancy state of the nearby region in the processing object layer of the processing object frame, a context vector corresponding to the occupancy state of the nearby region in the processing object layer of the frame immediately before the processing object frame, a context vector corresponding to the occupancy state of the nearby region in the processing object layer of the frame immediately after the processing object frame, and a context vector corresponding to the occupancy state of the nearby region in a layer lower than the processing object layer of the frame immediately before the processing object frame by using mutually different neural networks.

[0506] (26) The information processing apparatus according to (25), wherein,

[0507] The dimension of the importance coefficient vector is four.

[0508] (27) The information processing apparatus according to (25) or (26) further includes:

[0509] A nearby area occupancy state setting unit, the nearby area occupancy state setting unit being configured to set: the occupancy state of the nearby area in the processing object layer of the processing object frame; the occupancy state of the nearby area in the processing object layer of the frame immediately preceding the processing object frame; the occupancy state of the nearby area in the processing object layer of the frame immediately following the processing object frame; and the occupancy state of the nearby area in the layer lower than the processing object layer of the frame immediately preceding the processing object frame, where,

[0510] The context vector derivation unit derives each context vector based on the occupancy state of each nearby area set by the nearby area occupancy state setting unit.

[0511] (28) The information processing apparatus according to any one of (21) to (27) further includes:

[0512] An importance coefficient vector decoding unit, the importance coefficient vector decoding unit being configured to decode a bitstream to generate the importance coefficient vector.

[0513] (29) The information processing apparatus according to (28), where,

[0514] The importance coefficient vector decoding unit performs entropy decoding on the bitstream to generate an index indicating the importance coefficient vector.

[0515] (30) The information processing apparatus according to any one of (21) to (29), where,

[0516] The vector synthesis unit generates the synthesis vector by performing scaled dot product attention calculation using a query vector having the same dimension as each of the context vectors.

[0517] (31) The information processing apparatus according to (30) further includes:

[0518] A context vector derivation unit, the context vector derivation unit being configured to derive each of the context vectors by using a neural network for each layer.

[0519] (32) The information processing apparatus according to (30) or (31) further includes:

[0520] A query vector decoding unit, the query vector decoding unit being configured to decode a bitstream to generate the query vector.

[0521] (33) The information processing apparatus according to (32), where,

[0522] The query vector decoding unit performs entropy decoding on the bitstream to generate an index indicating the query vector.

[0523] (34) The information processing apparatus according to (33) further includes:

[0524] An entropy model derivation unit configured to derive an entropy model of the query vector, where

[0525] The query vector decoding unit applies the derived entropy model to perform entropy decoding on the bitstream to generate the index.

[0526] (35) The information processing apparatus according to any one of (32) to (34), where

[0527] The query vector decoding unit performs entropy decoding on the bitstream to generate each element of the query vector.

[0528] (36) The information processing apparatus according to any one of (30) to (35), where

[0529] The vector synthesis unit generates the synthesis vector by performing multi-head attention calculation using the query vector.

[0530] (37) The information processing apparatus according to any one of (30) to (36), where

[0531] The vector synthesis unit generates the synthesis vector by using the query vector common to each predetermined region.

[0532] (38) The information processing apparatus according to any one of (21) to (37), where

[0533] Each of the context vectors corresponds to an occupancy state of a sub-near region formed in the near region, and

[0534] The vector synthesis unit combines the context vectors of the respective sub-near regions.

[0535] (39) The information processing apparatus according to any one of (21) to (38), where

[0536] Each of the context vectors includes metadata.

[0537] (40) An information processing method, including:

[0538] Combining a plurality of context vectors corresponding to processing object nodes of 3D data having a tree structure by using an importance coefficient vector for controlling the degree of contribution to prediction to generate a synthesis vector;

[0539] Deriving a prediction probability vector indicating the probability values of the occupancy states that each child node of the processing object node can take based on the synthesized vector; and

[0540] Decoding the bitstream by using the prediction probability vector to generate information indicating the occupancy states of the child nodes of the processing object node, where

[0541] The context vector in the plurality of context vectors corresponds to the occupancy state of the vicinity region in the spatial direction of the processing object node in the processing object frame, or corresponds to the occupancy state of the vicinity region in the spatial direction of the node corresponding to the processing object node in the vicinity frame in the time direction,

[0542] The plurality of context vectors include: a context vector corresponding to the occupancy state of the vicinity region in the processing object layer of the processing object frame; a context vector corresponding to the occupancy state of the vicinity region in the processing object layer of the vicinity frame; and a context vector corresponding to the occupancy state of the vicinity region in the layer lower than the processing object layer of the vicinity frame, and

[0543] The prediction is a prediction of the occupancy state of the child nodes of the processing object node by using intra-frame correlation and inter-frame correlation.

[0544] List of reference marks

[0545] 100 Geometric information encoding device

[0546] 111 Quantization unit

[0547] 112 Octree construction unit

[0548] 113 Octree encoding unit

[0549] 131 Codebook storage unit

[0550] 132 Vicinity voxel setting unit

[0551] 133 Frame memory

[0552] 134 Context vector derivation unit

[0553] 135 Vector synthesis unit

[0554] 136 MLP

[0555] 137 Bit size approximation derivation unit

[0556] 138 Codeword selection unit

[0557] 139 Importance Coefficient Vector Encoding Unit

[0558] 140 Occupancy State Encoding Unit

[0559] 151 to 154 3DCNN

[0560] 200 Geometric Information Decoding Device

[0561] 211 Octree Decoding Unit

[0562] 212 Point Cloud Construction Unit

[0563] 231 Frame Memory

[0564] 232 Nearby Voxel Setting Unit

[0565] 233 Context Vector Derivation Unit

[0566] 234 Importance Coefficient Vector Decoding Unit

[0567] 235 Codebook Storage Unit

[0568] 236 Vector Synthesis Unit

[0569] 237 MLP

[0570] 238 Occupancy State Decoding Unit

[0571] 331 Codebook Storage Unit

[0572] 334 Context Vector Derivation Unit

[0573] 335 Vector Synthesis Unit

[0574] 337 Bit Size Approximation Derivation Unit

[0575] 339 Query Vector Encoding Unit

[0576] 433 Context Vector Derivation Unit

[0577] 435 Codebook Storage Unit

[0578] 436 Vector Synthesis Unit

[0579] 511 Probability Vector Generation Unit

[0580] 535 Vector Synthesis Unit

[0581] 539 Query Vector Encoding Unit

[0582] 551 Element Average Value Calculation Unit

[0583] 552 MLP

[0584] 611 Probability Vector Generation Unit

[0585] 634 Query Vector Decoding Unit

[0586] 636 Vector Composition Unit

[0587] 900 Computer

Claims

1. An information processing apparatus, comprising: a vector synthesis unit configured to generate a synthesis vector by combining a plurality of context vectors corresponding to a processing object node of 3D data having a tree structure by using an importance coefficient vector for controlling a contribution degree to prediction; a prediction probability vector derivation unit configured to derive a prediction probability vector indicating probability values of occupancy states that each child node of the processing object node can take based on the synthesis vector; and an occupancy state encoding unit configured to encode information indicating an occupancy state of the child node of the processing object node by using the prediction probability vector, wherein the context vectors among the plurality of context vectors correspond to occupancy states of a vicinity region in a spatial direction of the processing object node in a processing object frame, or correspond to occupancy states of a vicinity region in a spatial direction of a node corresponding to the processing object node in a vicinity frame in a time direction, the plurality of context vectors include: a context vector corresponding to an occupancy state of the vicinity region in a processing object layer of the processing object frame; a context vector corresponding to an occupancy state of the vicinity region in the processing object layer of the vicinity frame; and a context vector corresponding to an occupancy state of the vicinity region in a layer lower than the processing object layer of the vicinity frame, and the prediction is a prediction of an occupancy state of the child node of the processing object node by using intra-frame correlation, inter-frame correlation, or both the intra-frame correlation and the inter-frame correlation.

2. The information processing apparatus according to claim 1, wherein the vector synthesis unit derives a weighted sum by weighting each of the plurality of context vectors by each element of the importance coefficient vector, and sets the weighted sum as the synthesis vector.

3. The information processing apparatus according to claim 1, wherein the prediction probability vector derivation unit derives the prediction probability vector by using a multi-layer perceptron having the synthesis vector as an input.

4. The information processing apparatus according to claim 1, further comprising: a context vector derivation unit configured to derive each of the context vectors based on the occupancy state of the vicinity region.

5. The information processing apparatus according to claim 4, wherein the context vector derivation unit derives a context vector corresponding to an occupancy state of the vicinity region in a processing object layer of the processing object frame, a context vector corresponding to an occupancy state of the vicinity region in a processing object layer of a frame immediately before the processing object frame, a context vector corresponding to an occupancy state of the vicinity region in a processing object layer of a frame immediately after the processing object frame, and a context vector corresponding to an occupancy state of the vicinity region in a layer lower than the processing object layer of a frame immediately before the processing object frame by using mutually different neural networks.

6. The information processing apparatus according to claim 1, further comprising: An importance coefficient vector encoding unit configured to encode the importance coefficient vector.

7. The information processing apparatus according to claim 6, wherein The importance coefficient vector encoding unit performs entropy encoding on an index indicating the importance coefficient vector.

8. The information processing apparatus according to claim 1, wherein The vector synthesis unit generates the synthesized vector by performing scaled dot product attention calculation using a query vector having the same dimension as each of the context vectors.

9. The information processing apparatus according to claim 8, further comprising: A context vector derivation unit configured to derive each of the context vectors through a neural network of each layer.

10. The information processing apparatus according to claim 8, further comprising: A query vector encoding unit configured to encode the query vector.

11. The information processing apparatus according to claim 10, wherein The query vector encoding unit performs entropy encoding on an index indicating the query vector.

12. The information processing apparatus according to claim 11, further comprising: An entropy model derivation unit configured to derive an entropy model of the query vector, wherein The query vector encoding unit applies the derived entropy model to perform entropy encoding on the index.

13. The information processing apparatus according to claim 10, wherein The query vector encoding unit performs entropy encoding on each element of the query vector.

14. The information processing apparatus according to claim 8, wherein The vector synthesis unit generates the synthesized vector by performing multi-head attention calculation using the query vector.

15. The information processing apparatus according to claim 8, wherein The vector synthesis unit generates the synthesized vector by using the query vector common to each predetermined region.

16. The information processing apparatus according to claim 1, wherein Each of the context vectors corresponds to an occupancy state of a sub-near region formed in the near region, and The vector synthesis unit combines the context vectors of the respective sub-near regions.

17. The information processing apparatus according to claim 1, wherein Each of the context vectors includes metadata.

18. An information processing method, comprising: Combining a plurality of context vectors corresponding to a processing object node of 3D data having a tree structure by using an importance coefficient vector for controlling a contribution degree to a prediction to generate a synthesized vector; Deriving a prediction probability vector indicating a probability value of an occupancy state that each child node of the processing object node can take based on the synthesized vector; And Encoding information indicating the occupancy state of the child nodes of the processing object node by using the prediction probability vector, wherein The context vectors among the multiple context vectors correspond to the occupancy state of a nearby area in the spatial direction of the processing object node in the processing object frame, or correspond to the occupancy state of a nearby area in the spatial direction of the node corresponding to the processing object node in a nearby frame in the time direction. The multiple context vectors include: the context vector corresponding to the occupancy state of the nearby area in the processing object layer of the processing object frame; the context vector corresponding to the occupancy state of the nearby area in the processing object layer of the nearby frame; and the context vector corresponding to the occupancy state of the nearby area in a layer lower than the processing object layer of the nearby frame. The prediction is a prediction of the occupancy state of the child nodes of the processing object node by using intra-frame correlation, inter-frame correlation, or both the intra-frame correlation and the inter-frame correlation.

19. An information processing apparatus, comprising: a vector synthesis unit configured to generate a synthesized vector by combining multiple context vectors corresponding to a processing object node of 3D data having a tree structure by using an importance coefficient vector for controlling the contribution degree to the prediction; a prediction probability vector derivation unit configured to derive a prediction probability vector indicating probability values of occupancy states that each child node of the processing object node can take based on the synthesized vector; and an occupancy state decoding unit configured to decode a bitstream by using the prediction probability vector to generate information indicating the occupancy state of the child nodes of the processing object node, wherein the context vectors among the multiple context vectors correspond to the occupancy state of a nearby area in the spatial direction of the processing object node in the processing object frame, or correspond to the occupancy state of a nearby area in the spatial direction of the node corresponding to the processing object node in a nearby frame in the time direction. The multiple context vectors include: the context vector corresponding to the occupancy state of the nearby area in the processing object layer of the processing object frame; the context vector corresponding to the occupancy state of the nearby area in the processing object layer of the nearby frame; and the context vector corresponding to the occupancy state of the nearby area in a layer lower than the processing object layer of the nearby frame. The prediction is a prediction of the occupancy state of the child nodes of the processing object node by using intra-frame correlation, inter-frame correlation, or both the intra-frame correlation and the inter-frame correlation.

20. An information processing method, comprising: generating a synthesized vector by combining multiple context vectors corresponding to a processing object node of 3D data having a tree structure by using an importance coefficient vector for controlling the contribution degree to the prediction; deriving a prediction probability vector indicating probability values of occupancy states that each child node of the processing object node can take based on the synthesized vector; and The bitstream is decoded by using the predicted probability vector to generate information indicating the occupancy state of the child node of the processing object node, where the context vector among the plurality of context vectors corresponds to the occupancy state of the nearby area in the spatial direction of the processing object node in the processing object frame, or corresponds to the occupancy state of the nearby area in the spatial direction of the node corresponding to the processing object node in the nearby frame in the time direction, the plurality of context vectors include: a context vector corresponding to the occupancy state of the nearby area in the processing object layer of the processing object frame; a context vector corresponding to the occupancy state of the nearby area in the processing object layer of the nearby frame; and a context vector corresponding to the occupancy state of the nearby area in the layer lower than the processing object layer of the nearby frame, and the prediction is a prediction of the occupancy state of the child node of the processing object node by using intra-frame correlation, inter-frame correlation, or both the intra-frame correlation and the inter-frame correlation.

Images