Method and apparatus for extracting multi-dimensional features for dnn-based point cloud encoding

Abstract

The multidimensional feature extraction method and apparatus for DNN-based point cloud encoding of the present disclosure may include a point cloud input step; a DNN-based point cloud encoding step; a DNN-based point cloud decoding step; and a reconstructed point cloud output step.

Description

Technology Field

[0001] The present disclosure may be a technology that effectively extracts three-dimensional spatial or axial characteristics where points are located to improve the compression performance of point cloud content, thereby performing probability estimation or prediction of more accurate occupancy information and attribute information. Accordingly, the present disclosure may be utilized in the field of point cloud content compression technology. Background Technology

[0002] The field of 3D graphic content technology has grown significantly due to the rapid development of 3D sensors, including 3D scanners, LiDAR, and vision-based 3D reconstruction technology. In addition, as a result of sensor-based data collection, point clouds, which are raw 3D data formats defined by a series of points, have become widely adopted.

[0003] Each point in a point cloud can be represented as a location using 3D coordinates. As a set, a point cloud can represent the geometry of an object or scene. Optionally, each point in a point cloud can be associated with different attributes depending on the application. Examples include RGB colors for immersive applications such as VR and AR, and reflectivity for LiDAR used in autonomous driving and robotic airports. Point cloud data can enable further processing for more complete 3D modeling, such as mesh representation. Additionally, point cloud data can be directly utilized in subsequent machine-driven cognitive tasks, such as sensing and recognition.

[0004] Point clouds can often consist of a series of frames representing dynamic objects or dynamic scenes. This is particularly true for immersive applications such as AR and VR, as well as for LiDAR used in autonomous driving and robotics. Recently, AI technology has demonstrated state-of-the-art performance in the field of point cloud compression. However, unlike processing, enhancement, and rendering recognition tasks, AI-based point cloud compression is not widely used due to the lack of standards. The problem to be solved

[0005] The present disclosure aims to present a method for efficiently extracting structural correlations between points in each point cloud layer and constructing features based thereon. means of solving the problem

[0006] The present disclosure effectively reflects correlations existing along each spatial axis in point cloud data to more efficiently learn structural correlations between points and improve compression performance based thereon. In particular, it is characterized by significantly improving transmission and storage efficiency by minimizing data redundancy through feature extraction layers specialized for each spatial axis direction, thereby more accurately predicting occupancy information and attribute information of the point cloud data or their probability distributions. Effects of the invention

[0007] According to the present disclosure, structural characteristics between points can be learned more effectively by utilizing correlations existing in each axis direction in point cloud data. Based on the features of points extracted in a multidimensional manner, the prediction accuracy of occupancy information and attribute information, as well as the accuracy of distribution estimation, are improved, and efficient data compression becomes possible. Brief explanation of the drawing

[0008] Figure 1 illustrates an example of a 4-bit binary tree structure storing voxels. Figure 2 illustrates an example of an octree structure for a voxel. Figure 3 illustrates an example of an array of coding units in a bitstream. Figure 4 illustrates an example of the overall decoding process. Figure 5 illustrates examples of lossless-based decoding and loss-based decoding of the geometric decoding process. Figure 6 illustrates an example of the ancestor occupancy analysis step. Figure 7 illustrates an example of a feature decoding step. Figure 8 illustrates an example of a sparse 3D convolution-based point synthesis method. Figure 9 illustrates an example of the possession decoding step. FIG. 10 illustrates an example of a learning-based motion decoding process. FIG. 11 illustrates an example of the inter-predictor derivation step. Figure 12 illustrates an example of a motion compensation step. Figure 13 illustrates an example of the overall encoding process. Figure 14 illustrates examples of lossless-based encoding and lossy-based encoding of the geometric encoding process. Figure 15 illustrates an example of a feature encoding step. Figure 16 illustrates an example of a possession encoding step. FIG. 17 illustrates an example of a learning-based motion encoding process. FIG. 18 illustrates an example of the inter-predictor derivation step. FIG. 19 illustrates a flowchart of a multidimensional feature extraction method for DNN-based point cloud encoding according to the present disclosure. FIG. 20 illustrates an embodiment of a device in which a Deep Neural Network (DNN)-based point cloud encoding step is performed. FIGS. 21 to 23 illustrate an embodiment of a downsampler (2001). FIG. 24 illustrates an embodiment of a device in which a DNN-based point cloud decoding step is performed. FIGS. 25 to 27 illustrate an embodiment of an upsampler (2401). FIG. 28 illustrates one embodiment of a point cloud input to the present disclosure. Figure 29 illustrates coordinate systems. FIG. 30 illustrates embodiments of dimension reduction and orientation. Figure 31 illustrates the feature processing unit of a DNN-based point cloud encoder. Figure 32 illustrates the feature encoding (feature encoding section) of a DNN-based point cloud encoder. Figure 33 illustrates the feature decoding (feature decoding unit) of a DNN-based point cloud encoder. Figure 34 illustrates the probability estimation (probability estimation unit) of a DNN-based point cloud encoder. Figure 35 illustrates the residual information encoding (residual information encoding section) of a DNN-based point cloud encoder. Figure 36 illustrates the residual information decoding (residual information decoding section) of a DNN-based point cloud decoder. Figure 37 illustrates the feature decoding (feature decoding unit) of a DNN-based point cloud decoder. Figure 38 illustrates the feature processing unit of a DNN-based point cloud decoder. Figure 39 illustrates the probability estimation (probability estimation unit) of a DNN-based point cloud decoder. FIG. 40 illustrates a specific example of a parallel production structure of a downsampler of the present disclosure. FIG. 41 illustrates a specific example of a parallel production structure of an upsampler of the present disclosure. Specific details for implementing the invention

[0009] The present disclosure is subject to various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure. Similar reference numerals in the drawings refer to the same or similar functions across various aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation. The detailed description of exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments as examples. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present disclosure in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the following detailed description is not intended to be taken in a limiting sense, and the scope of exemplary embodiments is limited only by the appended claims, together with all equivalents to those claimed therein, provided they are properly described.

[0010] In this disclosure, terms such as first, second, etc. may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of this disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0011] Where it is stated that any component of the present disclosure is "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, or that there may be other components in between. On the other hand, where it is stated that a component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0012] The components shown in the embodiments of the present disclosure are depicted independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, each component is listed and included as a separate component for convenience of explanation; however, at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separated embodiments of each component are included within the scope of the rights of the present disclosure as long as they do not depart from the essence of the present disclosure.

[0013] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit this disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this disclosure, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. That is, the description in this disclosure that a specific configuration "comprising" does not exclude configurations other than that configuration, but means that additional configurations may be included within the scope of the practice or technical concept of this disclosure.

[0014] Some components of the present disclosure may not be essential components performing an essential function in the present disclosure, but may be optional components merely for enhancing performance. The present disclosure may be implemented by including only the components essential to embody the essence of the present disclosure, excluding components used merely for enhancing performance, and a structure including only the essential components, excluding optional components used merely for enhancing performance, is also included within the scope of the rights of the present disclosure.

[0015] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In describing the embodiments of this specification, if it is determined that a detailed description of related known configurations or functions may obscure the gist of this specification, such detailed description is omitted, and the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0016] The present disclosure discloses the syntax, semantics, and decoding processes of point cloud data using an Artificial Intelligence (AI)-based coding method. Additionally, the present disclosure discloses processes necessary for point cloud reconstruction and discloses additional processes such as post-decoding processes, pre-reconstruction processes, post-reconstruction processes, and adaptive processes.

[0017] The definitions of the terms of the present disclosure may be as follows.

[0018] Augmented reality (AR) can be an environment that overlays computer-generated content onto the real world viewed by the user.

[0019] Virtual reality (VR) is a computer-generated environment featuring realistic-looking scenes and objects, allowing users to feel immersed in their surroundings.

[0020] The world coordinate system can be the coordinate system that captures the point cloud. The world coordinate system can also be referred to as the original coordinate system.

[0021] A coding coordinate system can be a coordinate system in which a point cloud is coded. In a coding coordinate system, geometry can have non-negative values.

[0022] The definitions of the abbreviations of the present disclosure may be as follows.

[0023] AU: Access Unit

[0024] IDR: Instantaneous Decoding Refresh

[0025] IRAP: Intra Random Access Point

[0026] LSB: Least Significant Bit

[0027] MSB: Most Significant Bit

[0028] PCC: Point Cloud Compression

[0029] AI-PCC: AI-based Point Cloud Compression

[0030] Vector can be a vector with a number of elements.

[0031] Matrix<r, c> It can be a matrix with r rows and c columns.

[0032] Tensor<r, a> can be a sparse tensor specified as a-dimensional vector data for a list of 3-dimensional coordinates. Here, the spatial resolution for each x, y, and z direction is from 0 to 2 r It can be between -1. Also, parameter a can be the number of feature channels associated with each 3D point.

[0033] Concat(T1, T2, T3, ...) can be a concatenated layer in which multiple tensors T1, T2, T3, ... are connected in a single order as they appear in the list. Here, all connected tensors must have the same resolution and may have the same set of points. Additionally, each tensor may have a different number of feature channels.

[0034] ReLU (rectified linear unit) is a type of non-linear activation function used in artificial neural networks, and can be called a rectified linear unit. For example, the function outputs the input value as is when it is greater than a threshold value, but outputs 0 otherwise (when the input value is less than or equal to the threshold value).

[0035] Conv<a, b, c, d, e> can be a 3D synthesizing layer with a kernel size of a×a×a. Here, b is a stride parameter, c is a dilation size, d is an input channel size, and e is an output channel size.

[0036] Deconv<a, b, c, d, e> can be a 3D inverse convolutional layer with a kernel size of a×a×a. Here, b is the stride parameter, c is the dilation size, d is the input channel size, and e is the output channel size.

[0037] MLP <c0, c1, c2, ..., c n > can be a multi-layer perceptron (MLP) having n layers. Here, c0 is the size of the input channel, and c i ≡ can be the size of the output channel of each layer (1 ≤ i ≤ n).

[0038] Variables (or terms) related to the point cloud geometry of the present disclosure may be referred to as points or voxels. A point explicitly indicates the location (x, y, z) of a sample but may not specify the size of the occupied space. A voxel implicitly indicates the location of a sample through its position in an octree and may represent a cube of a specific size. The center of a voxel may be used as the location of the corresponding point.

[0039] When point cloud geometry is represented by points, each point cloud frame may consist of an array of 3D positions (x, y, z). Here, the coordinate values ​​of the 3D positions may be represented as signed integers in the world coordinate system. Additionally, the coordinate values ​​of the 3D positions may be represented as n-bit unsigned integers in the shifted coding coordinate system of the present disclosure. Here, n is the bit depth and may represent the precision of the point position. Additionally, n may be explicitly determined by information signaled from the bitstream or implicitly determined by other point cloud-related information. In this case, in the shifted coding coordinate system, one edge is (0,0,0) and the opposite edge is (2 n-1 , 2 n-1 , 2 n-1 )person 2 n The size of the cubic bounding box can be defined.

[0040] The scaled coding coordinate system of the present disclosure may be a coordinate system obtained by scaling a shifted coding coordinate system to define a scaled coding coordinate system for compression. Whether scaling is applied, the degree of scaling, etc., may be explicitly determined by information signaled from the bitstream or implicitly determined by other point cloud-related information.

[0041] Figure 1 illustrates an example of a 4-bit binary tree structure storing voxels.

[0042] An octree (or octree structure) is another way of representing a point that can hierarchically partition space. In point cloud geometry with n-bit coordinates, an octree can have a total of n+1 levels from 0 to n. Here, 0 can represent the root level. An octree, which serves to store voxels—3D data—can be a tree structure in which each node has 8 children.

[0043] Before describing the octree, referring to Fig. 1, the binary tree structure of Fig. 1 is 4 bits, so it can have a total of 5 levels from 0 to 4. In addition, each node represents a voxel, and an occupied voxel can have 2 child nodes.

[0044] Figure 2 illustrates an example of an octree structure for a voxel.

[0045] In an octree (or octree structure), for octree level i = 0 (root level), a voxel can occupy the entire space within the point cloud bounding box. For each level, the space occupied by the voxel in the parent level can be divided equally in the x, y, and z directions to generate eight child voxels of equal size. In other words, each child voxel can occupy 1 / 8 of the parent voxel's space. Accordingly, the voxel size is determined by the octree level i to which it belongs, and when normalized, it is 2 n-i or 2 n-i / 2 n It may be. In the present disclosure, the child voxel may also be expressed as an octant.

[0046] In the present disclosure, occupied voxels may be represented by a value of 1, and empty voxels (unoccupied voxels) may be represented by a value of 0.

[0047] Occupancy map (or voxel occupancy map) of the present disclosure O i can represent the point cloud for octree level i. Also, occupancy map O i is Tensor<i, 1> It can be expressed as such. In this case, the spatial resolution of the tensor can be specified as the current octree level i. The one-dimensional value for each point of the tensor can indicate whether the voxel is occupied (=1) or empty (=0).

[0048] At a specific octree level, the voxelCount variable can be defined as the number of occupied voxels in the parent octree level. The said variable can be obtained by the GetVoxelCountParent() function, which counts the number of occupied voxels in the parent octree level.

[0049] Attributes of a point cloud may provide additional information related to each point of a source point cloud. If attributes exist, each attribute may be associated with the corresponding point of the source point cloud. The attribute types of the present disclosure may include color and reflectance.

[0050] The color attributes of the present disclosure may be represented in an RGB color space. For example, each color channel (red, green, blue) may be represented as an 8-bit unsigned integer value within the range [0, 255].

[0051] When point cloud geometry is represented as an array of 3D positions (x, y, z), color attributes can be represented as an array with an extended field (x, y, z, r, g, b).

[0052] When point cloud geometry is represented by octree voxels, the color property is a Tensor<n, 3> It can be expressed as follows. Here, n can represent the bit depth of the point cloud geometry. The 3D vector for each point in the tensor can represent the associated color value (r, g, b).

[0053] When point cloud geometry is represented as an array of 3D positions (x, y, z), the reflectance property can be represented as an array with an extended field (x, y, z, r). Here, r can represent reflectance.

[0054] When point cloud geometry is represented by octree voxels, the reflectance property is a Tensor<n, 1> It can be expressed as follows. Here, n can represent the bit depth of the point cloud geometry. A one-dimensional scalar for each point of the tensor can represent the associated reflectance value r.

[0055] Point cloud features can be used as latent representations of neural network-based processes during point cloud encoding and decoding. Point cloud features can be used as a format other than the occupancy map in the encoded point cloud. Point cloud features may not be included in the source point cloud or the decoded point cloud.

[0056] A set of features spanning all voxels within an occupancy map can be called a feature map. At octree level i, the feature map is a Tensor<j, a> It can be expressed as. Here, j is feature map 2 j ×2 j ×2 j It represents the spatial resolution of and may be equal to or different from octree level i. Here, a may represent the number of feature channels.

[0057] Each point in the feature map can correspond to an occupied or empty voxel. Each point in the feature map can be associated with an a-dimensional feature vector. Each a-dimensional feature vector can encode geometric information for intra-coding. Alternatively, it can encode temporal dynamics and motion patterns for inter-predictive coding.

[0058] In addition, the feature map can be used for lossy decoding of the occupancy map or to support lossless decoding.

[0059] Points in a point cloud can be sorted according to the raster scanning order. Raster scanning can proceed in the order of the z-direction first, then the y-direction, and finally the x-direction. This can be equivalent to sorting points in ascending order using a key value k calculated based on the point location (x, y, z). Here, k is x × 2 r+r + y×2 r It could be + z.

[0060] In addition, using raster scanning order, the tensor T is converted into a 2D matrix M = Matrix<N, c> You can map it to or to the opposite.

[0061] A point cloud frame can be defined as a complete set of point cloud data representing a 3D scene or object at a specific point in time. A point cloud sequence can consist of a series of point cloud frames.

[0062] A point cloud frame is divided into one or more point cloud slices, and a point cloud slice may be a spatial subunit of the frame. Each slice contains a subset of points within the frame and can generally be grouped according to spatial locality, coding efficiency, or parallel processing requirements. Encoding of the point cloud frame can be performed by independently encoding each constituent slice.

[0063] There may be four types of data coding units: raw coding units, feature coding units, occupancy coding units, and motion coding units. When reproducible coding is enabled, a protection coding unit may be additionally used. All data coding units, except for the raw coding unit, can be connected to a protection coding unit. Through this connection, the corresponding data can be decoded reliably and consistently, which can be particularly important for cross-platform compatibility and deterministic decoding motion.

[0064] Figure 3 illustrates an example of an array of coding units in a bitstream.

[0065] Coding units can be hierarchically organized from Octree Level 1 to the last coded Octree Level. The last coded Octree Level may differ from the last Octree Level of the input point cloud.

[0066] At octree level i=0, which corresponds to the root node of an octree structure, no data may be explicitly coded into the bitstream. By definition, the root node can be considered to always be occupied.

[0067] For octree level i=1, raw coding units can be used to code the occupancy status of 8 octants.

[0068] Using a raw coding unit, 1 byte is used to encode an occupancy of 8 octants, and each bit can correspond to one octant.

[0069] For octree levels 1 < i ≤ j, one feature coding unit may be present in the bitstream for each coded octree level. Additionally, each level may encode eight occupying coding units after encoding one feature coding unit. That is, the feature coding unit can be used to assist the occupying coding unit. Here, the eight occupying coding units may correspond to eight octants according to the raster scan order ((0,0,0), (1,0,0), (0,1,0), (1,1,0), (0,0,1), (1,0,1), (0,1,1), (1,1,1)).

[0070] The occupancy coding unit may include encoded occupancy data indicating the occupancy status of a voxel for a specific octant of the current octree level.

[0071] Referring to Fig. 3, lossless coded octree levels (or lossless-based octree levels) can represent octree levels 1 ≤ i ≤ j.

[0072] Referring to Fig. 3, lossy-coded octree levels (or losssy-based octree levels) can represent octree levels j < i ≤ n.

[0073] For octree levels j < i ≤ n, each level may need to encode one feature coding unit. Additionally, the feature coding unit can be used to derive an occupancy map through a point synthesis process using a lossy coding method. Furthermore, for lossy-coded octree levels, there may be no occupancy coding unit.

[0074] For inter-coded frames, for each coded octree level where i > 1, one motion coding unit must be inserted before the feature coding unit.

[0075] A protection coding unit may be encoded in the bitstream to protect the decoding of a feature coding unit, a occupancy coding unit, and a motion coding unit (all data coding units excluding raw coding units). Each protection coding unit is associated with a single data coding unit, and the associated single data coding unit may be a feature coding unit, an occupancy coding unit, or a motion coding unit. The role of the protection coding unit may be to protect the reproducibility of the relevant data during decoding. For example, if protection for cross-platform reproducibility is enabled for a specific feature coding unit, the protection coding unit may be encoded in the bitstream to protect the decoding of said feature coding unit.

[0076] The motion coding unit may include encoded motion feature data representing the temporal dynamics between the current frame and the reference frame of the current octree level.

[0077] For each intercoded octree level, one motion coding unit may exist in the bitstream. Additionally, as shown in FIG. 3, the motion coding unit may take precedence over the feature coding unit at the same octree level.

[0078] Motion features can be extracted, quantized, and entropy-encoded by an encoder and stored in a bitstream. The motion features can capture the time relationship and motion pattern between the two slices of the current frame and the decoded reference frame, respectively, at the current octree level.

[0079] Figure 4 illustrates an example of the overall decoding process.

[0080] The decoding process of the present disclosure may include learning-based point cloud geometric decoding and Region Adaptive Hierarchical Transform (RAHT)-based attribute decoding.

[0081] Both geometry decoding and attribute decoding can have intra mode and inter mode as decoding modes.

[0082] Geometric decoding is neural network-based decoding, and neural network weights can be obtained according to model identifier information before the geometric decoding process is executed. Here, the model identifier information may be information explicitly signaled from the bitstream, information pre-defined in the device, or information implicitly determined by other information.

[0083] For octree level i=1, eight binary occupancy values ​​can be decoded into a raw coding unit consisting of a single byte. The eight voxels can be arranged according to raster scanning order. The bits of the byte in the raw coding unit can be indexed from the Most Significant Bit (MSB) to the Least Significant Bit (LSB). Each bit can represent the occupancy status of the corresponding voxel. Residual features at this level , Tensor<0, 8> can be set as a constant dummy feature where all elements are 1.

[0084] In the case of intra-point cloud geometry decoding, the geometry of the decoded point cloud can be constructed in a Learning-based Geometry Decoding Unit based on the geometry bitstream of the point cloud. Intra-point cloud geometry decoding can be executed for each coded octree level, starting from octree level i=2.

[0085] For inter-point cloud geometry decoding, geometry predictor information can be generated in a Learning-based Motion Decoding Unit based on the motion bitstream and reference frame. This geometry predictor information is shared as an additional input with a Learning-based Geometry Decoding Unit for intra-decoding, allowing the geometry of the decoded point cloud to be constructed in the Learning-based Geometry Decoding Unit. Inter-point cloud geometry decoding can be executed for each coded octree level, starting from octree level i=2.

[0086] In the case of intra-point cloud attribute decoding, the decoded point cloud attributes can be constructed in a RAHT-based Attribute Decoding Unit based on the attribute bitstream and the decoded point cloud geometry.

[0087] In the case of inter point cloud attribute decoding, attribute predictor information can be generated in a recoloring unit based on the decoded point cloud geometry and attribute reference frame. The attribute predictor information is shared as an additional input to a RAHT-based attribute decoding unit of intra decoding, and the decoded point cloud attributes can be constructed in the RAHT-based attribute decoding unit of intra decoding.

[0088] The point cloud geometry intra-decoding process (=geometry intra-decoding process) may be called at each octree level when the current point cloud slice is determined to be in geometry intra mode (e.g., when the unit type is determined to be Geometry Intra-Slice). Here, the unit type may be explicitly signaled from the bitstream or implicitly determined by other information.

[0089] Additionally, the geometry intra-decoding process may be called for each octree level if the current octree level is greater than the threshold octree level, even if the current point cloud slice is determined to be in geometry inter-mode (e.g., even if the unit type is determined to be Geometry Inter-Slice). Here, the threshold octree level may be explicitly signaled from the bitstream or implicitly determined by other information.

[0090] The geometric intra-decoding process can be performed in the learning-based geometric decoding unit of Fig. 4.

[0091] The unit types of the present disclosure may include types of Geometry Intra-Slice, Geometry Inter-Slice, Attribute Intra-Slice, and Attribute Inter-Slice.

[0092] Types of geometric intra-decoding processes may include lossless-based decoding and lossy-based decoding.

[0093] Figure 5 illustrates examples of lossless-based decoding and loss-based decoding of the geometric decoding process.

[0094] The loss-based decoding process may include 1) an ancestor occupancy analysis step, 2) a feature decoding step, and 3) a point synthesis step.

[0095] The lossless-based decoding process may include 1) an ancestor occupancy analysis step, 2) a feature decoding step, and 4) an octree decoding step.

[0096] That is, in the loss-based decoding and lossless-based decoding processes, 1) the ancestor occupancy analysis step and 2) the feature decoding step may be common processes.

[0097] 1) Ancestor Occupancy Analysis Stage

[0098] In the ancestor occupancy analysis step, if occupancy information of the ancestor octree levels (i-1, i-2, i-3, i-4) of the current octree level is available, ancestor features are based on that information. You can obtain.

[0099] The ancestor occupancy analysis step may be an initialization step performed before the 2) feature decoding step.

[0100] Figure 6 illustrates an example of the ancestor occupancy analysis step.

[0101] The ancestor occupancy analysis step may include an ancestor feature initialization step, a feature mix step, and an adaptive affine step.

[0102] The ancestor feature initialization step is the occupancy map of the parent octree level Ancestor information that is a 32-dimensional vector based on can be obtained. Here, the ancestor information is [j, p j , o j , v j It is defined by the connection of ], where j is a scalar value representing the octree level index, p is a 3D vector indicating the 3D position, o is a 3D vector indicating the binary octant, and v is a scalar value representing the voxel size.

[0103] The Feature Mix step may be a step that performs a basic neural network-based process for feature aggregation and channel size transformed from a to c. a and c are the FeatureMix neural network layers of the Picture Mix step.<a, b, c, d, e> It can be determined through. For example, in the feature mix step of FIG. 6, the channel size can be changed from 32 to 16. All or some of the parameters used in the feature mix step may be determined differently based on bit depth-related information.

[0104] The adaptive affine stage is a style condition f representing the desired coding movement. style Based on this, an adaptive affine process can be performed to transform input features through learnable affine transformations. The adaptive affine process can be inserted into various processes such as feature extraction, point synthesis, feature encoding, feature decoding, octree encoding, and octree decoding to transform the features of the corresponding processes.

[0105] 2) Feature decoding step

[0106] In the feature decoding stage, the feature bitstream for the current octree level , ancestor characteristics of the current octree level , constant dummy prediction feature F pred , decoded residual features of the parent octree level , and the occupancy map of the parent octree level Based on, the decoded features of the current octree level and decoded residual features of the current octree level You can obtain.

[0107] Constant dummy prediction feature F pred All elements of can be equal to 1. Constant dummy prediction features can be used to match geometric feature decoding for intra-coding with inter-coding.

[0108] If no features were decoded in the previous level, the decoded residual features of the parent octree level It can be used as a dummy feature where all elements are 1.

[0109] Cases where there are no features decoded at the previous level may include i) cases where octree level i=2 is decoded (since features are not encoded at octree level i=1) and ii) cases where the value of a flag instructing to set residual features as dummy features for all octree levels for lossless coding is 1.

[0110] Figure 7 illustrates an example of a feature decoding step.

[0111] The feature decoding step may include a super-prescription synthesis step, an arithmetic decoding step, and a conditional decoding step.

[0112] The superpredictive synthesis step can synthesize feature variance parameters based on decoded residual features of the parent octree level.

[0113] The arithmetic decoding step can obtain the decoded residual features of the current octree level by arithmetic decoding the feature bitstream for the current octree level based on the feature distribution parameter obtained in the superprescription synthesis step.

[0114] The conditional decoding step can conditionally decode the decoded features of the current octree level based on the decoded residual features of the current octree level, using the ancestor features and constant dummy prediction features of the current octree level as conditions.

[0115] 3) Point composite step

[0116] In the point synthesis step, the decoded features of the current octree level Based on the decrypted occupancy map of the parent octree level, the (lossy) decrypted occupancy map of the current octree level can be obtained.

[0117] The point synthesis step can be performed for each octree level configured for loss-based decoding.

[0118] The point synthesis step can be performed in different ways based on a flag indicating whether a MultiLayer Perceptron (MLP) is used.

[0119] For example, the point synthesis step can be performed by a sparse 3D convolution-based point synthesis method when a flag indicating whether to use an MLP has a first value, and by an MLP-based point synthesis method when a flag indicating whether to use an MLP has a second value.

[0120] Figure 8 illustrates an example of a sparse 3D convolution-based point synthesis method.

[0121] For the sparse 3D convolution-based point synthesis method, as an additional input, style condition f style , can have a parameter representing the number of voxels of the current octree level obtained from the slice geometry header.

[0122] A sparse 3D convolution-based point synthesis method may include a feature aggregation step, a probability estimation step, and a classification step.

[0123] The feature aggregation step of the sparse 3D convolution-based point synthesis method is the decoded features at the current octree level Improve, and aggregate and upsample the improved features to aggregate and upsample the features You can obtain.

[0124] The probability estimation step of the sparse 3D convolution-based point synthesis method is aggregated and upsampled features Using a multi-perceptron (MLP) network as input, the predicted occupancy probability of voxels at the current octree level with one channel can be calculated. In this case, the channel size of the MLP layer used can be determined differently based on bit depth information.

[0125] The classification step of the sparse 3D convolution-based point synthesis method can obtain an occupancy map of the current octree level by sorting all child voxels (current octree level) that have an occupied parent voxel, classifying the K voxels with the highest occupancy probability as occupied and the remaining voxels as empty.

[0126] An MLP-based point synthesis method may include an offset estimation step, a point construction step, and a point clipping and pruning step.

[0127] The offset estimation step of the MLP-based point synthesis method is the decoded features of the current octree level After converting it into a feature having 15 channels, three consecutive channels can be considered as one 3D offset vector to generate five 3D offset vectors for each occupied voxel of the parent octree level.

[0128] The point construction step of the MLP-based point synthesis method can generate five 3D points of the current octree level by moving each occupied voxel of the parent octree level according to five 3D offset vectors.

[0129] The point clipping and pruning steps of the MLP-based point synthesis method clip the coordinates of 3D points based on the bounding box of the point cloud frame and prun the duplicate points to obtain the current octree level occupancy map.

[0130] 4) Octree decoding step

[0131] In the octree decoding stage, the decoded features of the current octree level Based on the occupancy map of the parent octree level and the occupancy bitstream of the current octree level, the (lossless) decrypted occupancy map of the current octree level can be obtained.

[0132] The octree decoding step can be performed for each octree level configured for lossless-based decoding.

[0133] The mode of the octree decoding step can be determined based on bit depth information. The mode of the octree decoding step may include a current level autoregressive mode and a parent level autoregressive mode.

[0134] Depending on the mode, the processes of the octree decoding step can be performed at the resolution of the current octree level or the resolution of the parent octree level.

[0135] The octree decoding step may include an initialization step, a feature aggregation step, and a possession decoding step.

[0136] In the initialization phase, depending on the mode of the octree decoding phase, the occupancy map of the parent octree level is expanded or upsampled to the occupancy map of the current octree level to obtain the initial occupancy map of the current octree level. It can generate.

[0137] In the case of the current level autoregressive mode, during the initialization phase, the occupancy map of the parent octree level is upsampled to the occupancy map of the current octree level to obtain the initial occupancy map of the current octree level. , Tensor<i, 1> You can obtain. The initial occupancy map of the current octree level has 8 × voxelCount (number of occupied voxels in the parent octree level) points, and all child voxels of an occupied parent voxel can be set to 0.

[0138] In the case of parent-level autoregressive mode, during the initialization phase, the occupancy map of the parent octree level is extended to the occupancy map of the current octree level, and the initial occupancy map of the current octree level , Tensor<i-1, 8> You can obtain. The initialization phase initializes all channels to 0 and Repeat the channel 8 times You can generate 8 child voxels. The 8 child voxels are mapped to 8 channels indexed from 0 to 7, and the order can be (0,0,0), (1,0,0), (0,1,0), (1,1,0), (0,0,1), (1,0,1), (0,1,1), (1,1,1).

[0139] In the feature aggregation stage, the decoded features of the current octree level Based on, aggregated features according to mode You can obtain.

[0140] In the case of the current level autoregressive mode, the decoded features of the current octree level are used in the feature aggregation step. Improves, aggregates the improved features, and upsamples the aggregated features You can obtain.

[0141] In the case of parent-level autoregressive mode, the decoded features of the current octree level are used in the feature aggregation step. Aggregated features obtained by aggregating without changing the resolution (having the resolution of the parent octree level). You can obtain.

[0142] Figure 9 illustrates an example of the possession decoding step.

[0143] The occupancy decoding step can be called repeatedly 8 times to decode the occupancy map of octree level i. k, representing the numbering of the iterations, can range from 1 to 8. The iterations for 8 octants can follow the order of (0,0,0), (1,0,0), (0,1,0), (1,1,0), (0,0,1), (1,0,1), (0,1,1), (1,1,1).

[0144] The possession decoding step for the k-th iteration can be performed by i) a possession buffer, ii) a sibling feature initialization unit, iii) a sibling feature mixing unit, iv) a probability estimation unit, v) an arithmetic decoding unit, and vi) a possession deserialization unit.

[0145] i) Occupied buffer

[0146] The occupancy buffer contains all sibling occupancy maps decrypted in the previous iteration of the current octree level. (from 1 to k-1) can be stored and provided so that it can be referenced in the next decoding iteration. Additionally, a mask, which is a decoding mask indicating whether a child voxel is decoded, can be managed. Also, for the first iteration (k=1), the occupied buffer can be initialized.

[0147] ii) Sibling characteristic initialization section

[0148] In the sibling feature initialization section, all sibling occupancy maps decrypted in the previous iteration Initialize features based on to initial sibling features You can obtain.

[0149] In the case of the current level autoregressive mode, the sibling feature initializer performs initialization using a 5-dimensional vector to initialize the sibling features It is possible to obtain. In this case, 1 dimension represents a scalar value for the octree level, 3 dimensions represent a scalar value for the 3D position, and 1 dimension represents a binary value of the decoding mask map.

[0150] In the case of parent-level autoregressive mode, the input in the sibling feature initialization section Initialize identically to the initial sibling characteristics You can obtain.

[0151] iii) Brothers' Characteristics Mixing Department

[0152] In the sibling characteristic mixing section, the aggregated characteristics Characteristics of early siblings Mixed characteristics by mixing You can obtain.

[0153] The mixing process of the sibling feature mixing section may vary depending on whether the mode is the current level autoregressive mode or the parent level autoregressive mode.

[0154] iv) Probability Estimator

[0155] In the probability estimation unit, mixed features Based on, the occupancy probability of voxels at the current octree level for each occupied parent voxel It can predict.

[0156] In the case of the current level autoregressive mode, the probability estimation unit uses a multi-perceptron (MLP) network to estimate the occupancy probabilities of voxels at the current octree level having one channel. It can be predicted. In this case, the channel size of the MLP layer used may be determined differently by bit depth-related information.

[0157] In the case of parent-level autoregressive mode, the probability estimator uses convolution layers, ReLU layers, and Inception ResNet (IRN) layers to estimate the occupancy probabilities of the voxels at the current octree level. It can predict.

[0158] v) Arithmetic decoding part

[0159] In the arithmetic decoding unit, based on the occupied bitstream and the predicted probability for the current octant of the current octree level, the binary value You can obtain the sequence of.

[0160] vi) Occupied deserializer

[0161] In the possession deserialization section, the decoded possession map of the previous iteration and binary sequence Use the occupancy map of the current iteration You can acquire. The occupied status can be set to 1, and unoccupied to 0. After completing all 8 iterations, the current Octree level The decrypted occupancy map of is , , ..., It can be set as a combination of.

[0162] The point cloud geometry inter-decoding process can be called for each octree level when the current point cloud slice is signaled in geometry inter mode (i.e., the unit type is signaled as Geometry Inter-Slice) and the current octree level is less than or equal to the threshold octree level.

[0163] The point cloud geometry inter-decoding process can be performed by a combination of the learning-based geometry decoding unit, the learning-based motion decoding unit, and the reference frame unit of FIG. 4.

[0164] The point cloud geometry inter-decoding process may include a motion decoding process and a main decoding process.

[0165] In the motion decoding process, a predictor for the current frame can be obtained based on the motion between the reference frame and the current frame. This may correspond to the learning-based motion decoding unit and the reference frame unit of FIG. 4.

[0166] In the main decoding process, geometric decoding can be performed based on a predictor for the current frame. This may correspond to the learning-based geometric decoding unit of Fig. 4.

[0167] The global motion compensation process can be called as a preprocessing process at every octree level before calling the inter-decoding process.

[0168] The inputs and outputs of the global motion compensation process can belong to the transformed coding coordinate system.

[0169] 1) First, the reference point cloud frame The coordinates of each point in can be transformed based on the point cloud bounding box center (c, c, c). The above transformation is It can be performed by. The position of the bounding box center in the transformed coding coordinate system is It can be defined as follows.

[0170] 2) Next, the reference point cloud frame using the rotation matrix R and translation vector T specified by the fps_transform_matrix array of the bitstream It can be converted as follows.

[0171] 3) Finally, reference point cloud frame A global motion compensation reference frame can be obtained by applying an inverse transformation to move it back to the transformed coding coordinate system. The above inverse transformation is It can be performed by.

[0172] Unless there are special circumstances, if gps_global_motion_flag is equal to 1, the reference point cloud geometry of the present disclosure can be interpreted as a global motion-compensated reference point cloud.

[0173] The learning-based motion decoding process performed in the learning-based motion decoding unit can be called for each octree level for a slice at the start of the point cloud geometry inter-decoding process.

[0174] The learning-based motion decoding process can generate predicted features to support the main decoding process.

[0175] FIG. 10 illustrates an example of a learning-based motion decoding process.

[0176] The learning-based motion decoding process may include a reference feature derivation step and an inter-predictor derivation step.

[0177] In the reference feature derivation step, reference features of the current octree level can be obtained by utilizing style conditions and the decoded occupancy map of the current octree level. Additionally, if the flag indicating whether to use a MultiLayer Perceptron (MLP) is the first value, reference features of the current octree level can be obtained by additionally utilizing the occupancy map of the previous octree level.

[0178] FIG. 11 illustrates an example of the inter-predictor derivation step.

[0179] The inter-predictor derivation step may include a motion feature buffer management step, a super-predicted synthesis step, an arithmetic decoding step, and a motion compensation step.

[0180] The motion feature buffer management step can store and provide decoded motion features. For example, the motion feature buffer management process can provide the decoded motion features of the parent octree level to the superpre-synthesis step for the current octree level.

[0181] The superpredictive synthesis step can synthesize feature variance parameters using motion features of the parent octree level. The superpredictive synthesis step can operate identically to the previously described superpredictive synthesis step. However, the input can be the decoded motion features of the parent octree level instead of the decoded residual features of the parent octree level.

[0182] The arithmetic decoding step can operate in the same manner as the previously described arithmetic decoding step. However, the input bitstream can be replaced with a motion bitstream. That is, the arithmetic decoding step can obtain the decoded motion features of the current octree level by arithmetic decoding the motion bitstream based on the feature distribution parameters obtained in the superpredictive synthesis step.

[0183] In the motion compensation step, based on the decoded motion features of the current octree level, the reference features of the current octree level, and the style conditions, the predicted features of the current octree level You can obtain.

[0184] Figure 12 illustrates an example of a motion compensation step.

[0185] The main decoding process can be called at each octree level of the slice after the learning-based motion decoding process is completed.

[0186] The main decoding process is performed in the learning-based geometric decoding unit of FIG. 4, and the unit may be a neural network shared with the geometric intra-decoding process.

[0187] Similar to the geometric intra-decoding process, the types of the main decoding process may include lossless-based decoding and loss-based decoding.

[0188] Compared to the geometric intra-decoding process, the difference in the main decoding process may be that it uses predictive features generated from the learning-based motion decoding process instead of dummy features.

[0189] The loss-based decoding process may include 1) an ancestor occupancy analysis step, 2) a feature decoding step, and 3) a point synthesis step.

[0190] The lossless-based decoding process may include 1) an ancestor occupancy analysis step, 2) a feature decoding step, and 4) an octree decoding step.

[0191] That is, in the loss-based decoding and lossless-based decoding processes, 1) the ancestor occupancy analysis step and 2) the feature decoding step may be common processes.

[0192] 1) Ancestor Occupancy Analysis Stage

[0193] In the ancestor occupancy analysis step, if occupancy information of the ancestor octree levels (i-1, i-2, i-3, i-4) of the current octree level is available, ancestor features are based on that information. You can obtain.

[0194] In the ancestor occupancy analysis step, if occupancy information of the ancestor octree levels (i-1, i-2, i-3, i-4) of the current octree level is available, ancestor features are based on that information. You can obtain.

[0195] The ancestor occupancy analysis step may include an ancestor feature initialization step, a feature mixing step, and an adaptive affine step. The specific details of each step of the ancestor occupancy analysis step are as previously described and are omitted below.

[0196] 2) Feature decoding step

[0197] Bitstream featuring the current octree level , in the feature decoding stage, the ancestor features of the current octree level , predictive features of the current octree level , decoded residual features of the parent octree level , and the occupancy map of the parent octree level Based on, the decoded features of the current octree level and decoded residual features of the current octree level You can obtain.

[0198] 3) Point composite step

[0199] In the point synthesis step, the decoded features of the current octree level Based on the occupancy map of the parent octree level, the (lossy) decrypted occupancy map of the current octree level can be obtained.

[0200] The point synthesis step can be performed for each octree level configured for loss-based decoding.

[0201] The point synthesis step can be performed in different ways based on a flag indicating whether a MultiLayer Perceptron (MLP) is used.

[0202] For example, if a flag indicating whether to use an MLP has a first value, a sparse 3D convolution-based point synthesis method may be performed, and if a flag indicating whether to use an MLP has a second value, an MLP-based point synthesis method may be performed. The specific details of the point synthesis step are as described above and are omitted below.

[0203] 4) Octree decoding step

[0204] In the octree decoding stage, the decoded features of the current octree level Based on the occupancy map of the parent octree level, the (lossless) decrypted occupancy map of the current octree level can be obtained.

[0205] The octree decoding step can be performed for each octree level configured for lossless-based decoding.

[0206] The mode of the octree decoding step can be determined based on bit depth information. The mode of the octree decoding step may include a current level autoregressive mode and a parent level autoregressive mode.

[0207] The specific details of the octree decoding step are as previously described and will be omitted below.

[0208] Region Adaptive Hierarchical Transform (RAHT)-based attribute decoding may include intra-point cloud attribute decoding and inter-point cloud attribute decoding.

[0209] In the case of intra-point cloud attribute decoding, the decoded point cloud attribute can be obtained from a RAHT-based Attribute Decoding Unit based on the attribute bitstream and the decoded point cloud geometry.

[0210] In the case of inter-point cloud attribute decoding, attribute predictor information can be generated in an attribute predictor generation unit based on the decoded point cloud geometry and attribute reference frame. For example, if the attribute is color, attribute predictor information can be generated in a recoloring unit. The attribute predictor information is shared as an additional input to a RAHT-based attribute decoding unit of intra-decoding, and the decoded point cloud attribute can be obtained in the RAHT-based attribute decoding unit of intra-decoding.

[0211] After the point cloud geometry slice is decoded, whether to perform an inverse scaling process can be determined based on a coordinate transformation flag indicating that the point cloud geometry is transformed from the world coordinate system to the coding coordinate system, and scaling ratio information indicating the amount of scaling from the shifted coding coordinate system to the scaled coding coordinate system.

[0212] In one embodiment, when the coordinate transformation flag is 1 and the scaling amount of the scaling ratio information is less than 1, an inverse scaling process can be performed.

[0213] In the inverse scaling process, the decoded point cloud geometry, which is a set of 3D point locations in a coded coordinate system Based on the scaling coefficient specified by the , and scaling ratio information, an unscaled point cloud geometry, which is a set of 3D point locations in a transformed coding coordinate system, can be obtained.

[0214] In addition, based on reverse scaling type information indicating the type of reverse scaling process, the reverse scaling process can be performed by a simple reverse scaling process or an enhanced reverse scaling process.

[0215] The inverse transform process can be called when the point cloud geometric slice is decoded and the coordinate transformation flag has a value equal to 1. Additionally, if the scaling ratio information indicates a value less than 1, the inverse transform process can be called additionally after the inverse scaling process is completed.

[0216] The inverse transformation process can move the point cloud geometry of the shifted coding coordinate system back to the world coordinate system.

[0217] The assembly process is called after all point cloud slices are decoded, where information related to the assembly process indicates that the point cloud frame is divided into slices and coded individually, and the point cloud frame can be obtained by assembling all decoded point cloud slices.

[0218] Figure 13 illustrates an example of the overall encoding process.

[0219] The encoding process of the present disclosure may include learning-based point cloud geometric encoding and Region Adaptive Hierarchical Transform (RAHT)-based attribute encoding.

[0220] Both geometry encoding and attribute encoding can have intra mode and inter mode as encoding modes.

[0221] Geometric coding is a neural network-based coding method in which neural network weights can be determined before the geometric coding process is executed. Model identifier information representing the neural network weights can be encoded directly or indirectly as a bitstream. Alternatively, the neural network weights can be pre-defined values ​​between the encoder and the decoder.

[0222] For octree level i=1, eight binary occupancy values ​​can be encoded in a raw coding unit consisting of a single byte. The eight voxels can be arranged according to raster scanning order. The bits of the byte in the raw coding unit can be indexed from the Most Significant Bit (MSB) to the Least Significant Bit (LSB). Each bit can represent the occupancy status of the corresponding voxel. At this level, the residual feature, Tensor<0, 8>, can be set as a constant dummy feature where all elements are 1.

[0223] In the case of intra-point cloud geometry encoding, the geometry of the point cloud encoded in the Learning-based Geometry Encoding Unit can be constructed and encoded into the bitstream. Intra-point cloud geometry encoding can be executed for each octree level, starting from octree level i=2.

[0224] In the case of inter-point cloud geometry encoding, geometry predictor information is generated in the Learning-based Motion Decoding Unit, and a motion bitstream can be encoded based on the geometry predictor information and a reference frame. The geometry predictor information is shared as an additional input with the Learning-based Geometry Encoding Unit of intra-coding, allowing the geometry of the encoded point cloud to be constructed in the Learning-based Geometry Encoding Unit of intra-coding. Inter-point cloud geometry encoding can be executed for each octree level, starting from octree level i=2.

[0225] In the case of intra-point cloud attribute encoding, the point cloud attributes encoded in the RAHT-based Attribute Decoding Unit are configured, and the attribute bitstream can be encoded based on the encoded point cloud attributes and the encoded point cloud geometry.

[0226] In the case of inter point cloud attribute encoding, attribute predictor information can be generated in a recoloring unit based on the encoded point cloud geometry and attribute reference frame. The attribute predictor information is shared as an additional input to a RAHT-based attribute encoding unit of intra encoding, and the encoded point cloud attributes are constructed in the RAHT-based attribute encoding unit of intra encoding, and an attribute bitstream can be encoded based on the encoded point cloud attributes.

[0227] The specific details of the encoding process can correspond to the specific details of the decoding process.

[0228] The (geometric) transformation process can be called as a preprocessing step. The transformation process can convert a point cloud in the world coordinate system into a transformed coding coordinate system. Subsequently, the decoder can perform a process to convert the output back to the world coordinate system.

[0229] After the transformation process is completed, whether to perform a (geometric) scaling process can be determined based on a coordinate transformation flag indicating that the point cloud geometry is transformed from the world coordinate system to the coding coordinate system, and scaling ratio information indicating the amount of scaling from the transformed (shifted) coding coordinate system to the scaled coding coordinate system.

[0230] In one embodiment, a scaling process can be performed when the coordinate transformation flag is 1 and the scaling amount of the scaling ratio information is less than 1.

[0231] In the scaling process, point cloud geometry, which is a set of 3D point locations in the transformed coding coordinate system, Based on the scaling coefficient specified by the , and scaling ratio information, the scaled point cloud geometry of the transformed coding coordinate system can be obtained.

[0232] The point cloud geometry intra-coding process (geometry intra-coding process) can be called at each octree level when the current point cloud slice is determined to be in geometry intra mode (e.g., when the unit type is determined to be Geometry Intra-Slice).

[0233] Additionally, the geometry intra-coding process may be called for each octree level if the current octree level is greater than the threshold octree level, even if the current point cloud slice is determined to be in geometry inter-mode (e.g., even if the unit type is determined to be Geometry Inter-Slice). Here, the threshold octree level can be encoded directly or indirectly as a bitstream.

[0234] The geometric intra-coding process can be performed in the learning-based geometric encoding unit of Fig. 13.

[0235] The unit types of the present disclosure may include types of Geometry Intra-Slice, Geometry Inter-Slice, Attribute Intra-Slice, and Attribute Inter-Slice.

[0236] Types of geometric intra-coding processes may include lossless-based encoding and lossy-based encoding.

[0237] Figure 14 illustrates examples of lossless-based encoding and lossy-based encoding of the geometric encoding process.

[0238] The loss-based encoding process may include 1) an ancestor occupancy analysis step, 2) a feature extraction step, 3) a feature encoding step, and 4) a point synthesis step.

[0239] The lossless-based encoding process may include 1) an ancestor occupancy analysis step, 2) a feature extraction step, 3) a feature encoding step, and 5) an octree encoding step.

[0240] That is, in loss-based encoding and lossless-based encoding processes, 1) the ancestor occupancy analysis step, 2) the feature extraction step, and 3) the feature encoding step may be common processes.

[0241] 1) Ancestor Occupancy Analysis Stage

[0242] In the ancestor occupancy analysis step, if occupancy information of the ancestor octree levels (i-1, i-2, i-3, i-4) of the current octree level is available, ancestor features are based on that information. You can obtain.

[0243] The ancestor occupancy analysis step may be an initialization step performed before the 3) feature encoding step.

[0244] The ancestor occupancy analysis step may include an ancestor feature initialization step, a feature mixing step, and an adaptive affine step. The specific details of each step are identical to those in the geometric decoding process and are therefore omitted below.

[0245] 2) Feature extraction step

[0246] Voxel occupancy map of the current octree level of the input point cloud and style conditions f style Based on this, features representing local geometric information of the current octree level of the input point cloud You can obtain.

[0247] The feature extraction step can be performed in different ways based on a flag indicating whether a MultiLayer Perceptron (MLP) is used.

[0248] For example, when a flag indicating whether to use an MLP has a first value, the voxel occupancy map of the parent octree level of the input point cloud It can be provided as additional input.

[0249] For example, if a flag indicating whether to use an MLP has a second value, a sparse 3D convolution-based feature extraction method can be performed.

[0250] 3) Feature encoding step

[0251] In the feature encoding stage, features representing geometric information for the current octree level , ancestor characteristics of the current octree level , constant dummy prediction feature F pred , decoded residual features of the parent octree level Based on, the characteristic bitstream for the current octree level , current octree level decrypted features and decoded residual features of the current octree level You can obtain.

[0252] Constant dummy prediction feature F pred All elements of can be equal to 1. Constant dummy prediction features can be used to match geometric feature encoding for intra-coding with inter-coding.

[0253] Figure 15 illustrates an example of a feature encoding step.

[0254] The feature encoding step may include a superprescription synthesis step, a conditional encoding step, an arithmetic encoding step, and a conditional decoding step.

[0255] The superpredictive synthesis step can synthesize feature variance parameters based on decoded residual features of the parent octree level.

[0256] The conditional encoding step can conditionally encode the features of the current octree level based on the residual features of the current octree level, using the ancestor features and constant dummy prediction features of the current octree level as conditions.

[0257] The arithmetic encoding step can obtain the decoded residual features and feature bitstream of the current octree level based on the feature dispersion parameters obtained in the superprescription synthesis step and the residual features of the current octree level.

[0258] The conditional decoding step can conditionally decode the decoded features of the current octree level based on the decoded residual features of the current octree level, using the ancestor features and constant dummy prediction features of the current octree level as conditions.

[0259] 4) Point composite step

[0260] In the point synthesis step, the decoded features of the current octree level Based on the occupancy map of the parent octree level, the (lossy) decrypted occupancy map of the current octree level can be obtained.

[0261] The point synthesis step can be performed in different ways depending on whether a MultiLayer Perceptron (MLP) is used.

[0262] For example, when an MLP is used, a sparse 3D convolution-based point synthesis method is performed, and when an MLP is not used, an MLP-based point synthesis method can be performed.

[0263] Information indicating whether the MLP is used can be encoded in the bitstream.

[0264] The specific details of the sparse 3D convolution-based point synthesis method and the MLP-based point synthesis method are as described above and will be omitted below.

[0265] 5) Octree encoding step

[0266] In the octree encoding stage, the decoded features of the current octree level , based on the occupancy map of the parent octree level and the occupancy map of the current octree level, the occupied bitstream for the current octree level You can obtain.

[0267] The octree encoding step can be performed for each octree level configured for lossless-based encoding.

[0268] The mode of the octree encoding stage can be determined based on bit depth information. The mode of the octree encoding stage may include a current level autoregressive mode and a parent level autoregressive mode.

[0269] Depending on the mode, intermediate processes of the octree encoding step can be performed at the resolution of the current octree level or the parent octree level.

[0270] The octree encoding step may include an initialization step, a feature aggregation step, and a occupancy encoding step.

[0271] In the initialization phase, depending on the mode of the octree encoding phase, the occupancy map of the parent octree level is expanded or upsampled to the occupancy map of the current octree level to obtain the initial occupancy map of the current octree level. It can generate.

[0272] In the case of the current level autoregressive mode, during the initialization phase, the occupancy map of the parent octree level is upsampled to the occupancy map of the current octree level to obtain the initial occupancy map of the current octree level. , Tensor<i, 1> You can obtain. The initial occupancy map of the current octree level has 8 × voxelCount (number of occupied voxels in the parent octree level) points, and all child voxels of an occupied parent voxel can be set to 0.

[0273] In the case of parent-level autoregressive mode, during the initialization phase, the occupancy map of the parent octree level is extended to the occupancy map of the current octree level, and the initial occupancy map of the current octree level , Tensor<i-1, 8> You can obtain. The initialization step initializes all channels to 0. Repeat the channel 8 times You can generate 8 child voxels. The 8 child voxels are mapped to 8 channels indexed from 0 to 7, and the order can be (0,0,0), (1,0,0), (0,1,0), (1,1,0), (0,0,1), (1,0,1), (0,1,1), (1,1,1).

[0274] In the feature aggregation stage, the decoded features of the current octree level Improves aggregated features according to mode You can obtain.

[0275] In the case of the current level autoregressive mode, the decoded features of the current octree level are used in the feature aggregation step. Improves, aggregates the improved features, and upsamples the aggregated features You can obtain.

[0276] In the case of parent-level autoregressive mode, the decoded features of the current octree level are used in the feature aggregation step. Aggregated features obtained by aggregating without changing the resolution (having the resolution of the parent octree level). You can obtain.

[0277] Figure 16 illustrates an example of a possession encoding step.

[0278] The occupancy encoding step can be called repeatedly 8 times to encode the occupancy map of octree level i. k, representing the numbering of the repetitions, can range from 1 to 8. The repetitions for 8 octants can follow the order of (0,0,0), (1,0,0), (0,1,0), (1,1,0), (0,0,1), (1,0,1), (0,1,1), (1,1,1).

[0279] The occupy encoding step for the k-th iteration can be performed by i) an occupancy buffer, ii) a sibling feature initializer, iii) a sibling feature mixing unit, iv) a probability estimator, v) an occupancy masking unit, vi) an occupancy serializer, and vii) an arithmetic encoding unit.

[0280] i) Occupied buffer

[0281] The occupancy buffer contains all sibling occupancy maps decrypted in the previous iteration of the current octree level. (from 1 to k-1) can be stored and provided so that it can be referenced in the next decoding iteration. Additionally, a mask, which is a decoding mask indicating whether a child voxel is decoded, can be managed. Also, for the first iteration (k=1), the occupied buffer can be initialized.

[0282] ii) Sibling characteristic initialization section

[0283] In the sibling feature initialization section, all sibling occupancy maps decrypted in the previous iteration Initialize features based on to initial sibling features You can obtain.

[0284] In the case of the current level autoregressive mode, the sibling feature initializer performs initialization using a 5-dimensional vector to initialize the sibling features It is possible to obtain. In this case, 1 dimension represents a scalar value for the octree level, 3 dimensions represent a scalar value for the 3D position, and 1 dimension represents a binary value of the decoding mask map.

[0285] In the case of parent-level autoregressive mode, the input in the sibling feature initialization section Initialize identically to the initial sibling characteristics You can obtain.

[0286] iii) Brothers' Characteristics Mixing Department

[0287] In the sibling characteristic mixing section, the aggregated characteristics Characteristics of early siblings Mixed characteristics by mixing You can obtain.

[0288] The mixing process of the sibling feature mixing section may vary depending on whether the mode is the current level autoregressive mode or the parent level autoregressive mode.

[0289] iv) Probability Estimator

[0290] In the probability estimation unit, mixed features Based on, the occupancy probability of voxels at the current octree level for each occupied parent voxel It can predict.

[0291] In the case of the current level autoregressive mode, the probability estimation unit uses a multi-perceptron (MLP) network to estimate the occupancy probabilities of voxels at the current octree level having one channel. It can be predicted. In this case, the channel size of the MLP layer used may be determined differently by bit depth-related information.

[0292] In the case of parent-level autoregressive mode, the probability estimator uses convolution layers, ReLU layers, and Inception ResNet (IRN) layers to estimate the occupancy probabilities of the voxels at the current octree level. It can predict.

[0293] v) Occupancy masking part

[0294] Upsampled occupancy map of the current octree level based on the parent octree level Based on, the k-th occupying map of the current iteration You can obtain.

[0295] vi) Possession serialization part

[0296] In the possession serialization section, the possession map of the current iteration Based on, binary sequence You can obtain.

[0297] vii) Arithmetic encoding unit

[0298] In the arithmetic encoding section, the binary value uses the predicted probability for the current octant of the current octree level. A sequence of can be encoded to generate an occupied bitstream.

[0299] The point cloud geometry inter-coding process can be called for each octree level when the current point cloud slice is determined to be in geometry inter-mode (i.e., the unit type is determined to be Geometry Inter-Slice) and the current octree level is less than or equal to a critical octree level.

[0300] The point cloud geometry inter-coding process can be performed by a combination of the learning-based geometry encoding unit, the learning-based motion encoding unit, and the reference frame unit of FIG. 13.

[0301] The point cloud geometry intercoding process may include a motion coding process and a main coding process.

[0302] In the motion encoding process, the motion between the reference frame and the current frame and the reference frame can be estimated, and a predictor for the current frame can be obtained. This may correspond to the learning-based motion encoding unit and the reference frame unit of FIG. 13.

[0303] In the main encoding process, geometric encoding can be performed based on a predictor for the current frame. This may correspond to the learning-based geometric encoding unit of FIG. 13.

[0304] The global motion evaluation and compensation process can be called as a preprocessing process at every octree level before calling the inter-coding process.

[0305] The inputs and outputs of the global motion evaluation and compensation process can belong to a transformed coding coordinate system.

[0306] 1) First, the current point cloud frame and reference point cloud frame The coordinates of each point in can be transformed based on the point cloud bounding box center (c, c, c). The above transformation is and It can be performed by.

[0307] The position of the bounding box center in the transformed coding coordinate system is It can be defined as follows.

[0308] 2) Next, the global motion represented by the rotation matrix R and the translation vector T is determined, and the reference point cloud frame using the rotation matrix R and translation vector T It can be converted as follows.

[0309] 3) Finally, reference point cloud frame A transformation can be applied to move it back to the transformed coding coordinate system, thereby obtaining a global motion-compensated reference frame. The above transformation is It can be performed by.

[0310] The learning-based motion encoding process performed in the learning-based motion encoding unit can be called for each octree level for a slice at the start of the point cloud geometry inter-decoding process.

[0311] The learning-based motion encoding process can generate predicted features to support the main encoding process.

[0312] FIG. 17 illustrates an example of a learning-based motion encoding process.

[0313] The learning-based motion encoding process may include a feature extraction step, a reference feature extraction step, and an inter-predictor extraction step.

[0314] In the feature extraction step, features of the current octree level can be obtained by utilizing style conditions and the occupancy map of the current octree level. Additionally, if a MultiLayer Perceptron (MLP) is used, features of the current octree level can be obtained by additionally utilizing the current occupancy map of the previous octree level.

[0315] In the reference feature derivation step, reference features of the current octree level can be obtained based on style conditions and the reference occupancy map of the reference point cloud for the current octree level. Additionally, when a MultiLayer Perceptron (MLP) is used, reference features of the current octree level can be obtained by additionally utilizing the reference occupancy map of the reference point cloud of the previous octree level.

[0316] FIG. 18 illustrates an example of the inter-predictor derivation step.

[0317] The inter-predictor derivation step may include a motion feature buffer management step, a super-predicted synthesis step, a motion evaluation step, an arithmetic encoding step, and a motion compensation step.

[0318] The motion feature buffer management step can store and provide decoded motion features. For example, the motion feature buffer management process can provide the decoded motion features of the parent octree level to the superpre-synthesis step for the current octree level.

[0319] The superpredictive synthesis step can synthesize feature variance parameters using motion features of the parent octree level. The superpredictive synthesis step can operate identically to the previously described superpredictive synthesis step. However, the input can be the decoded motion features of the parent octree level instead of the decoded residual features of the parent octree level.

[0320] The motion evaluation step can obtain motion features of the current octree level based on the features of the current point cloud of the current octree level, the reference features of the reference point cloud of the current octree level, and style conditions.

[0321] The arithmetic encoding step can operate in the same manner as the previously described arithmetic encoding step. However, the residual features of the input current octree level can be replaced with the motion features of the current octree level. That is, the arithmetic encoding step can obtain a motion bitstream by arithmetic encoding the motion features of the current octree level based on the feature distribution parameters obtained in the superpredictive synthesis step.

[0322] In the motion compensation step, based on the decoded motion features of the current octree level, the reference features of the current octree level, and the style conditions, the predicted features of the current octree level You can obtain.

[0323] The main encoding process can be called at each octree level of the slice after the learning-based motion encoding process is completed.

[0324] The main encoding process is performed in the learning-based geometric encoding unit of FIG. 13, and the unit may be a neural network shared with the geometric intra-coding process.

[0325] Similar to the geometric intra-coding process, the types of main coding processes may include lossless-based coding and lossy-based coding.

[0326] Compared to the geometric intra-coding process, the difference in the main coding process may be that it uses predicted features generated from the learning-based motion coding process instead of dummy features.

[0327] The loss-based encoding process may include 1) an ancestor occupancy analysis step, 2) a feature extraction step, 3) a feature encoding step, and 4) a point synthesis step.

[0328] The lossless-based encoding process may include 1) an ancestor occupancy analysis step, 2) a feature extraction step, 3) a feature encoding step, and 5) an octree encoding step.

[0329] That is, in loss-based encoding and lossless-based encoding processes, 1) the ancestor occupancy analysis step, 2) the feature extraction step, and 3) the feature encoding step may be common processes.

[0330] 1) Ancestor Occupancy Analysis Stage

[0331] In the ancestor occupancy analysis step, if occupancy information of the ancestor octree levels (i-1, i-2, i-3, i-4) of the current octree level is available, ancestor features are based on that information. You can obtain.

[0332] The ancestor occupancy analysis step may be an initialization step performed before the 3) feature encoding step.

[0333] The ancestor occupancy analysis step may include an ancestor feature initialization step, a feature mixing step, and an adaptive affine step. The specific details of each step are identical to those in the geometric decoding process and are therefore omitted below.

[0334] 2) Feature extraction step

[0335] Voxel occupancy map of the current octree level of the input point cloud and style conditions f style Based on this, features representing local geometric information of the current octree level of the input point cloud You can obtain.

[0336] The feature extraction step can be performed in different ways depending on whether a MultiLayer Perceptron (MLP) is used.

[0337] For example, when an MLP is used, the voxel occupancy map of the parent octree level of the input point cloud It can be additionally provided as input.

[0338] For example, if an MLP is not used, a sparse 3D convolution-based feature extraction method can be performed.

[0339] 3) Feature encoding step

[0340] In the feature encoding stage, features representing geometric information for the current octree level , ancestor characteristics of the current octree level , predictive features of the current octree level , decoded residual features of the parent octree level Based on, the characteristic bitstream for the current octree level , current octree level decrypted features and decoded residual features of the current octree level You can obtain.

[0341] 4) Point composite step

[0342] In the point synthesis step, the decoded features of the current octree level Based on the occupancy map of the parent octree level, the (lossy) decrypted occupancy map of the current octree level can be obtained.

[0343] The point synthesis step can be performed in different ways depending on whether a MultiLayer Perceptron (MLP) is used.

[0344] For example, when an MLP is used, a sparse 3D convolution-based point synthesis method is performed, and when an MLP is not used, an MLP-based point synthesis method may be performed. The specific details of each method have been described above and are omitted below.

[0345] 5) Octree encoding step

[0346] In the octree encoding stage, the decoded features of the current octree level , based on the occupancy map of the parent octree level and the occupancy map of the current octree level, the occupied bitstream for the current octree level You can obtain.

[0347] The octree encoding step can be performed for each octree level configured for lossless-based encoding.

[0348] The mode of the octree encoding stage can be determined based on bit depth information. The mode of the octree encoding stage may include a current level autoregressive mode and a parent level autoregressive mode.

[0349] Depending on the mode, intermediate processes of the octree encoding step can be performed at the resolution of the current octree level or the parent octree level.

[0350] The octree encoding step may include an initialization step, a feature aggregation step, and a occupancy encoding step.

[0351] The specific details of the octree encoding step are as previously described and will be omitted below.

[0352] Region Adaptive Hierarchical Transform (RAHT)-based attribute encoding may include intra-point cloud attribute encoding and inter-point cloud attribute encoding.

[0353] In the case of intra-point cloud attribute encoding, an attribute bitstream can be encoded in a RAHT-based Attribute Encoding Unit based on input point cloud attributes and encoded point cloud geometry.

[0354] In the case of inter-point cloud attribute encoding, attribute predictor information can be generated in an attribute predictor generation unit based on the restored point cloud geometry and attribute reference frame. For example, if the attribute is color, attribute predictor information can be generated in a recoloring unit. The attribute predictor information is shared as an additional input to an intra-coding RAHT-based attribute encoding unit, and an attribute bitstream can be obtained in the intra-coding RAHT-based attribute encoding unit.

[0355] The layers of the present disclosure can be represented as a network.

[0356] FIG. 19 illustrates a flowchart of a multidimensional feature extraction method for DNN-based point cloud encoding according to the present disclosure.

[0357] The multidimensional feature extraction method for DNN-based point cloud coding of the present disclosure may be composed of steps such as those shown in FIG. 19.

[0358] Specifically, a multidimensional feature extraction method for DNN-based point cloud encoding may include at least one of a point cloud input step (1901), a DNN (Deep Neural Network)-based point cloud encoding step (1902), a DNN-based point cloud decoding step (1903), or a reconstructed point cloud output step (1904).

[0359] FIG. 20 illustrates an embodiment of a device in which a Deep Neural Network (DNN)-based point cloud encoding step is performed.

[0360] A device (DNN-based point cloud encoder, 2000) in which a Deep Neural Network (DNN)-based point cloud encoding step (1902) of one embodiment of FIG. 20 is performed may include at least one of a downsampler (2001), a feature processing unit (2002), a feature encoding (2003), a feature decoding (2004), a probability estimation (2005), or a residual information encoding (2006).

[0361] Here, feature encoding (2003), feature decoding (2004), probability estimation (2005), and residual information encoding (2006) are sub-devices of a DNN-based point cloud encoder and can also be described as a feature encoding unit, a feature decoding unit, a probability estimation unit, and a residual information encoding unit, respectively.

[0362] FIGS. 21 to 23 illustrate an embodiment of a downsampler (2001).

[0363] The downsampler (2001) in the device in which the Deep Neural Network (DNN)-based point cloud encoding step (1902) of the present disclosure is performed may include at least one of a first spatial axis-based feature extraction layer (2101), a second spatial axis-based feature extraction layer (2102), a third spatial axis-based feature extraction layer (2103), or a combination layer (2104).

[0364] Here, the number of each feature extraction layer (2101 to 2103) and combination layer (2104) included in the downsampler (2001) may be a natural number greater than or equal to 0. For example, in the case of the embodiment of FIG. 23, the number of each feature extraction layer (2101 to 2103) and combination layer (2104) may be 2.

[0365] In one embodiment, referring to FIG. 21, the downsampler (2001) may be in the form of a first spatial axis-based feature extraction layer (2101), a second spatial axis-based feature extraction layer (2102), and a third spatial axis-based feature extraction layer (2103) connected in series.

[0366] In one embodiment, referring to FIG. 22, the downsampler (2001) may be in the form in which a first spatial axis-based feature extraction layer (2101), a second spatial axis-based feature extraction layer (2102), and a third spatial axis-based feature extraction layer (2103) are connected in parallel through a combination layer (2104).

[0367] In one embodiment, referring to FIG. 23, the downsampler (2001) may be in the form of a first downsampler in which a first spatial axis-based feature extraction layer (2101), a second spatial axis-based feature extraction layer (2102), and a third spatial axis-based feature extraction layer (2103) are connected in series, and a second downsampler in which the first spatial axis-based feature extraction layer (2101), a second spatial axis-based feature extraction layer (2102), and a third spatial axis-based feature extraction layer (2103) are connected in parallel through a combining layer (2104).

[0368] FIG. 24 illustrates an embodiment of a device in which a DNN-based point cloud decoding step is performed.

[0369] A device (DNN-based point cloud decoder, 2400) in which the DNN-based point cloud decoding step (1903) of one embodiment of FIG. 24 is performed may include at least one of an upsampler (2401), a feature processing unit (2402), a feature decoding (2403), a probability estimation (2404), or a residual information decoding (2405).

[0370] Here, feature decoding (2403), probability estimation (2404), and residual information decoding (2405) are sub-devices of a DNN-based point cloud encoder and can also be described as a feature decoding unit, a probability estimation unit, and a residual information decoding unit, respectively.

[0371] FIGS. 25 to 27 illustrate an embodiment of an upsampler (2401).

[0372] The downsampler (2401) in the device in which the Deep Neural Network (DNN)-based point cloud decoding step (1903) of the present disclosure is performed may include at least one of a first spatial axis-based feature generation layer (2501), a second spatial axis-based feature generation layer (2502), a third spatial axis-based feature generation layer (2503), or a combination layer (2504).

[0373] Here, the number of each feature extraction layer (2501 to 2503) and combination layer (2504) included in the upsampler (2401) may be a natural number greater than or equal to 0. For example, in the case of the embodiment of FIG. 23, the number of each feature extraction layer (2501 to 2503) and combination layer (2504) may be 2.

[0374] In one embodiment, referring to FIG. 25, the upsampler (2401) may be in the form of a first spatial axis-based feature extraction layer (2501), a second spatial axis-based feature extraction layer (2502), and a third spatial axis-based feature extraction layer (2503) connected in series.

[0375] In one embodiment, referring to FIG. 26, the upsampler (2401) may be in the form in which a first spatial axis-based feature extraction layer (2501), a second spatial axis-based feature extraction layer (2502), and a third spatial axis-based feature extraction layer (2503) are connected in parallel through a combination layer (2504).

[0376] In one embodiment, referring to FIG. 27, the upsampler (2401) may be in the form of a first upsampler in which a first spatial axis-based feature extraction layer (2501), a second spatial axis-based feature extraction layer (2502), and a third spatial axis-based feature extraction layer (2503) are connected in series, and a second upsampler in which the first spatial axis-based feature extraction layer (2501), a second spatial axis-based feature extraction layer (2502), and a third spatial axis-based feature extraction layer (2503) are connected in parallel through a combining layer (2504).

[0377] Hereinafter, each step and each unit (or part) of the multidimensional feature extraction method and apparatus for DNN-based point cloud coding of the present disclosure will be described in detail.

[0378] FIG. 28 illustrates one embodiment of a point cloud input to the present disclosure.

[0379] In the point cloud input step (1901), a point cloud image representing a three-dimensional image with points having three-dimensional coordinate values ​​(orthogonal coordinate system, spherical coordinate system, etc.) and attribute values ​​corresponding to each point (color, transparency, reflectance, normal vector, spherical harmonic function, etc.) may be input.

[0380] In the DNN-based point cloud encoding step (1902), structural correlations along each spatial axis of the input point cloud can be learned, and features can be extracted through multidimensional convolution operations. Based on the extracted features, occupancy information and attribute information can be efficiently compressed to generate a bitstream.

[0381] A device (DNN-based point cloud encoder) in which a DNN-based point cloud encoding step is performed may be composed of at least one of a downsampler (2001), a feature processing unit (2002), a feature encoding (2003), a feature decoding (2004), a probability estimation (2005), or a residual information encoding (2006). Although FIG. 20 depicts three feature extractions and dimensionality reduction (downsampling) for an input point cloud, one or more structures based on the structure may be repeated continuously.

[0382] In one embodiment, a DNN-based point cloud encoder may include at least one structure comprising three downsamplers as shown in FIG. 20.

[0383] To transmit data characteristics processed by the encoder (DNN-based point cloud encoder) to the decoder (DNN-based point cloud decoder), information regarding the number of iterations can be additionally transmitted to the decoder via a bitstream. The number of iterations may be specified as the number of network iterations, based on the number of down- or up-sampling iterations, or based on the bit depth or precision of the input point cloud.

[0384] According to one embodiment of the present disclosure, the following parameters may be defined to transmit the number of repetitions through a bitstream. The network structure and a single network may be predefined, and only the single network or network structure information may exist.

[0385] nn_structure_id is a structure identifier for repeated networks, a parameter for identifying network structures in a predefined table, and can be stored as an integer value.

[0386] nn_structure_iterations[nn_structure_id] can store the number of iterations of the network structure identified by nn_structure_id as an integer value.

[0387] nn_structure_Bit-depth[nn_structure_id] represents the bit depth of the input / output network structure identified by nn_structure_id, the bit depth of the output point cloud point, and may also be the difference from the sub-network to the output, and may be stored as an integer value.

[0388] nn_structure_precision[nn_structure_id] represents the precision of the output point cloud points of the network structure identified by nn_structure_id, and may also specify the difference from the sub-network to the output, and can be stored as a normalized integer or floating-point value.

[0389] nn_id[nn_structure_id] is an identifier for a single network within a network structure, a parameter that identifies a single network within a network structure in a predefined table, and can be stored as an integer value.

[0390] nn_iterations[nn_structure_id][nn_id] can store the number of iterations of the network identified by nn_id as an integer value.

[0391] nn_Bit-depth[nn_structure_id][nn_id] represents the bit depth of the input / output network identified by nn_id, the bit depth of the output point cloud point, and may also indicate the difference from the sub-network to the output, and can be stored as an integer value.

[0392] nn_precision[nn_structure_id][nn_id] represents the precision of the output point cloud points of the network identified by nn_id, and may also indicate the difference from the sub-network to the output, and can be stored as a normalized integer or floating-point value.

[0393] The downsampler (2001) of the present disclosure may have three structures.

[0394] Specifically, there are serial extraction structures (Fig. 21) and parallel extraction structures (Fig. 22) for multidimensional feature extraction methods, and the serial extraction structure of Fig. 21 may be composed of a first spatial axis-based feature extraction layer (2101), a second spatial axis-based feature extraction layer (2102), and a third spatial axis-based feature extraction layer (2103). The presented structures may be performed as iterative or recursive structures within a single downsampler (2001).

[0395] Information about the structure of the spatial axis-based feature extraction network used (=network structure information) can be additionally transmitted to the decoder via a bitstream. Network structure information such as the network type (convolution, MLP, etc.), kernel size, input channel size, output channel size, and whether bias values ​​are used can be transmitted to the decoder. In addition, if the same structure is repeated, the number of repetitions can be specified and the network identifier can be transmitted. This is because a specific structure of the encoder may be used within the decoder depending on the configuration of the decoder.

[0396] In one embodiment, the types of downsamplers included in a single structure may differ from each other. For example, in the case of FIG. 20, the three downsamplers may be of different types. Here, the meaning of different types may include cases where at least one of i) the structure of the downsampler (serial, parallel, serial-parallel) or ii) the structure information of the downsampler (network type (convolutional, MLP, etc.), kernel size, input channel size, output channel size, whether or not bias values ​​are used, etc.) is different.

[0397] In one embodiment, the following parameters can be defined to additionally transmit information about the structure of the spatial axis-based feature extraction network used via a bitstream.

[0398] multi_dimension_ext_flag is an identifier for whether to use a spatial axis-based feature extraction network, and if it is 0, the lower parameters may not be transmitted.

[0399] mde_structure_id is an identifier for the spatial axis-based feature extraction network structure used; it is a parameter used to identify the network structure used in a predefined table and can be stored as an integer value.

[0400] mde_nn_id[mde_structure_id] is an identifier for the iterated spatial axis-based feature extraction network structure, a parameter used to identify a predefined table structure, and can be stored as an integer value.

[0401] mde_nn_type[mde_structure_id][mde_nn_id] is a parameter used to indicate the type of network used in the spatial axis-based feature extraction network identified by mde_nn_id. It is used to identify the type of network used, such as convolution, MLP, or transformer, which can be defined in a predefined table, and can be stored as an integer value.

[0402] mde_nn_iterations[mde_structure_id][mde_nn_id]: The number of iterations of the spatial axis-based feature extraction network structure identified by mde_nn_id can be stored as an integer value.

[0403] In one embodiment, if the structure of the spatial axis-based feature extraction network used and the spatial axis-based feature generation network of the decoder are not in reverse order or if the structure is optionally used, the structure information of the spatial axis-based feature generation network can be additionally transmitted to the decoder via a bitstream. Additionally, network structure information such as the network type (convolution, MLP, etc.), kernel size, input channel size, output channel size, and whether bias values ​​are used can be transmitted to the decoder, and if the same structure is repeated, the number of repetitions can be specified and the network identifier can be transmitted. To additionally transmit information about the structure of the spatial axis-based feature generation network used via a bitstream, the following parameters can be defined.

[0404] multi_dimension_gen_flag is an identifier for whether to use a spatial axis-based feature generation network, and if it is 0, the lower parameters may not be transmitted.

[0405] mdg_structure_id is an identifier for the spatial axis-based feature generation network structure used, a parameter for identifying the network structure used in a predefined table, and can be stored as an integer value.

[0406] mdg_nn_id[mdg_structure_id] is an identifier for the iterated spatial axis-based feature generation network structure, a parameter for identifying a predefined table structure, and can be stored as an integer value.

[0407] mdg_nn_type[mdg_structure_id][mdg_nn_id] is a parameter for indicating the type of network used in the spatial axis-based feature generation network identified by mdg_nn_id, and is a parameter for identifying the type of network used, such as convolution, MLP, transformer, etc., which can be defined in a predefined table, and can be stored as an integer value.

[0408] mdg_nn_iterations[mdg_structure_id][mdg_nn_id] can store the number of iterations of the spatial axis-based feature generation network structure identified by mdg_nn_id as an integer value.

[0409] The embodiment of FIG. 21 is an input tensor It may be designed to process feature extraction in each axis direction continuously.

[0410] The embodiment of FIG. 22 is also composed of a first spatial axis-based feature extraction layer (2101), a second spatial axis-based feature extraction layer (2102), and a third spatial axis-based feature extraction layer (2103), but unlike the embodiment of FIG. 21, the input tensor Each result can be simultaneously input to each spatial axis-based feature extraction layer and output through a combining layer (2104) which combines each output result.

[0411] At this time, the combining layer (2104) can combine tensors output from each spatial axis-based feature extraction layer. As an example of the combining layer (2104), there is a method of combining features corresponding to each coordinate or expanding the feature dimension and connecting them in the row direction, and there is a method of pruning points that do not match based on the coordinates of a specific tensor to expand the feature dimension and connect them.

[0412] Information regarding the coupling layer can be additionally transmitted to the decoder via a bitstream. This can be used in the decoder's coupling layer. Such information may not match the encoder's usage method.

[0413] concat_nn_id is an identifier for the coupling layer network, and can transmit an identifier for the type if different coupling methods are used for each individual coupling layer. If the same coupling method is used for all, a value of 0 can be used to indicate that the coupling method of all coupling layers is the same.

[0414] concat_type[concat_nn_id] is an identifier for the combination method, a parameter used to identify combination methods such as concatenation, addition, and subtraction that can be defined in a predefined table, and can be stored as an integer value.

[0415] The embodiment of FIG. 23 may be a case where a serial execution method and a parallel execution method are combined and executed.

[0416] All three structures of FIGS. 21 to 23 are input tensors It performs feature extraction from, in a dimensionality-reduced form It can be characterized by outputting.

[0417] Figure 29 illustrates coordinate systems.

[0418] FIG. 30 illustrates embodiments of dimension reduction and orientation.

[0419] The first spatial axis-based feature extraction layer (2101), the second spatial axis-based feature extraction layer (2102), and the third spatial axis-based feature extraction layer (2103) may represent a space of a generally known space (orthogonal coordinate system, spherical coordinate system, etc.), and may represent no dimensionality reduction, 1-dimensional reduction, 2-dimensional reduction, 3-dimensional reduction, etc. depending on each axis of the defined space.

[0420] N predefined spaces and M dimensionality reductions and orientations can be used, and the number of layers may vary depending on the characteristics of the point cloud. Since the extracted spatial and dimensional features are based on user definition, it is possible to configure more layers than specified or fewer layers.

[0421] In one embodiment, information regarding the spatial axis-based feature extraction layer can be additionally transmitted to the decoder via a bitstream. Information such as the identifier of the coordinate system, the dimension axis, the depth of the reduced dimension, the number of spatial axis-based feature extraction layers, the identifier of the spatial axis-based feature extraction layer, the number of iterations of the spatial axis-based feature extraction layer, and the structural form of the spatial axis-based feature extraction layer can be transmitted to the decoder. This is because a specific structure of the encoder may be used within the decoder depending on the configuration of the decoder.

[0422] mde_coordinate_system_id[mde_structure_id][mde_nn_id] is an identifier for the coordinate system used in the spatial axis-based feature extraction network. It is a parameter used to identify the type of coordinate system used, such as Cartesian coordinates, spherical coordinates, or cylindrical coordinates, which can be defined in a predefined table, and can be stored as an integer value.

[0423] mde_number_of_dimension[mde_structure_id][mde_nn_id] is a parameter used to identify the depth of the dimension that the network reduces in a spatial axis-based feature extraction network, and can be stored as an integer value.

[0424] mde_direction[mde_structure_id][mde_nn_id] is an identifier representing the dimensionality reduction direction of a spatial axis-based feature extraction network. It is a parameter used to identify the dimensionality reduction direction that can be predefined in a table based on the dimensionality reduction depth, and can be stored as an integer value. For example, in a Cartesian coordinate system, identifiers for linear axes such as the x-axis, y-axis, and z-axis can be stored for 1 dimension, and identifiers for planar axes such as xy, yz, and zx can be stored for 2 dimensions.

[0425] In one embodiment, the spatial axis-based feature extraction layer and the spatial axis-based feature generation layer may consist of a set of predefined structures. Optimal efficiency can be achieved by trying all predefined structures or by selectively using a suitable structure through parameters that allow for the calculation of compression efficiency. When trying all predefined structures, both encoding and decoding can be performed, and the encoding structure and the decoding structure can be selectively transmitted via the bitstream depending on the structure of the decoder. When using predefined structures, the structure is described through a predefined table, and only the identifier of the structure used can be transmitted to the decoder.

[0426] In one embodiment, when a predefined table is separated into multiple groups, a parent index for a group and a specific lower index for any one of the structures within the group can be transmitted to a decoder.

[0427] Figure 31 illustrates the feature processing unit of a DNN-based point cloud encoder.

[0428] The feature processing unit can process the features of the i-th layer output through the previous stage downsampler (2001). The main purpose of the feature processing unit may be to process a series of feature information for the purpose of predicting or calculating probabilities regarding the occupancy information or attribute information of the current layer.

[0429] Figure 32 illustrates the feature encoding (feature encoding section) of a DNN-based point cloud encoder.

[0430] In feature coding, the processed features of the current layer are entropy-coded, and can primarily be encoded through context-based arithmetic coding or hyperprior synthesis.

[0431] Figure 33 illustrates the feature decoding (feature decoding unit) of a DNN-based point cloud encoder.

[0432] In feature decoding, feature information that has undergone encoding / decoding at an upper layer (one level lower in resolution) is transmitted in order to provide a prior distribution for encoding feature information of the current encoding target in feature encoding (2003), which may be because the decoder is designed to utilize feature information that has already been decoded.

[0433] Figure 34 illustrates the probability estimation (probability estimation unit) of a DNN-based point cloud encoder.

[0434] In probability estimation, the restored feature information of the previous layer output from feature decoding (2004) is received as input, and the probability of the processed feature information belonging to the current layer can be estimated. The estimated probability distribution is input into the feature encoding (2003) step described above to perform arithmetic encoding.

[0435] Figure 35 illustrates the residual information encoding (residual information encoding section) of a DNN-based point cloud encoder.

[0436] In residual information coding, the location of the final geometric information is encoded, and as an example thereof, Octree Coding, Trisoup Coding, Predictive Geometry Coding, etc. can be performed.

[0437] In the DNN-based point cloud decoding step, the bitstream generated in the encoding step (1902) is received as input and decoded, and features are generated through multidimensional neural network operations learned in each spatial axis direction to restore a point cloud that is identical or similar to the input point cloud.

[0438] The DNN-based point cloud decoding step may consist of an upsampler (2401), a feature processing unit (2402), a feature decoding unit (2403), a probability estimation unit (2404), and a residual information decoding unit (2405). The figure below depicts three feature extractions and dimension expansions for an input point cloud, but one or more structures based on this structure may be repeated continuously.

[0439] In a decoder (DNN-based point cloud decoder), information regarding the number of iterations can be signaled via a bitstream. The number of iterations can be specified as the number of network iterations, based on the number of down- or up-sampling iterations, or based on the bit depth or precision of the input point cloud.

[0440] Information regarding the number of repetitions and related parameters has been described in the incubation stage as a specific example, so the explanation in the decoding stage is omitted.

[0441] Figure 36 illustrates the residual information decoding (residual information decoding section) of a DNN-based point cloud decoder.

[0442] In residual information decoding (2405), decoding can be performed on the residual information bitstream output from residual information encoding (2006).

[0443] Figure 37 illustrates the feature decoding (feature decoding unit) of a DNN-based point cloud decoder.

[0444] In feature decoding (2403), decoding can be performed on the feature bitstream output from feature encoding (2004). The initial input tensor of the decoder It can be constructed by mapping decoded features based on decoded residual information (geometric information).

[0445] Figure 38 illustrates the feature processing unit of a DNN-based point cloud decoder.

[0446] The feature processing unit (2402) can process the features of the i-th layer output through the previous stage upsampler (2401). The main purpose of the feature processing unit may be a series of feature information processing processes for predicting or calculating probabilities regarding the occupancy information or attribute information of the current layer.

[0447] Figure 39 illustrates the probability estimation (probability estimation unit) of a DNN-based point cloud decoder.

[0448] In the probability estimation (2404), the restored feature information output from the feature decoding (2403) of the previous layer is received as input, and the probability of the processed feature information belonging to the current layer can be estimated. The estimated probability distribution is input into the feature decoding (2404) stage of the current layer and utilized during arithmetic decoding, thereby allowing the target to be restored to the same value as the one encoded in the DNN-based point cloud encoding stage (1902).

[0449] The upsampler is composed of the outputs of residual information decoding (2405) and feature decoding (2403). It can take as input and reconstruct the point cloud tensor.

[0450] The upsampler applies upsampling to the input tensor and can compute features using a neural network.

[0451] In this case, a neural network may be used for upsampling or applied after upsampling, and a deep neural network composed of convolutions, MLPs (Multi-Layer Perceptrons), etc., may be used for the neural network.

[0452] In this disclosure, three methods for generating point cloud multidimensional characteristics are presented as examples of upsamplers (2401).

[0453] There are serial generation structures and parallel generation structures for a multidimensional feature extraction method, and each structure may include a first spatial axis-based feature generation layer (2501), a second spatial axis-based feature generation layer (2502), and a third spatial axis-based feature generation layer (2503). The presented structures may be performed as iterative or recursive structures within a single upsampler (2401).

[0454] Information regarding the structure of the spatial axis-based feature extraction network used in the encoder (network structure information) can be signaled to the decoder via a bitstream. Network structure information, such as the network type (convolutional, MLP, etc.), kernel size, input channel size, output channel size, and whether bias values ​​are used, can be signaled to the decoder. Additionally, if the same structure is repeated, only the number of repetitions and the network identifier may be signaled. This is because specific structures from the encoder may be used within the decoder depending on the configuration of the decoder.

[0455] In one embodiment, the types of upsamplers included in a single structure may differ from each other. For example, in the case of FIG. 20, the three upsamplers may be of different types. Here, the meaning of different types may include cases where at least one of i) the structure of the upsampler (serial, parallel, serial-parallel) or ii) the structure information of the upsampler (network type (convolutional, MLP, etc.), kernel size, input channel size, output channel size, whether or not bias values ​​are used, etc.) is different.

[0456] In the embodiment of FIG. 25, the input tensor It may be designed so that feature generation based on each spatial axis is processed continuously.

[0457] In the embodiment of FIG. 26, it is composed of a first spatial axis-based feature generation layer (2501), a second spatial axis-based feature generation layer (2502), and a third spatial axis-based feature generation layer (2503), but unlike a serial generation structure, the input tensor Each result can be simultaneously input to and output through a combining layer (2504), which is a layer that combines each output result.

[0458] At this time, the combining layer (2504) combines the tensors output from each spatial axis-based feature generation layer, and as an example thereof, similar to the combining layer (2104), there is a method of combining features corresponding to each coordinate or expanding the feature dimension and connecting them in the row direction, and there is a method of pruning points that do not match based on the coordinates of a specific tensor to expand the feature dimension and connect them.

[0459] The embodiment of FIG. 27 may be a case where a serial execution method and a parallel execution method are combined and executed.

[0460] All three structures of FIGS. 25 to 27 are input tensors It performs feature generation from, in a dimensionally extended form It can be characterized by outputting.

[0462] FIG. 40 illustrates a specific example of a parallel production structure of a downsampler of the present disclosure.

[0463] The above parallel production structure maintains a stride of 2 and a kernel size of 2 in the input and output dimensions, and can include three feature extraction processes in parallel. Additionally, in the case of FIG. 40, an example is illustrated based on an input tensor composed of point coordinates and feature values ​​within a 2x2x2 voxel.

[0464] In one embodiment, in each feature extraction layer of a parallel structure, the size of the input kernel is different, and in each feature production layer, the size of the kernel used is also different, and different features can be extracted. The different features can be combined through a combination layer.

[0465] For example, referring to FIG. 40, the kernel size of the input for feature extraction in the X-axis direction for Branch A may be [2, 1, 1]. The kernel size of the input for feature extraction in the Y-axis direction for Branch B may be [1, 2, 1]. The kernel size of the input for feature extraction in the Z-axis direction for Branch C may be [1, 1, 2]. Additionally, the kernel size used in the feature extraction layer may be i) [1, 2, 2] for Branch A, ii) [2, 1, 2] for Branch B, and iii) [2, 2, 1] for Branch C.

[0466] FIG. 41 illustrates a specific example of a parallel production structure of an upsampler of the present disclosure.

[0467] The above parallel production structure maintains a stride of 2 and a kernel size of 2 in the input and output dimensions, and can include three feature generation processes in parallel. Additionally, in the case of FIG. 41, an example is illustrated based on an input tensor composed of point coordinates and feature values ​​within a 1x1x1 voxel.

[0468] In one embodiment, in each feature production layer of a parallel structure, the size of the input kernels differs, and the size of the kernels used also differs in each feature production layer, and different features can be produced. The different features can be combined through a combination layer.

[0469] For example, referring to FIG. 41, the kernel size of the input for generating features in the X-axis direction for Branch A may be [1, 2, 2]. The kernel size of the input for generating features in the Y-axis direction for Branch B may be [2, 1, 2]. The kernel size of the input for generating features in the Z-axis direction for Branch C may be [2, 2, 1]. Additionally, the kernel size used in the feature generation layer may be i) [2, 1, 1] for Branch A, ii) [1, 2, 1] for Branch B, and iii) [1, 1, 2] for Branch C.

[0471] In addition to the downsampler (2001) and upsampler (2401) in which the proposed invention is implemented, the structure proposed as an embodiment of dimension reduction or dimension expansion that occurs when estimating the probability of the feature or occupancy information or attribute information of the current layer based on the feature or occupancy information or attribute information of the upper layer, or generating a predicted value, can be utilized during the process of implementing the probability estimation (2005, 2404) and feature processing unit (2002, 2402).

[0472] Whether the proposed technology from the preceding steps is used and the parameters required for using the technology can be stored in various ways according to the conventional bitstream structure.

[0473] According to one embodiment of the present disclosure, information identifying whether the proposed technology is used can be recorded, stored, and transmitted in a Sequence parameter set, Neural network parameter set, Neural network data unit, Geometry parameter set, Geometry data unit, Attribute parameter set, Attribute data unit, etc., and can also be recorded in other structures that are easy to encode / decode. The information identifying whether the technology is used can be transmitted as a boolean variable consisting of 1 bit, and depending on whether it is used, parameters required for using the proposed technology may not be transmitted or the type representing the variable may be changed.

[0474] According to one embodiment of the present disclosure, parameters required for the use of the technology may be recorded, stored, and transmitted in a Neural network parameter set, Neural network data unit, Geometry parameter set, Geometry data unit, Attribute parameter set, Attribute data unit, etc., and may also be recorded in other structures that are easy to encode / decode. The parameters required for the use of the technology may be expressed as N-bit integers or floating-point numbers, or may be arithmetic encoded, and in the case of identifiers, may be linked to predefined names through a separate table.

[0475] The DNN-based multidimensional feature extraction method proposed in the present disclosure can be performed using at least one step. Explanation of the symbols

[0476] The exemplary methods of the present disclosure are described as a series of operations for clarity of description, but this is not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. To implement the method according to the present disclosure, additional steps may be included in addition to the steps exemplified, steps excluding some steps and including the remaining steps, or steps excluding some steps and including additional steps. The various embodiments of the present disclosure are not intended to list all possible combinations but to describe representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more. In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, it may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc. The scope of the present disclosure may include software or machine-executable instructions (e.g., operating system, application, firmware, program, etc.) that enable an operation according to a method of various embodiments to be executed on a device or computer, and a non-transitory computer-readable medium on which such software or instructions, etc. are stored and executable on a device or computer.

Claims

Claim 1 A DNN-based point cloud decoding method comprising: a DNN-based point cloud decoding step that receives a bitstream as input and generates features through multidimensional neural network operations to reconstruct a point cloud; and a reconstructed point cloud output step that outputs the decoded point cloud, wherein in the DNN-based point cloud decoding step, feature decoding, probability estimation, feature processing, residual information decoding, and upsampling are performed.

Images