Method, device, and recording medium for compressing geometric occupancy information for point cloud encoding / decoding

Abstract

The geometric occupancy information compression method, apparatus, and recording medium for point cloud encoding / decoding of the present disclosure may include: a step of obtaining an aggregate feature of the current level mapped to the geometry of the current level based on a feature of the current level restored using a residual feature of the current level; a step of predicting a geometric occupancy probability of the current level based on the aggregate feature of the current level; and a step of performing binary arithmetic encoding (or decoding) using the geometric occupancy probability of the current level and the geometric occupancy (or geometric bitstream) of the current level.

Description

Technology Field

[0001] The present disclosure can be utilized in the field of technology for performing point cloud data compression and restoration. Background Technology

[0002] The field of 3D graphic content technology has grown significantly due to the rapid development of 3D sensors, including 3D scanners, LiDAR, and vision-based 3D reconstruction technology. In addition, as a result of sensor-based data collection, point clouds, which are raw 3D data formats defined by a series of points, have become widely adopted.

[0003] Each point in a point cloud can be represented as a location using 3D coordinates. As a set, a point cloud can represent the geometry of an object or scene. Optionally, each point in a point cloud can be associated with different attributes depending on the application. Examples include RGB colors for immersive applications such as VR and AR, and reflectivity for LiDAR used in autonomous driving and robotic airports. Point cloud data can enable further processing for more complete 3D modeling, such as mesh representation. Additionally, point cloud data can be directly utilized in subsequent machine-driven cognitive tasks, such as sensing and recognition.

[0004] Point clouds can often consist of a series of frames representing dynamic objects or dynamic scenes. This is particularly true for immersive applications such as AR and VR, as well as for LiDAR used in autonomous driving and robotics. Recently, AI technology has demonstrated state-of-the-art performance in the field of point cloud compression. However, unlike processing, enhancement, and rendering recognition tasks, AI-based point cloud compression is not widely used due to the lack of standards.

[0005] Conventional point cloud compression technologies include a lossy compression process that downsamples the original high-resolution point cloud n times, and a lossless compression technology for the point cloud downsampled n times. The problem to be solved

[0006] As data throughput increases, there is a need for compression techniques that are more efficient than conventional techniques. Accordingly, this disclosure proposes a technique that provides a sequence for compressing occupancy information more efficiently. Furthermore, this disclosure proposes a technique that minimizes data loss and improves restoration performance through downsampling and upsampling processes combined with residual learning. means of solving the problem

[0007] The geometric occupancy information compression method, apparatus, and recording medium for point cloud encoding / decoding of the present disclosure may include: a step of obtaining an aggregate feature of the current level mapped to the geometry of the current level based on a feature of the current level restored using a residual feature of the current level; a step of predicting a geometric occupancy probability of the current level based on the aggregate feature of the current level; and a step of performing binary arithmetic encoding (or decoding) using the geometric occupancy probability of the current level and the geometric occupancy (or geometric bitstream) of the current level.

[0008] In the geometric occupancy information compression method, apparatus, and recording medium for point cloud encoding / decoding of the present disclosure, the step of performing binary arithmetic decoding may include the step of dividing the geometric occupancy probability of the current level and the geometric bitstream of the current level into a plurality of groups and the step of sequentially decoding the plurality of groups in any order.

[0009] In the geometric occupancy information compression method, apparatus, and recording medium for point cloud encoding / decoding of the present disclosure, the number of the plurality of groups may be 8.

[0010] In the geometric occupancy information compression method, apparatus, and recording medium for point cloud encoding / decoding of the present disclosure, the arbitrary order may be any one of the Hilbert curve scan order, Morton scan order, Peano Curve Order, and Spiral Scan Order.

[0011] In the geometric occupancy information compression method, apparatus, and recording medium for point cloud encoding / decoding of the present disclosure, the arbitrary order may be in order of smallest or largest occupancy probability distribution obtained through the average of each of the plurality of groups.

[0012] In the geometric occupancy information compression method, apparatus, and recording medium for point cloud encoding / decoding of the present disclosure, the arbitrary order may be in order of smallest or largest occupancy probability entropy obtained by calculating the entropy for each of the plurality of groups.

[0013] In the geometric occupancy information compression method, apparatus, and recording medium for point cloud encoding / decoding of the present disclosure, the arbitrary order may be in order of smallest or largest occupancy probability binary cross-entropy obtained by calculating the binary cross-entropy for each of the plurality of groups.

[0014] In the geometric occupancy information compression method, apparatus, and recording medium for point cloud encoding / decoding of the present disclosure, the step of sequentially decoding a plurality of groups in any order may include the step of dividing the plurality of groups according to any order of n arbitrary order sets, and the step of sequentially encoding (or decoding) the divided groups in any order of the divided groups. Effects of the invention

[0015] The present disclosure achieves high-performance compression and decompression by more efficiently processing occupancy information that is divided and sequentially encoded / decoded during the process of losslessly compressing and decomposing a point cloud, thereby maximizing the efficiency of data storage space. Brief explanation of the drawing

[0016] Figure 1 illustrates an example of a 4-bit binary tree structure storing voxels. Figure 2 illustrates an example of an octree structure for a voxel. Figure 3 illustrates an example of an array of coding units in a bitstream. Figure 4 illustrates an example of the overall decoding process. Figure 5 illustrates examples of lossless-based decoding and loss-based decoding of the geometric decoding process. Figure 6 illustrates an example of the ancestor occupancy analysis step. Figure 7 illustrates an example of a feature decoding step. Figure 8 illustrates an example of a sparse 3D convolution-based point synthesis method. Figure 9 illustrates an example of the possession decoding step. FIG. 10 illustrates an example of a learning-based motion decoding process. FIG. 11 illustrates an example of the inter-predictor derivation step. Figure 12 illustrates an example of a motion compensation step. Figure 13 illustrates an example of the overall encoding process. Figure 14 illustrates examples of lossless-based encoding and lossy-based encoding of the geometric encoding process. Figure 15 illustrates an example of a feature encoding step. Figure 16 illustrates an example of a possession encoding step. FIG. 17 illustrates an example of a learning-based motion encoding process. FIG. 18 illustrates an example of the inter-predictor derivation step. Figure 19 illustrates an example of the structure of an artificial intelligence-based input image compression framework. Figure 20 illustrates an example of a framework for hierarchical lossy compression and lossless compression. FIG. 21 illustrates an example of a flowchart of a geometric occupancy information compression method for point cloud encoding / decoding of the present disclosure. FIG. 22 illustrates an example of a feature information generation unit of a geometric occupancy information compression device for point cloud subcoding / decoding. FIG. 23 illustrates an example of a feature information coding section and a feature encoder of a geometric occupancy information compression device for point cloud encoding / decoding. FIG. 24 illustrates an example of a feature decoder. FIG. 25 illustrates an example of a occupancy information coding section of a geometric occupancy information compression device for point cloud encoding / decoding and an example of an octree encoder. Figure 26 illustrates an example of an octree decoder. Specific details for implementing the invention

[0017] The present disclosure is subject to various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present disclosure to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present disclosure. Similar reference numerals in the drawings refer to the same or similar functions across various aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation. The detailed description of exemplary embodiments described below refers to the accompanying drawings, which illustrate specific embodiments as examples. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that various embodiments are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present disclosure in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the following detailed description is not intended to be taken in a limiting sense, and the scope of exemplary embodiments is limited only by the appended claims, together with all equivalents to those claimed therein, provided they are properly described.

[0018] In this disclosure, terms such as first, second, etc. may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of this disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0019] Where it is stated that any component of the present disclosure is "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, or that there may be other components in between. On the other hand, where it is stated that a component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0020] The components shown in the embodiments of the present disclosure are depicted independently to represent different characteristic functions and do not imply that each component consists of separate hardware or a single software unit. That is, each component is listed and included as a separate component for convenience of explanation; however, at least two of the components may be combined to form a single component, or a single component may be divided into multiple components to perform a function, and such integrated and separated embodiments of each component are included within the scope of the rights of the present disclosure as long as they do not depart from the essence of the present disclosure.

[0021] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit this disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this disclosure, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. That is, the description in this disclosure that a specific configuration "comprising" does not exclude configurations other than that configuration, but means that additional configurations may be included within the scope of the practice or technical concept of this disclosure.

[0022] Some components of the present disclosure may not be essential components performing an essential function in the present disclosure, but may be optional components merely for enhancing performance. The present disclosure may be implemented by including only the components essential to embody the essence of the present disclosure, excluding components used merely for enhancing performance, and a structure including only the essential components, excluding optional components used merely for enhancing performance, is also included within the scope of the rights of the present disclosure.

[0023] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In describing the embodiments of this specification, if it is determined that a detailed description of related known configurations or functions may obscure the gist of this specification, such detailed description is omitted, and the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0024] The present disclosure discloses the syntax, semantics, and decoding processes of point cloud data using an Artificial Intelligence (AI)-based coding method. Additionally, the present disclosure discloses processes necessary for point cloud reconstruction and discloses additional processes such as post-decoding processes, pre-reconstruction processes, post-reconstruction processes, and adaptive processes.

[0025] The definitions of the terms of the present disclosure may be as follows.

[0026] Augmented reality (AR) can be an environment that overlays computer-generated content onto the real world viewed by the user.

[0027] Virtual reality (VR) is a computer-generated environment featuring realistic-looking scenes and objects, allowing users to feel immersed in their surroundings.

[0028] The world coordinate system can be the coordinate system that captures the point cloud. The world coordinate system can also be referred to as the original coordinate system.

[0029] A coding coordinate system can be a coordinate system in which a point cloud is coded. In a coding coordinate system, geometry can have non-negative values.

[0030] The definitions of the abbreviations of the present disclosure may be as follows.

[0031] AU: Access Unit

[0032] IDR: Instantaneous Decoding Refresh

[0033] IRAP: Intra Random Access Point

[0034] LSB: Least Significant Bit

[0035] MSB: Most Significant Bit

[0036] PCC: Point Cloud Compression

[0037] AI-PCC: AI-based Point Cloud Compression

[0038] Vector can be a vector with a number of elements.

[0039] Matrix<r, c> It can be a matrix with r rows and c columns.

[0040] Tensor<r, a> can be a sparse tensor specified as a-dimensional vector data for a list of 3-dimensional coordinates. Here, the spatial resolution for each x, y, and z direction is from 0 to 2 r It can be between -1. Also, parameter a can be the number of feature channels associated with each 3D point.

[0041] Concat(T1, T2, T3, ...) can be a concatenated layer in which multiple tensors T1, T2, T3, ... are connected in a single order as they appear in the list. Here, all connected tensors must have the same resolution and may have the same set of points. Additionally, each tensor may have a different number of feature channels.

[0042] ReLU (rectified linear unit) is a type of non-linear activation function used in artificial neural networks, and can be called a rectified linear unit. For example, the function outputs the input value as is when it is greater than a threshold value, but outputs 0 otherwise (when the input value is less than or equal to the threshold value).

[0043] Conv<a, b, c, d, e> can be a 3D synthesizing layer with a kernel size of a×a×a. Here, b is a stride parameter, c is a dilation size, d is an input channel size, and e is an output channel size.

[0044] Deconv<a, b, c, d, e> can be a 3D inverse convolutional layer with a kernel size of a×a×a. Here, b is the stride parameter, c is the dilation size, d is the input channel size, and e is the output channel size.

[0045] MLP <c0, c1, c2, ..., c n > can be a multi-layer perceptron (MLP) having n layers. Here, c0 is the size of the input channel, and c i ≡ can be the size of the output channel of each layer (1 ≤ i ≤ n).

[0046] Variables (or terms) related to the point cloud geometry of the present disclosure may be referred to as points or voxels. A point explicitly indicates the location (x, y, z) of a sample but may not specify the size of the occupied space. A voxel implicitly indicates the location of a sample through its position in an octree and may represent a cube of a specific size. The center of a voxel may be used as the location of the corresponding point.

[0047] When point cloud geometry is represented by points, each point cloud frame may consist of an array of 3D positions (x, y, z). Here, the coordinate values ​​of the 3D positions may be represented as signed integers in the world coordinate system. Additionally, the coordinate values ​​of the 3D positions may be represented as n-bit unsigned integers in the shifted coding coordinate system of the present disclosure. Here, n is the bit depth and may represent the precision of the point position. Additionally, n may be explicitly determined by information signaled from the bitstream or implicitly determined by other point cloud-related information. In this case, in the shifted coding coordinate system, one edge is (0,0,0) and the opposite edge is (2 n-1 , 2 n-1 , 2 n-1 )person 2 n The size of the cubic bounding box can be defined.

[0048] The scaled coding coordinate system of the present disclosure may be a coordinate system obtained by scaling a shifted coding coordinate system to define a scaled coding coordinate system for compression. Whether scaling is applied, the degree of scaling, etc., may be explicitly determined by information signaled from the bitstream or implicitly determined by other point cloud-related information.

[0049] Figure 1 illustrates an example of a 4-bit binary tree structure storing voxels.

[0050] An octree (or octree structure) is another way of representing a point that can hierarchically partition space. In point cloud geometry with n-bit coordinates, an octree can have a total of n+1 levels from 0 to n. Here, 0 can represent the root level. An octree, which serves to store voxels—3D data—can be a tree structure in which each node has 8 children.

[0051] Before describing the octree, referring to Fig. 1, the binary tree structure of Fig. 1 is 4 bits, so it can have a total of 5 levels from 0 to 4. In addition, each node represents a voxel, and an occupied voxel can have 2 child nodes.

[0052] Figure 2 illustrates an example of an octree structure for a voxel.

[0053] In an octree (or octree structure), for octree level i = 0 (root level), a voxel can occupy the entire space within the point cloud bounding box. For each level, the space occupied by the voxel in the parent level can be divided equally in the x, y, and z directions to generate eight child voxels of equal size. In other words, each child voxel can occupy 1 / 8 of the parent voxel's space. Accordingly, the voxel size is determined by the octree level i to which it belongs, and when normalized, it is 2 n-i or 2 n-i / 2 n It may be. In the present disclosure, the child voxel may also be expressed as an octant.

[0054] In the present disclosure, occupied voxels may be represented by a value of 1, and empty voxels (unoccupied voxels) may be represented by a value of 0.

[0055] Occupancy map (or voxel occupancy map) of the present disclosure O i can represent the point cloud for octree level i. Also, occupancy map O i is Tensor<i, 1> It can be expressed as such. In this case, the spatial resolution of the tensor can be specified as the current octree level i. The one-dimensional value for each point of the tensor can indicate whether the voxel is occupied (=1) or empty (=0).

[0056] At a specific octree level, the voxelCount variable can be defined as the number of occupied voxels in the parent octree level. The said variable can be obtained by the GetVoxelCountParent() function, which counts the number of occupied voxels in the parent octree level.

[0057] Attributes of a point cloud may provide additional information related to each point of a source point cloud. If attributes exist, each attribute may be associated with the corresponding point of the source point cloud. The attribute types of the present disclosure may include color and reflectance.

[0058] The color attributes of the present disclosure may be represented in an RGB color space. For example, each color channel (red, green, blue) may be represented as an 8-bit unsigned integer value within the range [0, 255].

[0059] When point cloud geometry is represented as an array of 3D positions (x, y, z), color attributes can be represented as an array with an extended field (x, y, z, r, g, b).

[0060] When point cloud geometry is represented by octree voxels, the color property is a Tensor<n, 3> It can be expressed as follows. Here, n can represent the bit depth of the point cloud geometry. The 3D vector for each point in the tensor can represent the associated color value (r, g, b).

[0061] When point cloud geometry is represented as an array of 3D positions (x, y, z), the reflectance property can be represented as an array with an extended field (x, y, z, r). Here, r can represent reflectance.

[0062] When point cloud geometry is represented by octree voxels, the reflectance property is a Tensor<n, 1> It can be expressed as follows. Here, n can represent the bit depth of the point cloud geometry. A one-dimensional scalar for each point of the tensor can represent the associated reflectance value r.

[0063] Point cloud features can be used as latent representations of neural network-based processes during point cloud encoding and decoding. Point cloud features can be used as a format other than the occupancy map in the encoded point cloud. Point cloud features may not be included in the source point cloud or the decoded point cloud.

[0064] A set of features spanning all voxels within an occupancy map can be called a feature map. At octree level i, the feature map is a Tensor<j, a> It can be expressed as. Here, j is feature map 2 j ×2 j ×2 j It represents the spatial resolution of and may be equal to or different from octree level i. Here, a may represent the number of feature channels.

[0065] Each point in the feature map can correspond to an occupied or empty voxel. Each point in the feature map can be associated with an a-dimensional feature vector. Each a-dimensional feature vector can encode geometric information for intra-coding. Alternatively, it can encode temporal dynamics and motion patterns for inter-predictive coding.

[0066] In addition, the feature map can be used for lossy decoding of the occupancy map or to support lossless decoding.

[0067] Points in a point cloud can be sorted according to the raster scanning order. Raster scanning can proceed in the order of the z-direction first, then the y-direction, and finally the x-direction. This can be equivalent to sorting points in ascending order using a key value k calculated based on the point location (x, y, z). Here, k is x × 2 r+r + y×2 r It could be + z.

[0068] In addition, using raster scanning order, the tensor T is converted into a 2D matrix M = Matrix<N, c> You can map it to or to the opposite.

[0069] A point cloud frame can be defined as a complete set of point cloud data representing a 3D scene or object at a specific point in time. A point cloud sequence can consist of a series of point cloud frames.

[0070] A point cloud frame is divided into one or more point cloud slices, and a point cloud slice may be a spatial subunit of the frame. Each slice contains a subset of points within the frame and can generally be grouped according to spatial locality, coding efficiency, or parallel processing requirements. Encoding of the point cloud frame can be performed by independently encoding each constituent slice.

[0071] There may be four types of data coding units: raw coding units, feature coding units, occupancy coding units, and motion coding units. When reproducible coding is enabled, a protection coding unit may be additionally used. All data coding units, except for the raw coding unit, can be connected to a protection coding unit. Through this connection, the corresponding data can be decoded reliably and consistently, which can be particularly important for cross-platform compatibility and deterministic decoding motion.

[0072] Figure 3 illustrates an example of an array of coding units in a bitstream.

[0073] Coding units can be hierarchically organized from Octree Level 1 to the last coded Octree Level. The last coded Octree Level may differ from the last Octree Level of the input point cloud.

[0074] At octree level i=0, which corresponds to the root node of an octree structure, no data may be explicitly coded into the bitstream. By definition, the root node can be considered to always be occupied.

[0075] For octree level i=1, raw coding units can be used to code the occupancy status of 8 octants.

[0076] Using a raw coding unit, 1 byte is used to encode an occupancy of 8 octants, and each bit can correspond to one octant.

[0077] For octree levels 1 < i ≤ j, one feature coding unit may be present in the bitstream for each coded octree level. Additionally, each level may encode eight occupying coding units after encoding one feature coding unit. That is, the feature coding unit can be used to assist the occupying coding unit. Here, the eight occupying coding units may correspond to eight octants according to the raster scan order ((0,0,0), (1,0,0), (0,1,0), (1,1,0), (0,0,1), (1,0,1), (0,1,1), (1,1,1)).

[0078] The occupancy coding unit may include encoded occupancy data indicating the occupancy status of a voxel for a specific octant of the current octree level.

[0079] Referring to Fig. 3, lossless coded octree levels (or lossless-based octree levels) can represent octree levels 1 ≤ i ≤ j.

[0080] Referring to Fig. 3, lossy-coded octree levels (or lossy-based octree levels) can represent octree levels j < i ≤ n.

[0081] For octree levels j < i ≤ n, each level may need to encode one feature coding unit. Additionally, the feature coding unit can be used to derive an occupancy map through a point synthesis process using a lossy coding method. Furthermore, for lossy-coded octree levels, there may be no occupancy coding unit.

[0082] For inter-coded frames, for each coded octree level where i > 1, one motion coding unit must be inserted before the feature coding unit.

[0083] A protection coding unit may be encoded in the bitstream to protect the decoding of a feature coding unit, a occupancy coding unit, and a motion coding unit (all data coding units excluding raw coding units). Each protection coding unit is associated with a single data coding unit, and the associated single data coding unit may be a feature coding unit, an occupancy coding unit, or a motion coding unit. The role of the protection coding unit may be to protect the reproducibility of the relevant data during decoding. For example, if protection for cross-platform reproducibility for a specific feature coding unit is enabled, the protection coding unit may be encoded in the bitstream to protect the decoding of said feature coding unit.

[0084] The motion coding unit may include encoded motion feature data representing the temporal dynamics between the current frame and the reference frame of the current octree level.

[0085] For each intercoded octree level, one motion coding unit may exist in the bitstream. Additionally, as shown in FIG. 3, the motion coding unit may take precedence over the feature coding unit at the same octree level.

[0086] Motion features can be extracted, quantized, and entropy-encoded by an encoder and stored in a bitstream. The motion features can capture the time relationship and motion pattern between the two slices of the current frame and the decoded reference frame, respectively, at the current octree level.

[0087] Figure 4 illustrates an example of the overall decoding process.

[0088] The decoding process of the present disclosure may include learning-based point cloud geometric decoding and Region Adaptive Hierarchical Transform (RAHT)-based attribute decoding.

[0089] Both geometry decoding and attribute decoding can have intra mode and inter mode as decoding modes.

[0090] Geometric decoding is neural network-based decoding, and neural network weights can be obtained according to model identifier information before the geometric decoding process is executed. Here, the model identifier information may be information explicitly signaled from the bitstream, information pre-defined in the device, or information implicitly determined by other information.

[0091] For octree level i=1, eight binary occupancy values ​​can be decoded into a raw coding unit consisting of a single byte. The eight voxels can be arranged according to raster scanning order. The bits of the byte in the raw coding unit can be indexed from the Most Significant Bit (MSB) to the Least Significant Bit (LSB). Each bit can represent the occupancy status of the corresponding voxel. Residual features at this level , Tensor<0, 8> can be set as a constant dummy feature where all elements are 1.

[0092] In the case of intra-point cloud geometry decoding, the geometry of the decoded point cloud can be constructed in a Learning-based Geometry Decoding Unit based on the geometry bitstream of the point cloud. Intra-point cloud geometry decoding can be executed for each coded octree level, starting from octree level i=2.

[0093] For inter-point cloud geometry decoding, geometry predictor information can be generated in a Learning-based Motion Decoding Unit based on the motion bitstream and reference frame. This geometry predictor information is shared as an additional input with a Learning-based Geometry Decoding Unit for intra-decoding, allowing the geometry of the decoded point cloud to be constructed in the Learning-based Geometry Decoding Unit. Inter-point cloud geometry decoding can be executed for each coded octree level, starting from octree level i=2.

[0094] In the case of intra-point cloud attribute decoding, the decoded point cloud attributes can be constructed in a RAHT-based Attribute Decoding Unit based on the attribute bitstream and the decoded point cloud geometry.

[0095] In the case of inter point cloud attribute decoding, attribute predictor information can be generated in a recoloring unit based on the decoded point cloud geometry and attribute reference frame. The attribute predictor information is shared as an additional input to a RAHT-based attribute decoding unit of intra decoding, and the decoded point cloud attributes can be constructed in the RAHT-based attribute decoding unit of intra decoding.

[0096] The point cloud geometry intra-decoding process (=geometry intra-decoding process) may be called at each octree level when the current point cloud slice is determined to be in geometry intra mode (e.g., when the unit type is determined to be Geometry Intra-Slice). Here, the unit type may be explicitly signaled from the bitstream or implicitly determined by other information.

[0097] Additionally, the geometry intra-decoding process may be called for each octree level if the current octree level is greater than the threshold octree level, even if the current point cloud slice is determined to be in geometry inter-mode (e.g., even if the unit type is determined to be Geometry Inter-Slice). Here, the threshold octree level may be explicitly signaled from the bitstream or implicitly determined by other information.

[0098] The geometric intra-decoding process can be performed in the learning-based geometric decoding unit of Fig. 4.

[0099] The unit types of the present disclosure may include types of Geometry Intra-Slice, Geometry Inter-Slice, Attribute Intra-Slice, and Attribute Inter-Slice.

[0100] Types of geometric intra-decoding processes may include lossless-based decoding and lossy-based decoding.

[0101] Figure 5 illustrates examples of lossless-based decoding and loss-based decoding of the geometric decoding process.

[0102] The loss-based decoding process may include 1) an ancestor occupancy analysis step, 2) a feature decoding step, and 3) a point synthesis step.

[0103] The lossless-based decoding process may include 1) an ancestor occupancy analysis step, 2) a feature decoding step, and 4) an octree decoding step.

[0104] That is, in the loss-based decoding and lossless-based decoding processes, 1) the ancestor occupancy analysis step and 2) the feature decoding step may be common processes.

[0105] 1) Ancestor Occupancy Analysis Stage

[0106] In the ancestor occupancy analysis step, if occupancy information of the ancestor octree levels (i-1, i-2, i-3, i-4) of the current octree level is available, ancestor features are based on that information. You can obtain.

[0107] The ancestor occupancy analysis step may be an initialization step performed before the 2) feature decoding step.

[0108] Figure 6 illustrates an example of the ancestor occupancy analysis step.

[0109] The ancestor occupancy analysis step may include an ancestor feature initialization step, a feature mix step, and an adaptive affine step.

[0110] The ancestor feature initialization step is the occupancy map of the parent octree level Ancestor information that is a 32-dimensional vector based on can be obtained. Here, the ancestor information is [j, p j , o j , v j It is defined by the connection of ], where j is a scalar value representing the octree level index, p is a 3D vector indicating the 3D position, o is a 3D vector indicating the binary octant, and v is a scalar value representing the voxel size.

[0111] The Feature Mix step may be a step that performs a basic neural network-based process for feature aggregation and channel size transformed from a to c. a and c are the FeatureMix neural network layers of the Picture Mix step.<a, b, c, d, e> It can be determined through. For example, in the feature mix step of FIG. 6, the channel size can be changed from 32 to 16. All or some of the parameters used in the feature mix step may be determined differently based on bit depth-related information.

[0112] The adaptive affine stage is a style condition f representing the desired coding movement. style Based on this, an adaptive affine process can be performed to transform input features through learnable affine transformations. The adaptive affine process can be inserted into various processes such as feature extraction, point synthesis, feature encoding, feature decoding, octree encoding, and octree decoding to transform the features of the corresponding processes.

[0113] 2) Feature decoding step

[0114] In the feature decoding stage, the feature bitstream for the current octree level , ancestor characteristics of the current octree level , constant dummy prediction feature F pred , decoded residual features of the parent octree level , and the occupancy map of the parent octree level Based on, the decoded features of the current octree level and decoded residual features of the current octree level You can obtain.

[0115] Constant dummy prediction feature F pred All elements of can be equal to 1. Constant dummy prediction features can be used to match geometric feature decoding for intra-coding with inter-coding.

[0116] If no features were decoded in the previous level, the decoded residual features of the parent octree level It can be used as a dummy feature where all elements are 1.

[0117] Cases where there are no features decoded at the previous level may include i) cases where octree level i=2 is decoded (since features are not encoded at octree level i=1) and ii) cases where the value of a flag instructing to set residual features as dummy features for all octree levels for lossless coding is 1.

[0118] Figure 7 illustrates an example of a feature decoding step.

[0119] The feature decoding step may include a super-prescription synthesis step, an arithmetic decoding step, and a conditional decoding step.

[0120] The superpredictive synthesis step can synthesize feature variance parameters based on decoded residual features of the parent octree level.

[0121] The arithmetic decoding step can obtain the decoded residual features of the current octree level by arithmetic decoding the feature bitstream for the current octree level based on the feature distribution parameter obtained in the superprescription synthesis step.

[0122] The conditional decoding step can conditionally decode the decoded features of the current octree level based on the decoded residual features of the current octree level, using the ancestor features and constant dummy prediction features of the current octree level as conditions.

[0123] 3) Point composite step

[0124] In the point synthesis step, the decoded features of the current octree level Based on the decrypted occupancy map of the parent octree level, the (lossy) decrypted occupancy map of the current octree level can be obtained.

[0125] The point synthesis step can be performed for each octree level configured for loss-based decoding.

[0126] The point synthesis step can be performed in different ways based on a flag indicating whether a MultiLayer Perceptron (MLP) is used.

[0127] For example, the point synthesis step can be performed by a sparse 3D convolution-based point synthesis method when a flag indicating whether to use an MLP has a first value, and by an MLP-based point synthesis method when a flag indicating whether to use an MLP has a second value.

[0128] Figure 8 illustrates an example of a sparse 3D convolution-based point synthesis method.

[0129] For the sparse 3D convolution-based point synthesis method, as an additional input, style condition f style , can have a parameter representing the number of voxels of the current octree level obtained from the slice geometry header.

[0130] A sparse 3D convolution-based point synthesis method may include a feature aggregation step, a probability estimation step, and a classification step.

[0131] The feature aggregation step of the sparse 3D convolution-based point synthesis method is the decoded features at the current octree level Improve, and aggregate and upsample the improved features to aggregate and upsample the features You can obtain.

[0132] The probability estimation step of the sparse 3D convolution-based point synthesis method is aggregated and upsampled features Using a multi-perceptron (MLP) network as input, the predicted occupancy probability of voxels at the current octree level with one channel can be calculated. In this case, the channel size of the MLP layer used can be determined differently based on bit depth information.

[0133] The classification step of the sparse 3D convolution-based point synthesis method can obtain an occupancy map of the current octree level by sorting all child voxels (current octree level) that have an occupied parent voxel, classifying the K voxels with the highest occupancy probability as occupied and the remaining voxels as empty.

[0134] An MLP-based point synthesis method may include an offset estimation step, a point construction step, and a point clipping and pruning step.

[0135] The offset estimation step of the MLP-based point synthesis method is the decoded features of the current octree level After converting it into a feature having 15 channels, three consecutive channels can be considered as one 3D offset vector to generate five 3D offset vectors for each occupied voxel of the parent octree level.

[0136] The point construction step of the MLP-based point synthesis method can generate five 3D points of the current octree level by moving each occupied voxel of the parent octree level according to five 3D offset vectors.

[0137] The point clipping and pruning steps of the MLP-based point synthesis method clip the coordinates of 3D points based on the bounding box of the point cloud frame and prun the duplicate points to obtain the current octree level occupancy map.

[0138] 4) Octree decoding step

[0139] In the octree decoding stage, the decoded features of the current octree level Based on the occupancy map of the parent octree level and the occupancy bitstream of the current octree level, the (lossless) decrypted occupancy map of the current octree level can be obtained.

[0140] The octree decoding step can be performed for each octree level configured for lossless-based decoding.

[0141] The mode of the octree decoding step can be determined based on information related to bit depth. The mode of the octree decoding step may include a current level autoregressive mode and a parent level autoregressive mode.

[0142] Depending on the mode, the processes of the octree decoding step can be performed at the resolution of the current octree level or the resolution of the parent octree level.

[0143] The octree decoding step may include an initialization step, a feature aggregation step, and a possession decoding step.

[0144] In the initialization phase, depending on the mode of the octree decoding phase, the occupancy map of the parent octree level is expanded or upsampled to the occupancy map of the current octree level to obtain the initial occupancy map of the current octree level. It can generate.

[0145] In the case of the current level autoregressive mode, during the initialization phase, the occupancy map of the parent octree level is upsampled to the occupancy map of the current octree level to obtain the initial occupancy map of the current octree level. , Tensor<i, 1> You can obtain. The initial occupancy map of the current octree level has 8 × voxelCount (number of occupied voxels in the parent octree level) points, and all child voxels of an occupied parent voxel can be set to 0.

[0146] In the case of parent-level autoregressive mode, during the initialization phase, the occupancy map of the parent octree level is extended to the occupancy map of the current octree level, and the initial occupancy map of the current octree level , Tensor<i-1, 8> You can obtain. The initialization phase initializes all channels to 0 and Repeat the channel 8 times You can generate 8 child voxels. The 8 child voxels are mapped to 8 channels indexed from 0 to 7, and the order can be (0,0,0), (1,0,0), (0,1,0), (1,1,0), (0,0,1), (1,0,1), (0,1,1), (1,1,1).

[0147] In the feature aggregation stage, the decoded features of the current octree level Based on, aggregated features according to mode You can obtain.

[0148] In the case of the current level autoregressive mode, the decoded features of the current octree level are used in the feature aggregation step. Improves, aggregates the improved features, and upsamples the aggregated features You can obtain.

[0149] In the case of parent-level autoregressive mode, the decoded features of the current octree level are used in the feature aggregation step. Aggregated features obtained by aggregating without changing the resolution (having the resolution of the parent octree level). You can obtain.

[0150] Figure 9 illustrates an example of the possession decoding step.

[0151] The occupancy decoding step can be called repeatedly 8 times to decode the occupancy map of octree level i. k, representing the numbering of the iterations, can range from 1 to 8. The iterations for 8 octants can follow the order of (0,0,0), (1,0,0), (0,1,0), (1,1,0), (0,0,1), (1,0,1), (0,1,1), (1,1,1).

[0152] The possession decoding step for the k-th iteration can be performed by i) a possession buffer, ii) a sibling feature initialization unit, iii) a sibling feature mixing unit, iv) a probability estimation unit, v) an arithmetic decoding unit, and vi) a possession deserialization unit.

[0153] i) Occupied buffer

[0154] The occupancy buffer contains all sibling occupancy maps decrypted in the previous iteration of the current octree level. (from 1 to k-1) can be stored and provided so that it can be referenced in the next decoding iteration. Additionally, a mask, which is a decoding mask indicating whether a child voxel is decoded, can be managed. Also, for the first iteration (k=1), the occupied buffer can be initialized.

[0155] ii) Sibling characteristic initialization section

[0156] In the sibling feature initialization section, all sibling occupancy maps decrypted in the previous iteration Initialize features based on to initial sibling features You can obtain.

[0157] In the case of the current level autoregressive mode, the sibling feature initializer performs initialization using a 5-dimensional vector to initialize the sibling features It is possible to obtain. In this case, 1 dimension represents a scalar value for the octree level, 3 dimensions represent a scalar value for the 3D position, and 1 dimension represents a binary value of the decoding mask map.

[0158] In the case of parent-level autoregressive mode, the input in the sibling feature initialization section Initialize identically to the initial sibling characteristics You can obtain.

[0159] iii) Brothers' Characteristics Mixing Department

[0160] In the sibling characteristic mixing section, the aggregated characteristics Characteristics of early siblings Mixed characteristics by mixing You can obtain.

[0161] The mixing process of the sibling feature mixing section may vary depending on whether the mode is the current level autoregressive mode or the parent level autoregressive mode.

[0162] iv) Probability Estimator

[0163] In the probability estimation unit, mixed features Based on, the occupancy probability of voxels at the current octree level for each occupied parent voxel It can predict.

[0164] In the case of the current level autoregressive mode, the probability estimation unit uses a multi-perceptron (MLP) network to estimate the occupancy probabilities of voxels at the current octree level having one channel. It can be predicted. In this case, the channel size of the MLP layer used may be determined differently by bit depth-related information.

[0165] In the case of parent-level autoregressive mode, the probability estimator uses convolution layers, ReLU layers, and Inception ResNet (IRN) layers to estimate the occupancy probabilities of the voxels at the current octree level. It can predict.

[0166] v) Arithmetic decoding part

[0167] In the arithmetic decoding unit, based on the occupied bitstream and the predicted probability for the current octant of the current octree level, the binary value You can obtain the sequence of.

[0168] vi) Occupied deserializer

[0169] In the possession deserialization section, the decoded possession map of the previous iteration and binary sequence Use the occupancy map of the current iteration You can acquire. The occupied status can be set to 1, and unoccupied to 0. After completing all 8 iterations, the current Octree level The decrypted occupancy map of is , , ..., It can be set as a combination of.

[0170] The point cloud geometry inter-decoding process can be called for each octree level when the current point cloud slice is signaled in geometry inter mode (i.e., the unit type is signaled as Geometry Inter-Slice) and the current octree level is less than or equal to the critical octree level.

[0171] The point cloud geometry inter-decoding process can be performed by a combination of the learning-based geometry decoding unit, the learning-based motion decoding unit, and the reference frame unit of FIG. 4.

[0172] The point cloud geometry inter-decoding process may include a motion decoding process and a main decoding process.

[0173] In the motion decoding process, a predictor for the current frame can be obtained based on the motion between the reference frame and the current frame. This may correspond to the learning-based motion decoding unit and the reference frame unit of FIG. 4.

[0174] In the main decoding process, geometric decoding can be performed based on a predictor for the current frame. This may correspond to the learning-based geometric decoding unit of Fig. 4.

[0175] The global motion compensation process can be called as a preprocessing process at every octree level before calling the inter-decoding process.

[0176] The inputs and outputs of the global motion compensation process can belong to the transformed coding coordinate system.

[0177] 1) First, the reference point cloud frame The coordinates of each point in can be transformed based on the point cloud bounding box center (c, c, c). The above transformation is It can be performed by. The position of the bounding box center in the transformed coding coordinate system is It can be defined as follows.

[0178] 2) Next, the reference point cloud frame using the rotation matrix R and translation vector T specified by the fps_transform_matrix array of the bitstream It can be converted as follows.

[0179] 3) Finally, reference point cloud frame A global motion compensation reference frame can be obtained by applying an inverse transformation to move it back to the transformed coding coordinate system. The above inverse transformation is It can be performed by.

[0180] Unless there are special circumstances, if gps_global_motion_flag is equal to 1, the reference point cloud geometry of the present disclosure can be interpreted as a global motion-compensated reference point cloud.

[0181] The learning-based motion decoding process performed in the learning-based motion decoding unit can be called for each octree level for a slice at the start of the point cloud geometry inter-decoding process.

[0182] The learning-based motion decoding process can generate predicted features to support the main decoding process.

[0183] FIG. 10 illustrates an example of a learning-based motion decoding process.

[0184] The learning-based motion decoding process may include a reference feature derivation step and an inter-predictor derivation step.

[0185] In the reference feature derivation step, reference features of the current octree level can be obtained by utilizing style conditions and the decoded occupancy map of the current octree level. Additionally, if the flag indicating whether to use a MultiLayer Perceptron (MLP) is the first value, reference features of the current octree level can be obtained by additionally utilizing the occupancy map of the previous octree level.

[0186] FIG. 11 illustrates an example of the inter-predictor derivation step.

[0187] The inter-predictor derivation step may include a motion feature buffer management step, a super-predicted synthesis step, an arithmetic decoding step, and a motion compensation step.

[0188] The motion feature buffer management step can store and provide decoded motion features. For example, the motion feature buffer management process can provide the decoded motion features of the parent octree level to the superpre-synthesis step for the current octree level.

[0189] The superpredictive synthesis step can synthesize feature variance parameters using motion features of the parent octree level. The superpredictive synthesis step can operate identically to the previously described superpredictive synthesis step. However, the input can be the decoded motion features of the parent octree level instead of the decoded residual features of the parent octree level.

[0190] The arithmetic decoding step can operate in the same manner as the previously described arithmetic decoding step. However, the input bitstream can be replaced with a motion bitstream. That is, the arithmetic decoding step can obtain the decoded motion features of the current octree level by arithmetic decoding the motion bitstream based on the feature distribution parameters obtained in the superpredictive synthesis step.

[0191] In the motion compensation step, based on the decoded motion features of the current octree level, the reference features of the current octree level, and the style conditions, the predicted features of the current octree level You can obtain.

[0192] Figure 12 illustrates an example of a motion compensation step.

[0193] The main decoding process can be called at each octree level of the slice after the learning-based motion decoding process is completed.

[0194] The main decoding process is performed in the learning-based geometric decoding unit of FIG. 4, and the unit may be a neural network shared with the geometric intra-decoding process.

[0195] Similar to the geometric intra-decoding process, the types of the main decoding process may include lossless-based decoding and loss-based decoding.

[0196] Compared to the geometric intra-decoding process, the difference in the main decoding process may be that it uses predictive features generated from the learning-based motion decoding process instead of dummy features.

[0197] The loss-based decoding process may include 1) an ancestor occupancy analysis step, 2) a feature decoding step, and 3) a point synthesis step.

[0198] The lossless-based decoding process may include 1) an ancestor occupancy analysis step, 2) a feature decoding step, and 4) an octree decoding step.

[0199] That is, in the loss-based decoding and lossless-based decoding processes, 1) the ancestor occupancy analysis step and 2) the feature decoding step may be common processes.

[0200] 1) Ancestor Occupancy Analysis Stage

[0201] In the ancestor occupancy analysis step, if occupancy information of the ancestor octree levels (i-1, i-2, i-3, i-4) of the current octree level is available, ancestor features are based on that information. You can obtain.

[0202] In the ancestor occupancy analysis step, if occupancy information of the ancestor octree levels (i-1, i-2, i-3, i-4) of the current octree level is available, ancestor features are based on that information. You can obtain.

[0203] The ancestor occupancy analysis step may include an ancestor feature initialization step, a feature mixing step, and an adaptive affine step. The specific details of each step of the ancestor occupancy analysis step are as previously described and are omitted below.

[0204] 2) Feature decoding step

[0205] Bitstream featuring the current octree level , in the feature decoding stage, the ancestor features of the current octree level , predictive features of the current octree level , decoded residual features of the parent octree level , and the occupancy map of the parent octree level Based on, the decoded features of the current octree level and decoded residual features of the current octree level You can obtain.

[0206] 3) Point composite step

[0207] In the point synthesis step, the decoded features of the current octree level Based on the occupancy map of the parent octree level, the (lossy) decrypted occupancy map of the current octree level can be obtained.

[0208] The point synthesis step can be performed for each octree level configured for loss-based decoding.

[0209] The point synthesis step can be performed in different ways based on a flag indicating whether a MultiLayer Perceptron (MLP) is used.

[0210] For example, if a flag indicating whether to use an MLP has a first value, a sparse 3D convolution-based point synthesis method may be performed, and if a flag indicating whether to use an MLP has a second value, an MLP-based point synthesis method may be performed. The specific details of the point synthesis step are as described above and are omitted below.

[0211] 4) Octree decoding step

[0212] In the octree decoding stage, the decoded features of the current octree level Based on the occupancy map of the parent octree level, the (lossless) decrypted occupancy map of the current octree level can be obtained.

[0213] The octree decoding step can be performed for each octree level configured for lossless-based decoding.

[0214] The mode of the octree decoding step can be determined based on information related to bit depth. The mode of the octree decoding step may include a current level autoregressive mode and a parent level autoregressive mode.

[0215] The specific details of the octree decoding step are as previously described and will be omitted below.

[0216] Region Adaptive Hierarchical Transform (RAHT)-based attribute decoding may include intra-point cloud attribute decoding and inter-point cloud attribute decoding.

[0217] In the case of intra-point cloud attribute decoding, the decoded point cloud attribute can be obtained from a RAHT-based Attribute Decoding Unit based on the attribute bitstream and the decoded point cloud geometry.

[0218] In the case of inter-point cloud attribute decoding, attribute predictor information can be generated in an attribute predictor generation unit based on the decoded point cloud geometry and attribute reference frame. For example, if the attribute is color, attribute predictor information can be generated in a recoloring unit. The attribute predictor information is shared as an additional input to a RAHT-based attribute decoding unit of intra-decoding, and the decoded point cloud attribute can be obtained in the RAHT-based attribute decoding unit of intra-decoding.

[0219] After the point cloud geometry slice is decoded, whether to perform an inverse scaling process can be determined based on a coordinate transformation flag indicating that the point cloud geometry is transformed from the world coordinate system to the coding coordinate system, and scaling ratio information indicating the amount of scaling from the shifted coding coordinate system to the scaled coding coordinate system.

[0220] In one embodiment, when the coordinate transformation flag is 1 and the scaling amount of the scaling ratio information is less than 1, an inverse scaling process can be performed.

[0221] In the inverse scaling process, the decoded point cloud geometry, which is a set of 3D point locations in a coded coordinate system Based on the scaling coefficient specified by the , and scaling ratio information, an unscaled point cloud geometry, which is a set of 3D point locations in a transformed coding coordinate system, can be obtained.

[0222] In addition, based on reverse scaling type information indicating the type of reverse scaling process, the reverse scaling process can be performed by a simple reverse scaling process or an enhanced reverse scaling process.

[0223] The inverse transform process can be called when the point cloud geometric slice is decoded and the coordinate transformation flag has a value equal to 1. Additionally, if the scaling ratio information indicates a value less than 1, the inverse transform process can be called additionally after the inverse scaling process is completed.

[0224] The inverse transformation process can move the point cloud geometry of the shifted coding coordinate system back to the world coordinate system.

[0225] The assembly process is called after all point cloud slices are decoded, where information related to the assembly process indicates that the point cloud frame is divided into slices and coded individually, and the point cloud frame can be obtained by assembling all decoded point cloud slices.

[0226] Figure 13 illustrates an example of the overall encoding process.

[0227] The encoding process of the present disclosure may include learning-based point cloud geometric encoding and Region Adaptive Hierarchical Transform (RAHT)-based attribute encoding.

[0228] Both geometry encoding and attribute encoding can have intra mode and inter mode as encoding modes.

[0229] Geometric coding is a neural network-based coding method in which neural network weights can be determined before the geometric coding process is executed. Model identifier information representing the neural network weights can be encoded directly or indirectly as a bitstream. Alternatively, the neural network weights can be pre-defined values ​​between the encoder and the decoder.

[0230] For octree level i=1, eight binary occupancy values ​​can be encoded in a raw coding unit consisting of a single byte. The eight voxels can be arranged according to raster scanning order. The bits of the byte in the raw coding unit can be indexed from the Most Significant Bit (MSB) to the Least Significant Bit (LSB). Each bit can represent the occupancy status of the corresponding voxel. At this level, the residual feature, Tensor<0, 8>, can be set as a constant dummy feature where all elements are 1.

[0231] In the case of intra-point cloud geometry encoding, the geometry of the point cloud encoded in the Learning-based Geometry Encoding Unit can be constructed and encoded into the bitstream. Intra-point cloud geometry encoding can be executed for each octree level, starting from octree level i=2.

[0232] In the case of inter-point cloud geometry encoding, geometry predictor information is generated in the Learning-based Motion Decoding Unit, and a motion bitstream can be encoded based on the geometry predictor information and a reference frame. The geometry predictor information is shared as an additional input with the Learning-based Geometry Encoding Unit of intra-coding, allowing the geometry of the encoded point cloud to be constructed in the Learning-based Geometry Encoding Unit of intra-coding. Inter-point cloud geometry encoding can be executed for each octree level, starting from octree level i=2.

[0233] In the case of intra-point cloud attribute encoding, the point cloud attributes encoded in the RAHT-based Attribute Decoding Unit are configured, and the attribute bitstream can be encoded based on the encoded point cloud attributes and the encoded point cloud geometry.

[0234] In the case of inter point cloud attribute encoding, attribute predictor information can be generated in a recoloring unit based on the encoded point cloud geometry and attribute reference frame. The attribute predictor information is shared as an additional input to a RAHT-based attribute encoding unit of intra encoding, and the encoded point cloud attributes are constructed in the RAHT-based attribute encoding unit of intra encoding, and an attribute bitstream can be encoded based on the encoded point cloud attributes.

[0235] The specific details of the encoding process can correspond to the specific details of the decoding process.

[0236] The (geometric) transformation process can be called as a preprocessing step. The transformation process can convert a point cloud in the world coordinate system into a transformed coding coordinate system. Subsequently, the decoder can perform a process to convert the output back to the world coordinate system.

[0237] After the transformation process is completed, whether to perform a (geometric) scaling process can be determined based on a coordinate transformation flag indicating that the point cloud geometry is transformed from the world coordinate system to the coding coordinate system, and scaling ratio information indicating the amount of scaling from the transformed (shifted) coding coordinate system to the scaled coding coordinate system.

[0238] In one embodiment, a scaling process can be performed when the coordinate transformation flag is 1 and the scaling amount of the scaling ratio information is less than 1.

[0239] In the scaling process, point cloud geometry, which is a set of 3D point locations in the transformed coding coordinate system, Based on the scaling coefficient specified by the , and scaling ratio information, the scaled point cloud geometry of the transformed coding coordinate system can be obtained.

[0240] The point cloud geometry intra-coding process (geometry intra-coding process) can be called at each octree level when the current point cloud slice is determined to be in geometry intra mode (e.g., when the unit type is determined to be Geometry Intra-Slice).

[0241] Additionally, the geometry intra-coding process may be called for each octree level if the current octree level is greater than the threshold octree level, even if the current point cloud slice is determined to be in geometry inter-mode (e.g., even if the unit type is determined to be Geometry Inter-Slice). Here, the threshold octree level can be encoded directly or indirectly as a bitstream.

[0242] The geometric intra-coding process can be performed in the learning-based geometric encoding unit of Fig. 13.

[0243] The unit types of the present disclosure may include types of Geometry Intra-Slice, Geometry Inter-Slice, Attribute Intra-Slice, and Attribute Inter-Slice.

[0244] Types of geometric intra-coding processes may include lossless-based encoding and lossy-based encoding.

[0245] Figure 14 illustrates examples of lossless-based encoding and lossy-based encoding of the geometric encoding process.

[0246] The loss-based encoding process may include 1) an ancestor occupancy analysis step, 2) a feature extraction step, 3) a feature encoding step, and 4) a point synthesis step.

[0247] The lossless-based encoding process may include 1) an ancestor occupancy analysis step, 2) a feature extraction step, 3) a feature encoding step, and 5) an octree encoding step.

[0248] That is, in loss-based encoding and lossless-based encoding processes, 1) the ancestor occupancy analysis step, 2) the feature extraction step, and 3) the feature encoding step may be common processes.

[0249] 1) Ancestor Occupancy Analysis Stage

[0250] In the ancestor occupancy analysis step, if occupancy information of the ancestor octree levels (i-1, i-2, i-3, i-4) of the current octree level is available, ancestor features are based on that information. You can obtain.

[0251] The ancestor occupancy analysis step may be an initialization step performed before the 3) feature encoding step.

[0252] The ancestor occupancy analysis step may include an ancestor feature initialization step, a feature mixing step, and an adaptive affine step. The specific details of each step are identical to those in the geometric decoding process and are therefore omitted below.

[0253] 2) Feature extraction step

[0254] Voxel occupancy map of the current octree level of the input point cloud and style conditions f style Based on this, features representing local geometric information of the current octree level of the input point cloud You can obtain.

[0255] The feature extraction step can be performed in different ways based on a flag indicating whether a MultiLayer Perceptron (MLP) is used.

[0256] For example, when a flag indicating whether to use an MLP has a first value, the voxel occupancy map of the parent octree level of the input point cloud It can be provided as additional input.

[0257] For example, if a flag indicating whether to use an MLP has a second value, a sparse 3D convolution-based feature extraction method can be performed.

[0258] 3) Feature encoding step

[0259] In the feature encoding stage, features representing geometric information for the current octree level , ancestor characteristics of the current octree level , constant dummy prediction feature F pred , decoded residual features of the parent octree level Based on, the characteristic bitstream for the current octree level , current octree level decrypted features and decoded residual features of the current octree level You can obtain.

[0260] Constant dummy prediction feature F pred All elements of can be equal to 1. Constant dummy prediction features can be used to match geometric feature encoding for intra-coding with inter-coding.

[0261] Figure 15 illustrates an example of a feature encoding step.

[0262] The feature encoding step may include a superprescription synthesis step, a conditional encoding step, an arithmetic encoding step, and a conditional decoding step.

[0263] The superpredictive synthesis step can synthesize feature variance parameters based on decoded residual features of the parent octree level.

[0264] The conditional encoding step can conditionally encode the features of the current octree level based on the residual features of the current octree level, using the ancestor features and constant dummy prediction features of the current octree level as conditions.

[0265] The arithmetic encoding step can obtain the decoded residual features and feature bitstream of the current octree level based on the feature dispersion parameters obtained in the superprescription synthesis step and the residual features of the current octree level.

[0266] The conditional decoding step can conditionally decode the decoded features of the current octree level based on the decoded residual features of the current octree level, using the ancestor features and constant dummy prediction features of the current octree level as conditions.

[0267] 4) Point composite step

[0268] In the point synthesis step, the decoded features of the current octree level Based on the occupancy map of the parent octree level, the (lossy) decrypted occupancy map of the current octree level can be obtained.

[0269] The point synthesis step can be performed in different ways depending on whether a MultiLayer Perceptron (MLP) is used.

[0270] For example, when an MLP is used, a sparse 3D convolution-based point synthesis method is performed, and when an MLP is not used, an MLP-based point synthesis method can be performed.

[0271] Information indicating whether the MLP is used can be encoded in the bitstream.

[0272] The specific details of the sparse 3D convolution-based point synthesis method and the MLP-based point synthesis method are as described above and will be omitted below.

[0273] 5) Octree encoding step

[0274] In the octree encoding stage, the decoded features of the current octree level , based on the occupancy map of the parent octree level and the occupancy map of the current octree level, the occupied bitstream for the current octree level You can obtain.

[0275] The octree encoding step can be performed for each octree level configured for lossless-based encoding.

[0276] The mode of the octree encoding stage can be determined based on bit depth information. The mode of the octree encoding stage may include a current level autoregressive mode and a parent level autoregressive mode.

[0277] Depending on the mode, intermediate processes of the octree encoding step can be performed at the resolution of the current octree level or the parent octree level.

[0278] The octree encoding step may include an initialization step, a feature aggregation step, and a occupancy encoding step.

[0279] In the initialization phase, depending on the mode of the octree encoding phase, the occupancy map of the parent octree level is expanded or upsampled to the occupancy map of the current octree level to obtain the initial occupancy map of the current octree level. It can generate.

[0280] In the case of the current level autoregressive mode, during the initialization phase, the occupancy map of the parent octree level is upsampled to the occupancy map of the current octree level to obtain the initial occupancy map of the current octree level. , Tensor<i, 1> You can obtain. The initial occupancy map of the current octree level has 8 × voxelCount (number of occupied voxels in the parent octree level) points, and all child voxels of an occupied parent voxel can be set to 0.

[0281] In the case of parent-level autoregressive mode, during the initialization phase, the occupancy map of the parent octree level is extended to the occupancy map of the current octree level, and the initial occupancy map of the current octree level , Tensor<i-1, 8> You can obtain. The initialization step initializes all channels to 0. Repeat the channel 8 times You can generate 8 child voxels. The 8 child voxels are mapped to 8 channels indexed from 0 to 7, and the order can be (0,0,0), (1,0,0), (0,1,0), (1,1,0), (0,0,1), (1,0,1), (0,1,1), (1,1,1).

[0282] In the feature aggregation stage, the decoded features of the current octree level Improves aggregated features according to mode You can obtain.

[0283] In the case of the current level autoregressive mode, the decoded features of the current octree level are used in the feature aggregation step. Improves, aggregates the improved features, and upsamples the aggregated features You can obtain.

[0284] In the case of parent-level autoregressive mode, the decoded features of the current octree level are used in the feature aggregation step. Aggregated features obtained by aggregating without changing the resolution (having the resolution of the parent octree level). You can obtain.

[0285] Figure 16 illustrates an example of a possession encoding step.

[0286] The occupancy encoding step can be called repeatedly 8 times to encode the occupancy map of octree level i. k, representing the numbering of the repetitions, can range from 1 to 8. The repetitions for 8 octants can follow the order of (0,0,0), (1,0,0), (0,1,0), (1,1,0), (0,0,1), (1,0,1), (0,1,1), (1,1,1).

[0287] The occupy encoding step for the k-th iteration can be performed by i) an occupancy buffer, ii) a sibling feature initializer, iii) a sibling feature mixing unit, iv) a probability estimator, v) an occupancy masking unit, vi) an occupancy serializer, and vii) an arithmetic encoding unit.

[0288] i) Occupied buffer

[0289] The occupancy buffer contains all sibling occupancy maps decrypted in the previous iteration of the current octree level. (from 1 to k-1) can be stored and provided so that it can be referenced in the next decoding iteration. Additionally, a mask, which is a decoding mask indicating whether a child voxel is decoded, can be managed. Also, for the first iteration (k=1), the occupied buffer can be initialized.

[0290] ii) Sibling characteristic initialization section

[0291] In the sibling feature initialization section, all sibling occupancy maps decrypted in the previous iteration Initialize features based on to initial sibling features You can obtain.

[0292] In the case of the current level autoregressive mode, the sibling feature initializer performs initialization using a 5-dimensional vector to initialize the sibling features It is possible to obtain. In this case, 1 dimension represents a scalar value for the octree level, 3 dimensions represent a scalar value for the 3D position, and 1 dimension represents a binary value of the decoding mask map.

[0293] In the case of parent-level autoregressive mode, the input in the sibling feature initialization section Initialize identically to the initial sibling characteristics You can obtain.

[0294] iii) Brothers' Characteristics Mixing Department

[0295] In the sibling characteristic mixing section, the aggregated characteristics Characteristics of early siblings Mixed characteristics by mixing You can obtain.

[0296] The mixing process of the sibling feature mixing section may vary depending on whether the mode is the current level autoregressive mode or the parent level autoregressive mode.

[0297] iv) Probability Estimator

[0298] In the probability estimation unit, mixed features Based on, the occupancy probability of voxels at the current octree level for each occupied parent voxel It can predict.

[0299] In the case of the current level autoregressive mode, the probability estimation unit uses a multi-perceptron (MLP) network to estimate the occupancy probabilities of voxels at the current octree level having one channel. It can be predicted. In this case, the channel size of the MLP layer used may be determined differently by bit depth-related information.

[0300] In the case of parent-level autoregressive mode, the probability estimator uses convolution layers, ReLU layers, and Inception ResNet (IRN) layers to estimate the occupancy probabilities of the voxels at the current octree level. It can predict.

[0301] v) Occupancy masking part

[0302] Upsampled occupancy map of the current octree level based on the parent octree level Based on, the k-th occupying map of the current iteration You can obtain.

[0303] vi) Possession serialization part

[0304] In the possession serialization section, the possession map of the current iteration Based on, binary sequence You can obtain.

[0305] vii) Arithmetic encoding unit

[0306] In the arithmetic encoding section, the binary value uses the predicted probability for the current octant of the current octree level. A sequence of can be encoded to generate an occupied bitstream.

[0307] The point cloud geometry inter-coding process can be called for each octree level when the current point cloud slice is determined to be in geometry inter-mode (i.e., the unit type is determined to be Geometry Inter-Slice) and the current octree level is less than or equal to a critical octree level.

[0308] The point cloud geometry inter-coding process can be performed with a combination of the learning-based geometry encoding unit, the learning-based motion encoding unit, and the reference frame unit of FIG. 13.

[0309] The point cloud geometry intercoding process may include a motion coding process and a main coding process.

[0310] In the motion encoding process, the motion between the reference frame and the current frame and the reference frame can be estimated, and a predictor for the current frame can be obtained. This may correspond to the learning-based motion encoding unit and the reference frame unit of FIG. 13.

[0311] In the main encoding process, geometric encoding can be performed based on a predictor for the current frame. This may correspond to the learning-based geometric encoding unit of FIG. 13.

[0312] The global motion evaluation and compensation process can be called as a preprocessing process at every octree level before calling the inter-coding process.

[0313] The inputs and outputs of the global motion evaluation and compensation process can belong to a transformed coding coordinate system.

[0314] 1) First, the current point cloud frame and reference point cloud frame The coordinates of each point in can be transformed based on the point cloud bounding box center (c, c, c). The above transformation is and It can be performed by.

[0315] The position of the bounding box center in the transformed coding coordinate system is It can be defined as follows.

[0316] 2) Next, the global motion represented by the rotation matrix R and the translation vector T is determined, and the reference point cloud frame using the rotation matrix R and translation vector T It can be converted as follows.

[0317] 3) Finally, reference point cloud frame A transformation can be applied to move it back to the transformed coding coordinate system, thereby obtaining a global motion-compensated reference frame. The above transformation is It can be performed by.

[0318] The learning-based motion encoding process performed in the learning-based motion encoding unit can be called for each octree level for a slice at the start of the point cloud geometry inter-decoding process.

[0319] The learning-based motion encoding process can generate predicted features to support the main encoding process.

[0320] FIG. 17 illustrates an example of a learning-based motion encoding process.

[0321] The learning-based motion encoding process may include a feature extraction step, a reference feature extraction step, and an inter-predictor extraction step.

[0322] In the feature extraction step, features of the current octree level can be obtained by utilizing style conditions and the occupancy map of the current octree level. Additionally, if a MultiLayer Perceptron (MLP) is used, features of the current octree level can be obtained by additionally utilizing the current occupancy map of the previous octree level.

[0323] In the reference feature derivation step, reference features of the current octree level can be obtained based on style conditions and the reference occupancy map of the reference point cloud for the current octree level. Additionally, when a MultiLayer Perceptron (MLP) is used, reference features of the current octree level can be obtained by additionally utilizing the reference occupancy map of the reference point cloud of the previous octree level.

[0324] FIG. 18 illustrates an example of the inter-predictor derivation step.

[0325] The inter-predictor derivation step may include a motion feature buffer management step, a super-predicted synthesis step, a motion evaluation step, an arithmetic encoding step, and a motion compensation step.

[0326] The motion feature buffer management step can store and provide decoded motion features. For example, the motion feature buffer management process can provide the decoded motion features of the parent octree level to the superpre-synthesis step for the current octree level.

[0327] The superpredictive synthesis step can synthesize feature variance parameters using motion features of the parent octree level. The superpredictive synthesis step can operate identically to the previously described superpredictive synthesis step. However, the input can be the decoded motion features of the parent octree level instead of the decoded residual features of the parent octree level.

[0328] The motion evaluation step can obtain motion features of the current octree level based on the features of the current point cloud of the current octree level, the reference features of the reference point cloud of the current octree level, and style conditions.

[0329] The arithmetic encoding step can operate in the same manner as the previously described arithmetic encoding step. However, the residual features of the input current octree level can be replaced with the motion features of the current octree level. That is, the arithmetic encoding step can obtain a motion bitstream by arithmetic encoding the motion features of the current octree level based on the feature distribution parameters obtained in the superprescription synthesis step.

[0330] In the motion compensation step, based on the decoded motion features of the current octree level, the reference features of the current octree level, and the style conditions, the predicted features of the current octree level You can obtain.

[0331] The main encoding process can be called at each octree level of the slice after the learning-based motion encoding process is completed.

[0332] The main encoding process is performed in the learning-based geometric encoding unit of FIG. 13, and the unit may be a neural network shared with the geometric intra-coding process.

[0333] Similar to the geometric intra-coding process, the types of main coding processes may include lossless-based coding and lossy-based coding.

[0334] Compared to the geometric intra-coding process, the difference in the main coding process may be that it uses predicted features generated from the learning-based motion coding process instead of dummy features.

[0335] The loss-based encoding process may include 1) an ancestor occupancy analysis step, 2) a feature extraction step, 3) a feature encoding step, and 4) a point synthesis step.

[0336] The lossless-based encoding process may include 1) an ancestor occupancy analysis step, 2) a feature extraction step, 3) a feature encoding step, and 5) an octree encoding step.

[0337] That is, in loss-based encoding and lossless-based encoding processes, 1) the ancestor occupancy analysis step, 2) the feature extraction step, and 3) the feature encoding step may be common processes.

[0338] 1) Ancestor Occupancy Analysis Stage

[0339] In the ancestor occupancy analysis step, if occupancy information of the ancestor octree levels (i-1, i-2, i-3, i-4) of the current octree level is available, ancestor features are based on that information. You can obtain.

[0340] The ancestor occupancy analysis step may be an initialization step performed before the 3) feature encoding step.

[0341] The ancestor occupancy analysis step may include an ancestor feature initialization step, a feature mixing step, and an adaptive affine step. The specific details of each step are identical to those in the geometric decoding process and are therefore omitted below.

[0342] 2) Feature extraction step

[0343] Voxel occupancy map of the current octree level of the input point cloud and style conditions f style Based on this, features representing local geometric information of the current octree level of the input point cloud You can obtain.

[0344] The feature extraction step can be performed in different ways depending on whether a MultiLayer Perceptron (MLP) is used.

[0345] For example, when an MLP is used, the voxel occupancy map of the parent octree level of the input point cloud It can be additionally provided as input.

[0346] For example, if an MLP is not used, a sparse 3D convolution-based feature extraction method can be performed.

[0347] 3) Feature encoding step

[0348] In the feature encoding stage, features representing geometric information for the current octree level , ancestor characteristics of the current octree level , predictive features of the current octree level , decoded residual features of the parent octree level Based on, the characteristic bitstream for the current octree level , current octree level decrypted features and decoded residual features of the current octree level You can obtain.

[0349] 4) Point composite step

[0350] In the point synthesis step, the decoded features of the current octree level Based on the occupancy map of the parent octree level, the (lossy) decrypted occupancy map of the current octree level can be obtained.

[0351] The point synthesis step can be performed in different ways depending on whether a MultiLayer Perceptron (MLP) is used.

[0352] For example, when an MLP is used, a sparse 3D convolution-based point synthesis method is performed, and when an MLP is not used, an MLP-based point synthesis method may be performed. The specific details of each method have been described above and are omitted below.

[0353] 5) Octree encoding step

[0354] In the octree encoding stage, the decoded features of the current octree level , based on the occupancy map of the parent octree level and the occupancy map of the current octree level, the occupied bitstream for the current octree level You can obtain.

[0355] The octree encoding step can be performed for each octree level configured for lossless-based encoding.

[0356] The mode of the octree encoding stage can be determined based on bit depth information. The mode of the octree encoding stage may include a current level autoregressive mode and a parent level autoregressive mode.

[0357] Depending on the mode, intermediate processes of the octree encoding step can be performed at the resolution of the current octree level or the parent octree level.

[0358] The octree encoding step may include an initialization step, a feature aggregation step, and a occupancy encoding step.

[0359] The specific details of the octree encoding step are as previously described and will be omitted below.

[0360] Region Adaptive Hierarchical Transform (RAHT)-based attribute encoding may include intra-point cloud attribute encoding and inter-point cloud attribute encoding.

[0361] In the case of intra-point cloud attribute encoding, an attribute bitstream can be encoded in a RAHT-based Attribute Encoding Unit based on input point cloud attributes and encoded point cloud geometry.

[0362] In the case of inter-point cloud attribute encoding, attribute predictor information can be generated in an attribute predictor generation unit based on the restored point cloud geometry and attribute reference frame. For example, if the attribute is color, attribute predictor information can be generated in a recoloring unit. The attribute predictor information is shared as an additional input to an intra-coding RAHT-based attribute encoding unit, and an attribute bitstream can be obtained in the intra-coding RAHT-based attribute encoding unit.

[0364] Figure 19 illustrates an example of the structure of an artificial intelligence-based input image compression framework.

[0365] In one embodiment, the encoder can encode data by gradually reducing the resolution of the input X0 (downsampling or downscaling).

[0366] As input images, 2D images, 3D volumes, point cloud data, etc. can be used.

[0367] In performing lossy compression of an input image, layered encoding can be performed on a point cloud downsampled m times using an entropy model.

[0368] The above m may be an integer greater than or equal to 1. For example, if m is an integer greater than 1, encoding can be performed layer by layer through an entropy model for every downsampled point cloud.

[0369] In performing lossless compression of an input image, lossless compression can be performed on the geometry of a point cloud downsampled n times.

[0370] The above n may be an integer greater than or equal to 1. For example, if n is an integer greater than 1, encoding can be performed layer by layer through an entropy model for every downsampled point cloud geometry.

[0371] The above lossless compression can be performed using a standardized codec such as MPEG's g-pcc.

[0372] In the above lossless compression, a framework similar or identical to that of lossy compression can be used, and additionally, encoding of occupancy information can be performed.

[0373] In one embodiment, the decoder decodes geometric and feature information Data can be restored by gradually increasing the resolution (upsampling or upscaling).

[0374] Data can be restored by utilizing geometric information restored from a bitstream generated through lossless compression and feature information generated through an entropy model during the lossy compression process.

[0375] Figure 20 illustrates an example of a framework for hierarchical lossy compression and lossless compression.

[0376] In one embodiment, lossless compression can additionally perform compression on occupancy information based on occupancy probability.

[0377] The above occupancy probability can be generated from feature information generated through a Feature Encoder and a Feature Decoder, which are similar modules in a hierarchical lossy compression framework, as shown in FIG. 20.

[0378] Compression of occupancy information based on the above occupancy probability can be performed using an Octree Encoder and an Octree Decoder as shown in Fig. 20.

[0379] In the above embodiment, occupancy information can be divided and compression can be performed sequentially. However, if compression is always performed in a single fixed order, encoding performance may be degraded.

[0380] Accordingly, encoding performance can be improved by explicitly or implicitly selecting an adaptive order based on an initial probability distribution, or by explicitly or implicitly selecting the best order among any defined n orders. Here, explicit selection may mean that the corresponding information is signaled from the bitstream.

[0381] The higher level of the present disclosure may mean a resolution greater than that of the current point cloud.

[0382] The above higher level can be achieved through upsampling or ConvTranspose of the current level.

[0383] A lower level of the present disclosure may mean a resolution smaller than the current point cloud.

[0384] The above lower level can be achieved through downsampling of the current level or Conv with stride > 1, etc.

[0385] The networks within the embodiments of the invention disclosed herein may be networks composed of activation functions and combinations of MLPs and CNNs, as well as Convolution and MLPs for 3D data such as Minkowski Engine used in the field of point cloud compression.

[0386] The meaning of 'coding' in the present disclosure may be 'encoding' or 'decoding'.

[0388] FIG. 21 illustrates an example of a flowchart of a geometric occupancy information compression method for point cloud encoding / decoding of the present disclosure.

[0389] The geometric occupancy information compression method for point cloud encoding / decoding of the present disclosure may largely include: [1] a feature information generation step of generating downsampled feature information through a downsampling and feature generation module (Point to Feature) for input point cloud data; [2] a feature information coding step of encoding / decoding feature information through a feature encoder and a feature decoder for the generated feature information; and [3] an occupancy information coding step of generating geometric occupancy probabilities from the feature information and encoding / decoding original geometric occupancy information based thereon.

[0390] In the present disclosure, the unit in which the feature information generation step is performed may be referred to as the feature information generation unit, the unit in which the feature information coding step is performed may be referred to as the feature information coding unit, and the unit in which the occupancy information coding step is performed may be referred to as the occupancy information coding unit.

[0392] FIG. 22 illustrates an example of a feature information generation unit of a geometric occupancy information compression device for point cloud subcoding / decoding.

[0393] [1] In the feature information generation step, the resolution of the input point cloud geometry is lowered through downsampling, and the features of the geometry can be generated through the Point to Feature module.

[0394] The above downsampling can be performed through processes such as quantization or pooling on geometric values.

[0395] The above feature generation module can use various network-based encoding techniques used in the field of Autoencoder-based point cloud compression.

[0396] FIG. 23 illustrates an example of a feature information coding section and a feature encoder of a geometric occupancy information compression device for point cloud encoding / decoding.

[0397] [2] In the feature information coding stage, the generated feature information can be encoded and decoded through a feature encoder and a feature decoder.

[0398] The right figure of Fig. 23 illustrates an example of a feature encoder.

[0399] In the feature encoder, residual features can be generated and encoded / decoded by additionally utilizing lower-level layer features and coded lower-level layer residual features for the features generated through the feature information generation unit.

[0400] Features of a lower-level layer can be generated through a feature generation module, such as Point to Feature of the feature information generation unit, using geometries that are downsampled up to n times more than the features of the current-level layer. Here, n can be a natural number greater than or equal to 1, such as 1, 2, 3, 4, 5, 6. The value of n can be signaled as a bitstream. Alternatively, the value of n can be a predefined value.

[0401] Residual features of the current level layer can be generated from features of the current level layer and features of lower level layers through a residual feature generation network.

[0402] The residual features of the coded lower-level layer may be lower-level residual features that have already been encoded / decoded through the Feature Encoder and Feature Decoder of [2] due to the hierarchical coding structure. Additionally, the residual features of the coded lower-level layer can be utilized for entropy model encoding for the residual features of the current layer through an additional superprediction model generation network, as shown in FIG. 23. For example, the output of the superprediction model generation network, which takes the residual features as input, can be utilized for entropy model encoding for the residual features of the current layer.

[0403] In summary, the residual feature generation network can output the residual feature of the current level layer by taking the features of the current level layer and the features of the lower level layer as input. The superprediction distribution model generation network can output probability information by taking the coded residual feature of the lower level layer as input, and said probability information can be utilized in an entropy encoder.

[0404] In the entropy encoder, entropy model encoding is performed based on the residual features of the current level layer and the probability information to output the coded features of the current level layer and the feature bitstream for the current level layer.

[0405] FIG. 24 illustrates an example of a feature decoder.

[0406] In the feature decoder, the features of the current level layer can be restored by additionally utilizing the features of the lower level layer and the residual features of the coded lower level layer, along with the residual features of the coded current level layer obtained by decoding the feature bitstream.

[0407] Features of a lower-level layer can be generated through a feature generation module, such as Point to Feature in the feature information generation unit, using geometries that are downsampled up to n times more than the features of the current-level layer. Here, n can be a natural number greater than or equal to 1, such as 1, 2, 3, 4, 5, 6. The value of n can be signaled via a bitstream. Alternatively, the value of n can be a predefined value.

[0408] The residual features of the coded lower-level layer may be lower-level residual features that have already been encoded / decoded through the Feature Encoder and Feature Decoder of [2] due to the hierarchical coding structure, and these residual features can be used to decode the entropy model for the residual features of the coded current layer through an additional superprediction model generation network. For example, the output of the superprediction model generation network that takes the residual features as input can be used to decode the entropy model for the residual features of the coded current layer.

[0409] As shown in Fig. 24, the features of the current level layer can be restored using the residual features of the lower level layer coded as well as the features of the lower level layer and the residual features of the current level coded as well.

[0410] In summary, in a hyperprediction model generation network, probability information can be output using residual features of a coded lower-level layer as input, and said probability information can be utilized in an entropy decoder.

[0411] In the entropy decoder, the feature bitstream can be decoded based on the above probability information to output the residual features of the coded current level layer.

[0412] In a feature recovery network, residual features of the current level layer can be output using coded residual features of the current level layer and features of the lower level layer as input.

[0414] FIG. 25 illustrates an example of a occupancy information coding section of a geometric occupancy information compression device for point cloud encoding / decoding and an example of an octree encoder.

[0415] The right figure of Fig. 25 illustrates an example of an octree encoder.

[0416] In the occupancy information coding section, [3] the occupancy probability of the geometry can be generated from the feature information, and based on this, the occupancy information of the original geometry can be encoded / decoded.

[0417] In the octree encoder of the occupancy information coding unit, an occupancy probability for the geometry of the current level can be generated from the features generated through the feature information coding process as shown in FIG. 25, and can be used to encode the occupancy information of the original geometry of the same level. The specific details of the octree encoder are described below.

[0418] From the restored current level features, aggregate features of the current level that map to the geometry of the current level can be generated through a feature aggregation network. Here, the restored current level features may be features restored based on the residual features of the current level in the feature information coding section.

[0419] The probability of occupying each geometry of the current level can be predicted from the aggregate features of the current level through a probability distribution prediction network. In this case, the probability of occupying can have a value between 0 and 1.

[0420] Based on the current level geometry-specific occupancy probability and occupancy information and the lower level hierarchy's occupancy information, a geometry bitstream can be generated for each geometry of the current level through binary arithmetic encoding.

[0421] In sequentially performing binary arithmetic coding for m partitioned groups, the coding efficiency can be improved by updating the probability distribution for the binary arithmetic coding of the next group each time. Here, m is a natural number greater than or equal to 2, which can be 8. The method for sequential coding below may be as follows.

[0422] In encoding using the geometric occupancy probability and occupancy information (0 or 1) of the current level, the 8 can be divided into 8 groups and encoded sequentially according to a predetermined arbitrary order.

[0423] For example, the above arbitrary order may be one of the methods such as Hilbert curve scan order, Morton scan order, Peano Curve Order, and Spiral Scan Order.

[0424] For each scan order, you can define which of the x, y, and z coordinates to scan.

[0425] In encoding using the geometric occupancy probability and occupancy information (0 or 1) of the current level, the coding can be performed sequentially by dividing into 8 groups and adaptively defining the order for the 8 groups using initial probabilities. Here, the initial probabilities may include an occupancy probability distribution, an occupancy probability entropy, an occupancy probability binary cross-entropy, etc.

[0426] For example, 8 occupancy probability distributions can be obtained by averaging each of the 8 groups above, and encoding can be performed in order of smallest or largest occupancy probability distributions.

[0427] For example, 8 occupancy probability entropies can be obtained by calculating the entropy for each of the 8 groups above, and encoding can be performed in order of smallest occupancy probability entropy.

[0428] For example, binary cross-entropy can be calculated for each of the 8 groups above to obtain 8 occupancy probability binary cross-entropys, and encoding can be performed in order of smallest occupancy probability binary cross-entropy.

[0429] In encoding using the geometric occupancy probability and occupancy information (0 or 1) of the current level, the data can be divided into 8 groups, and n fixed random order sets can be constructed for the 8 groups. For each of the n random order sets, the amount of bits generated for the 8 groups can be calculated, and the order of the order set with the minimum generation can be used for encoding.

[0430] Encoding multiple groups sequentially according to an arbitrary order can be performed by dividing the multiple groups according to an arbitrary order of n arbitrary order sets, and then sequentially encoding the divided groups according to an arbitrary order of the divided groups. For example, among 8 groups determined to use a first arbitrary order among n arbitrary order sets, the first group may be sequentially encoded in the first arbitrary order, and among 8 groups determined to use a second arbitrary order among n arbitrary order sets, the second group may be sequentially encoded in the second arbitrary order.

[0431] Here, n can be an integer greater than or equal to 2.

[0432] The composition of any set of orders can consist of a set of one or more methods such as the Hilbert curve scan order, Morton scan order, Peano curve order, and Spiral Scan order.

[0433] For each scan order, multiple sets can be configured depending on which position among the x, y, and z coordinates is prioritized for scanning.

[0434] Information regarding which of the arbitrary sequence set configurations was used for decoding can be transmitted as a flag. The number of bits required for the flag can be defined differently depending on the number of n.

[0435] For example, if n is 2, an additional 1 bit may be needed for the flag.

[0436] For example, if n is 3 or 4, 2 additional bits may be needed for the flag.

[0438] Figure 26 illustrates an example of an octree decoder.

[0439] In the octree decoder, the occupancy probability for the geometry of the current level can be generated from the features generated through the [2] process as shown in FIG. 26, and the occupancy information for the geometry of the current level can be restored through the decoded bitstream. The specific details of the octree decoder are described below.

[0440] From the restored current level features, aggregate features of the current level that map to the geometry of the current level can be generated through a feature aggregation network. Here, the restored current level features may be features restored based on the residual features of the current level in the feature information coding section.

[0441] For the aggregate feature of the current level, the occupancy probability for each geometry of the current level can be predicted through a probability distribution prediction network. In this case, the occupancy probability can have a value between 0 and 1.

[0442] Based on the current level geometry-specific occupancy probability, geometry bitstream, and occupancy information of the lower level layer, the occupancy information of the current level can be obtained through binary arithmetic decoding.

[0443] In sequentially performing binary arithmetic decoding for m partitioned groups, decoding efficiency can be improved by updating the probability distribution for the binary arithmetic decoding of the next group each time. Here, m is a natural number greater than or equal to 2, which can be 8. The method for sequentially decoding below may be as follows.

[0444] In decoding using the geometric occupancy probability and geometric bitstream of the current level, the data can be divided into 8 groups and decoded sequentially according to a predetermined arbitrary order.

[0445] For example, the above arbitrary order may be one of the methods such as Hilbert curve scan order, Morton scan order, Peano Curve Order, and Spiral Scan Order.

[0446] For each scan order, you can define which of the x, y, and z coordinates to scan.

[0447] In decoding using the geometric occupancy probability and geometric bitstream of the current level, decoding can be performed sequentially by dividing into 8 groups and adaptively defining the order for the 8 groups using initial probabilities. Here, the initial probabilities may include the occupancy probability distribution, occupancy probability entropy, occupancy probability binary cross-entropy, etc.

[0448] For example, 8 occupancy probability distributions can be obtained by averaging each of the 8 groups above, and decoding can be performed in order of smallest or largest occupancy probability distributions.

[0449] For example, 8 occupancy probability entropies can be obtained by calculating the entropy for each of the 8 groups above, and decoding can be performed in order of smallest occupancy probability entropy.

[0450] For example, the binary cross-entropy can be calculated for each of the 8 groups above to obtain 8 occupancy probability binary cross-entropies, and decoding can be performed in order of smallest occupancy probability binary cross-entropy.

[0451] In decoding using the geometric occupancy probability and geometric bitstream of the current level, it is possible to divide into 8 groups and construct n fixed random order sets for the 8 groups. Which of the n random order sets is used can be determined through additional bits transmitted within the bitstream, and decoding can be performed sequentially using the set.

[0452] Decoding multiple groups sequentially according to an arbitrary order can be performed by dividing the multiple groups according to an arbitrary order of n arbitrary order sets, and then sequentially decoding the divided groups according to an arbitrary order of the divided groups. For example, among 8 groups determined to use a first arbitrary order among n arbitrary order sets, the first group may be sequentially decoded according to the first arbitrary order, and among 8 groups determined to use a second arbitrary order among n arbitrary order sets, the second group may be sequentially decoded according to the second arbitrary order.

[0454] Flag information regarding whether the technology of the present disclosure is used and the use of the technology, as well as necessary parameters, can be transmitted and restored in accordance with the bitstream structure.

[0455] Here, the definitions of the above Sequence parameter set, Picture parameter set, and Geometry data unit may be as follows.

[0456] The sequence parameter set may be a set composed of high-level information required for transmission over the entire video sequence. The picture parameter set may be a set composed of high-level information required for transmission for each video. The geometry data unit may consist of bitstream information generated and required during the encoding process for each video.

[0457] According to one embodiment of the present disclosure, indicator information regarding whether the proposed technology is used can be recorded, stored, and transmitted in a sequence parameter set, SEI message, etc., and can also be recorded in other structures that facilitate encoding / decoding.

[0458] For example, indicator information regarding whether the technology is used can be transmitted in units of one frame.

[0459] For example, if there are additional division structures such as slices or tiles within a single frame, indicator information regarding the use of the technology can be transmitted on a slice or tile basis.

[0460] According to one embodiment of the present disclosure, model parameters required for the use of the technology can be recorded, stored, and transmitted in a Sequence parameter set, Picture parameter set, Geometry data unit, SEI message, etc.

[0461] According to one embodiment of the present disclosure, flag information required for the use of the technology can be recorded, stored, and transmitted in a Picture parameter set and a Geometry data unit.

[0462] In cases where a method of performing sequential encoding by adaptively defining is used, flag information can be transmitted in various ways.

[0463] For example, if lossless compression is encoded / decoded to perform adaptive execution at each level layer, flag information regarding whether adaptive execution is performed at each layer can be transmitted.

[0464] For example, if encoding / decoding is performed adaptively across all lossless compression level layers, flag information regarding whether adaptive execution is performed can be transmitted.

[0465] For example, if encoding / decoding is performed adaptively up to n times at the top level of lossless compression, values ​​of n or n-1 can be transmitted.

[0466] Here, n can be an integer greater than or equal to 1.

[0467] Flag information can be transmitted in various ways when using an arbitrary defined set of n sequences. Here, n can be an integer greater than or equal to 2.

[0468] For example, in the case of lossless compression where one of n arbitrary defined sequence sets is selected and used for decoding at each level layer, flag information indicating which set was used for decoding at each layer can be transmitted.

[0469] For example, if decoding is performed using the same one of n arbitrary defined sequence sets for all lossless compression level layers, flag information regarding which set was used for decoding can be transmitted.

[0470] For example, if the lossless compression top-level layer is encoded / decoded to use one of n arbitrary defined sequence sets only up to m times, values ​​of , m or m-1 can be transmitted.

[0471] Additionally, if decoding is performed by selecting and using one of n arbitrary defined sequence sets from the top level to the applied level layer, flag information indicating which set was used for decoding can be transmitted for each layer.

[0472] Additionally, if decoding is performed using the same one of n arbitrary defined sequence sets from the top level to the applied level hierarchy, flag information regarding which set was used for decoding can be transmitted.

[0473] Syntactic elements related to at least one of the indicators, indices, flags, etc., which are entropy-encoded in the encoder and entropy-decoded in the decoder, may utilize at least one of the following binaryization, debinarization, and entropy encoding / decoding methods.

[0474] Signed 0-th order Exp_Golomb binarization / debinarization method (se(v))

[0475] Signed k-th order Exp_Golomb binarization / debinarization method (sek(v))

[0476] 0-th order Exp_Golomb binarization / debinarization method for unsigned positive integers (ue(v))

[0477] k-th order Exp_Golomb binarization / debinarization method (uek(v)) for unsigned positive integers

[0478] Fixed-length binarization / debinarization method (f(n))

[0479] Truncated Rice binarization / inverse binarization method or Truncated Unary binarization / inverse binarization method (tu(v))

[0480] Truncated Binary Binarization / Debinaryization Method (tb(v))

[0481] Context-adaptive arithmetic encoding / decoding method (ae(v))

[0482] Byte-unit bit string (b(8))

[0483] Signed integer binary / debinary conversion method (i(n))

[0484] Unsigned positive integer binary / debinary conversion method (u(n))

[0485] In this case, u(n) may refer to a fixed-length binarization / debinarization method.

[0486] Unary Binary / Debinary Method

[0488] The encoding / decoding process of the target block is not limited to only one of the above embodiments, and a specific embodiment or at least one combination of the above embodiments may be applied to the encoding / decoding process of the target block.

[0490] The exemplary methods of the present disclosure are described as a series of operations for clarity of description, but this is not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. To implement the method according to the present disclosure, additional steps may be included in addition to the steps exemplified, steps excluding some steps and including the remaining steps, or steps excluding some steps and including additional steps.

[0491] The various embodiments of the present disclosure are not intended to list all possible combinations but to describe representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.

[0492] In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, it may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc.

[0493] The scope of the present disclosure may include software or machine-executable instructions (e.g., operating system, application, firmware, program, etc.) that enable an operation according to a method of various embodiments to be executed on a device or computer, and a non-transitory computer-readable medium on which such software or instructions, etc. are stored and executable on a device or computer.

Claims

Claim 1 A point cloud decoding method comprising: a step of obtaining an aggregate feature of the current level mapped to the geometry of the current level based on a feature of the current level restored using a residual feature of the current level; a step of predicting a geometry-specific occupancy probability of the current level based on the aggregate feature of the current level; and a step of performing binary arithmetic decoding using the geometry-specific occupancy probability of the current level and a geometry-specific bitstream of the current level. Claim 2 A point cloud decoding method according to claim 1, wherein the step of performing binary arithmetic decoding comprises: dividing the geometric occupancy probability of the current level and the geometric bitstream of the current level into a plurality of groups; and sequentially decoding the plurality of groups in any order. Claim 3 A point cloud decoding method according to paragraph 2, wherein the number of the plurality of groups is 8. Claim 4 A point cloud decoding method according to paragraph 2, wherein the arbitrary order is any one of the Hilbert curve scan order, Morton scan order, Peano Curve Order and Spiral Scan Order. Claim 5 A point cloud decoding method according to paragraph 2, wherein the arbitrary order is in order of smallest or largest occupancy probability distribution obtained through the average for each of the plurality of groups. Claim 6 A point cloud decoding method according to paragraph 2, wherein the arbitrary order is in order of smallest or largest occupancy probability entropy obtained by calculating the entropy for each of the plurality of groups. Claim 7 A point cloud decoding method according to paragraph 2, wherein the arbitrary order is in order of smallest or largest occupancy probability binary cross-entropy obtained by calculating the binary cross-entropy for each of the plurality of groups. Claim 8 A point cloud decoding method according to claim 2, wherein the step of sequentially decoding the plurality of groups in any order comprises: the step of dividing the plurality of groups according to any order of n arbitrary order sets; and the step of sequentially decoding the divided groups in any order of the divided groups. Claim 9 A point cloud encoding method comprising: a step of obtaining an aggregate feature of the current level mapped to the geometry of the current level based on a feature of the current level restored using a residual feature of the current level; a step of predicting a geometry-specific occupancy probability of the current level based on the aggregate feature of the current level; and a step of performing binary arithmetic encoding using the geometry-specific occupancy probability of the current level and the geometry-specific occupancy information of the current level. Claim 10 A point cloud encoding method according to claim 9, wherein the step of performing the binary arithmetic encoding comprises: a step of dividing the geometric occupancy probability of the current level and the geometric occupancy information of the current level into a plurality of groups; and a step of sequentially encoding the plurality of groups in an arbitrary order. Claim 11 A point cloud encoding method according to claim 10, wherein the number of the plurality of groups is 8. Claim 12 In claim 10, the above arbitrary order is any one of the Hilbert curve scan order, Morton scan order, Peano Curve Order and Spiral Scan Order, a point cloud encoding method. Claim 13 In claim 10, the above arbitrary order is a point cloud encoding method in which the occupancy probability distribution obtained through the average for each of the plurality of groups is in order of smallest or largest. Claim 14 A point cloud encoding method according to claim 10, wherein the above arbitrary order is in order of smallest or largest occupancy probability entropy obtained by calculating the entropy for each of the plurality of groups. Claim 15 In claim 10, the above arbitrary order is a point cloud encoding method in which the occupancy probability binary cross-entropy obtained by calculating the binary cross-entropy for each of the plurality of groups is in order of smallest or largest. Claim 16 A point cloud encoding method according to claim 10, wherein the step of sequentially encoding the plurality of groups in any order comprises: the step of dividing the plurality of groups according to any order of n arbitrary order sets; and the step of sequentially decoding the divided groups in any order of the divided groups. Claim 17 A non-transient computer-readable recording medium storing a bitstream generated by a point cloud encoding method, wherein the encoding method comprises: a step of obtaining an aggregate feature of a current level mapped to a geometry of the current level based on a feature of the current level restored using a residual feature of the current level; a step of predicting a geometry-specific occupancy probability of the current level based on the aggregate feature of the current level; and a step of performing binary arithmetic encoding using the geometry-specific occupancy probability of the current level and the geometry-specific occupancy information of the current level.

Images