TRISOUP node size per slice

Abstract

The per-slice trisoup node size enables flexibility when encoding point clouds. Instead of every block / node being the same size, the user or machine can dictate the block / node size so that regions of interest can have smaller node sizes for higher specificity in that region.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATION(S)

[0002] This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application Serial No. 63 / 043,116, filed on June 23, 2020, and entitled “TRISOUPNODE SIZE PER SLICE,” which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] The present invention relates to three-dimensional graphics, and more particularly to the coding and decoding of three-dimensional graphics. Background Art

[0004] A point cloud consists of a collection of points in 3D space, each associated with an (x, y, z) geometric position and attribute information (color, reflectivity, etc.). It can be used in several applications, such as virtual / augmented reality, immersive telepresence, autonomous driving, cultural heritage archiving, 3D free viewpoints, and geographic information systems. Point clouds can consist of thousands to billions of points. To make them workable, compression is essential.

[0005] MPEG initiated its Point Cloud Compression (PCC) standardization in 2017 with a Call for Proposals (CfP). Three categories of point clouds were identified: Category 1 for static point clouds, Category 2 for dynamic point clouds, and Category 3 for LiDAR sequences (dynamically acquired point clouds). Ultimately, two technologies were defined: G-PCC (Geometry-based PCC) for Category 1 and Category 3; and V-PCC (Video-based PCC) for Category 2. The first test models were developed in October 2017, one for G-PCC (TMC13) and the other for V-PCC (TMC2). Since then, the two test models have continued to evolve through technical contributions and collaboration, and the first version of the PCC standard specification is expected to be completed in 2020.

[0006] V-PCC first divides the point cloud into 3D connected regions called 3D patches. Each 3D patch is then projected onto a 2D patch. These projections act like virtual orthographic cameras, capturing specific parts of the point cloud. Combining these camera images generates a mosaic consisting of a collection of projected 2D patches. This process produces metadata information and a collection of up to three associated images: (1) an occupancy map; (2) a geometry image; and (3) several attribute images. The resulting 2D representation of the point cloud is then encoded using a 2D video encoder.

[0007] In G-PCC, geometry and attributes are encoded separately. Compressed geometry is typically represented as an octree from the root down to the leaf level of individual voxels. Alternatively, the representation can stop at the level of blocks larger than a voxel and use a triangle soup to approximate the surface within each leaf. Predictive geometry coding schemes are also available. Regarding attribute coding, there are three methods in G-PCC: Region Adaptive Hierarchical Transform (RAHT), Predictive Transform, and Lifting Transform. Summary of the Invention

[0008] Per-slice trisoup node size enables flexibility when encoding point clouds. Instead of each block / node having the same size, the user or machine can dictate the block / node size for each point cloud slice. For example, this feature enables region-of-interest encoding with smaller node sizes for greater specificity in that region.

[0009] In one aspect, a method programmed in a non-volatile memory of a device includes receiving point cloud information, segmenting the point cloud information into a plurality of slices, determining a plurality of node sizes, and encoding the point cloud using a node size from the plurality of node sizes for each slice in the plurality of slices. The node size may be determined based on a region of interest criterion. For the region of interest, the node size is reduced. The plurality of node sizes includes a smaller node size for a slice representing the region of interest and a larger node size for the remaining slices. The amount of the plurality of slices need not be equal to the amount of the plurality of node sizes. The plurality of node sizes is programmed by a user by specifying the node size in a configuration file. Machine learning is used to determine the plurality of node sizes. The plurality of node sizes is determined based on any arbitrary criterion other than the region of interest.

[0010] In another aspect, an apparatus includes a non-volatile memory for storing an application for receiving point cloud information, segmenting the point cloud information into a plurality of slices, determining a plurality of node sizes, and encoding the point cloud using a node size from the plurality of node sizes for each slice in the plurality of slices; and a processor coupled to the memory, the processor configured to process the application. The node size may be determined based on a region of interest criterion. For the region of interest, the node size is reduced. The plurality of node sizes includes a smaller node size for the slice representing the region of interest and a larger node size for the remaining slices. The amount of the plurality of slices need not be equal to the amount of the plurality of node sizes. The plurality of node sizes is programmable by a user by specifying the node size in a configuration file. Machine learning is used to determine the plurality of node sizes. The plurality of node sizes is determined based on any arbitrary criterion other than the region of interest.

[0011] In another aspect, a system includes an encoder configured to receive point cloud information, segment the point cloud information into a plurality of slices, determine a plurality of node sizes, and encode the point cloud using a node size from the plurality of node sizes for each slice in the plurality of slices; and a decoder configured to decode the encoded point cloud information. The node size may be determined based on a region of interest criterion. For the region of interest, the node size is reduced. The plurality of node sizes includes a smaller node size for slices representing the region of interest and a larger node size for the remaining slices. The amount of the plurality of slices need not be equal to the amount of the plurality of node sizes. The plurality of node sizes is programmable by a user by specifying the node size in a configuration file. Machine learning is used to determine the plurality of node sizes. The plurality of node sizes is determined based on any arbitrary criterion other than the region of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Illustrated is a block diagram depicting a G-PCC reference encoder, also referred to as TMC13, according to some embodiments.

[0013] Figure 2 The volume is illustrated as being partitioned vertically and horizontally into eight sub-cubes of dimensions D / 2×D / 2×D / 2 voxels according to some embodiments.

[0014] Figure 3 Illustrated is the number of derived points in the decoder determined by the voxel grid distance d which can be controlled according to some embodiments.

[0015] Figure 4 A Region Adaptive Hierarchical Transform (RAHT) is illustrated that is applied to each node and performed in three steps (one step in each of the x, y, and z directions) in accordance with some embodiments.

[0016] Figure 5 RAHT transform domain prediction according to some embodiments is illustrated.

[0017] Figure 6 Illustrated is an example of a sample point cloud organized in its original order and reorganized into three refinement levels and corresponding levels of detail (LoD0, LoD1, and LoD2) in accordance with some embodiments.

[0018] Figure 7 It is illustrated that prediction within the same refinement level may be performed by setting a flag to 1 according to some embodiments.

[0019] Figure 8 A prediction transform according to some embodiments is illustrated.

[0020] Figure 9 A lifting transform built on top of a prediction transform is illustrated according to some embodiments.

[0021] Figure 10 A flow chart illustrating a method of encoding a point cloud using a per-slice trisoup node size is illustrated in accordance with some embodiments.

[0022] Figure 11 A diagram illustrating a per-slice trisoup node size approach according to some embodiments.

[0023] Figure 12 Illustrated is a block diagram of an exemplary computing device configured to implement a per-slice trisoup node size method in accordance with some embodiments.

[0024] Figure 13 The graph shows the experimental results for a constant node size.

[0025] Figure 14 Experimental results with different node sizes according to some embodiments are illustrated.

[0026] Figure 15 A comparison of experimental results with a constant node size versus different node sizes is illustrated, according to some embodiments.

[0027] Figure 16 Illustrated is a PSNR graph according to some embodiments. DETAILED DESCRIPTION

[0028] Geometry-based Point Cloud Compression (G-PCC) is a standard for point cloud encoding and decoding techniques that offers compression capabilities exceeding other methods. G-PCC uses octrees, trisoups, or predictive geometry schemes to perform geometry encoding and decoding. In a trisoup, the geometry is represented by a pruned octree, constructed from the root to an arbitrary level, where leaves represent occupied nodes larger than a voxel. The surface of the object is approximated by a series of triangles, and since there is no connectivity information associated with multiple triangles, the technique is called a "triangle soup" or (trisoup).

[0029] G-PCC encodes content directly in 3D space. To achieve this, G-PCC utilizes data structures such as octrees that describe point positions in 3D space. Furthermore, G-PCC makes no assumptions about the input point cloud coordinate representation. Points have internal integer-based values converted from floating-point value representations. This conversion is conceptually similar to voxelization of the input point cloud and can be implemented through scaling, translation, and rounding.

[0030] Another important concept for G-PCC is the definition of tiles and slices to allow parallel encoding and decoding functionality. In G-PCC, a slice is defined as a collection of points (geometry and attributes) that can be independently encoded and decoded. A tile is a collection of slices with bounding box information. A tile can overlap with another tile, and the decoder can decode partial areas of the point cloud by accessing a specific slice.

[0031] A limitation of the current G-PCC standard is that it is only defined for intra prediction, i.e., it does not currently use temporal prediction tools. Nevertheless, techniques based on point cloud motion estimation and inter prediction are being considered for the next version of the standard.

[0032] Figure 1 A block diagram depicting the G-PCC reference encoder, also known as TMC13, according to some embodiments is shown. It is not meant to represent the complete functionality of TMC13, but rather some of its core modules. First, it can be seen that geometry and attributes are encoded separately. However, attribute encoding and decoding depends on the decoded geometry. Therefore, the point cloud positions are encoded and decoded first.

[0033] The source geometry points can be represented by floating-point numbers in the world coordinate system. Therefore, the first step in geometry coding and decoding is to perform a coordinate transformation, followed by voxelization. The second step involves geometric analysis using octrees, trisoups, or predictive geometry schemes. Finally, the resulting structure is arithmetic coded. Regarding attribute coding and decoding, TMC13 supports optional conversion from RGB to YCbCr. After that, one of the three available transformation tools is used, namely, region-adaptive hierarchical transform (RAHT), predictive transform, and lifting transform. After the transformation, the coefficients are quantized and arithmetic coded.

[0034] Octree encoding and decoding

[0035] The voxelized point cloud is represented in a lossless manner using an octree structure. Assume that the point cloud is contained in a quantized volume of D×D×D voxels. Initially, the volume is split vertically and horizontally into eight sub-cubes of dimensions D / 2×D / 2×D / 2 voxels, as Figure 2As illustrated in . This process is recursively repeated for each occupied sub-cube until D equals 1. In general, only one voxel position is occupied, which makes the octree very convenient for representing the geometry of a point cloud. At each decomposition step, it is verified which blocks are occupied and which blocks are not. Occupied blocks are marked with 1 and unoccupied blocks are marked with 0. The octets generated during this process represent the octree node occupancy state in a 1-byte word and are compressed by the entropy codec taking into account the correlation with neighboring octets. For the encoding and decoding of isolated points, since there are no other points in the volume that are correlated with them, an alternative method of entropy encoding and decoding the octets is used, namely direct coding mode (DCM). In DCM, the coordinates of the point are encoded and decoded directly without performing any compression. The DCM mode is inferred from the neighboring nodes to avoid signaling the use of DCM for all nodes of the tree.

[0036] Surface approximation via trisoup

[0037] Alternatively, the geometry can be represented by a pruned octree, which is constructed from the root to an arbitrary level, where the leaves represent occupied sub-blocks larger than a voxel. The object surface is approximated by a series of triangles, and since there is no connectivity information associated with multiple triangles, the technique is called "triangle soup" (or trisoup). It is an optional codec tool to improve subjective quality at lower bit rates, since quantization gives a coarse rate adaptation. If trisoup is enabled, the geometry bitstream becomes a combination of the octree, fragment indicators and vertex position information. During decoding, the decoder computes the intersection points between the trisoup mesh planes and the voxelized mesh. The number of points derived in the decoder is determined by the voxel grid distance d, which can be calculated as Figure 3 Control as shown in .

[0038] Attribute Encoding

[0039] In G-PCC, there are three methods for attribute encoding and decoding: RAHT; Prediction Transform; and Lifting Transform. The main idea behind RAHT is to use the attribute values of the lower octree levels to predict the values of the next level. Prediction Transform implements a hierarchical nearest neighbor prediction scheme based on interpolation. Lifting Transform builds on the Prediction Transform but has an additional update / lifting step. Therefore, from now on, they will be collectively referred to as Prediction / Lifting Transform. Users are free to choose any of the above-mentioned transforms. However, given a specific context, one method can be more appropriate than another. The common criterion for determining which method to use is a combination of rate-distortion performance and computational complexity.

[0040] RAHT transform

[0041] RAHT is performed by considering an octree representation of a point cloud. In its canonical formulation, it starts from the leaves (highest level) of the octree and then proceeds backwards until it reaches its root (lowest level). The transformation is applied to each node and is performed in three steps (one step in each of the x, y, and z directions), as Figure 4 As shown in . At each step, low-pass g n and high-pass h n coefficients are generated. RAHT is a hierarchical transform inspired by Haar. Therefore, it can be better understood if we use the 1D Haar transform as an initial example. Consider a signal v with N elements. The Haar decomposition of v generates g and h, which are the low-pass and high-pass components of the original signal, each with N / 2 elements. The nth coefficient of g and h is calculated using the following equation:

[0042]

[0043] The transform can be recursively performed with the current g as the new input signal v, and at each recursion the number of low-pass coefficients is divided by a factor of 2. The g component can be interpreted as the scaled sum of consecutive pairs of equal weights of v, and the h component can be interpreted as their scaled difference. However, if one chooses to encode the point cloud using the Haar transform, then the transform needs to be modified to account for the sparsity of the input point cloud. This can be achieved by allowing the weights to adapt based on the distribution of the points. Therefore, a recursive implementation of RAHT can be defined as follows:

[0044]

[0045]

[0046]

[0047] Where l is the decomposition level, w1 and w2 are the same as those at level l+1 and The weights associated with the low-pass coefficients, and is the low-pass coefficient at level l Therefore, higher weights are applied to dense area points, so that RAHT can better balance the signal in the transform domain than non-adaptive transform.

[0048] A fixed-point formulation of RAHT has been developed. It is based on matrix factorization and scaling of the quantization step. Simulations show that the fixed-point implementation can be considered equivalent to its floating-point counterpart.

[0049] Recently, transform domain prediction in RAHT has been developed and is available in the current test model TMC13. The main idea is that for each block, the sum of the transformed and upconverted attributes at level d, calculated from the sum of the decoded attributes at d-1, is used as a prediction for the sum of the transformed attributes at level d, generating a high-pass residual that can be further quantized and entropy coded. The upconversion process is done with the help of a weighted average of neighboring nodes. Figure 5 A simplified diagram of the RAHT transform domain prediction is shown. The reported gains show significant improvements in rate-distortion perception (overall average gains of up to around 30% for color and 16% for reflectance) compared to the unpredicted RAHT formulation.

[0050] Prediction / lifting transformation

[0051] The prediction transform is a distance-based prediction scheme for attribute encoding and decoding. It uses a level of detail (LoD) representation that distributes input points into a set of refinement levels (R) using a deterministic Euclidean distance criterion. Figure 6 An example of a sample point cloud organized in its original order and reorganized into three refinement levels, and the corresponding levels of detail (LoD0, LoD1, and LoD2) is shown. It can be noted that the level of detail l is obtained by taking the union of the refinement levels 0 to l.

[0052] The attributes of each point are encoded using a prediction determined by the LoD order. Figure 6 As an illustration, only LoD0 is considered. In this specific case, the attributes of P2 can be predicted by reconstructed versions of its nearest neighbors P4, P5 or P0 or by a distance-based weighted average of these points. The maximum number of prediction candidates can be specified, and the number of nearest neighbors is determined by the encoder for each point. In addition, a neighborhood variability analysis is performed. If the maximum difference between any two attributes in the neighborhood of a given point P is above a threshold, then a rate-distortion optimization process is used to control the best predictor. By default, the attribute value R(j) of a refinement level is predicted using the attribute values of its k nearest neighbors in the previous LoD (i.e., LoD(j-1)). However, prediction within the same refinement level can be performed by setting the flag to 1, as in Figure 7 As shown in .

[0053] The prediction transform is implemented using two operators based on the LoD structure, namely, the split and merge operators. Let L(j) and H(j) be the attribute sets associated with LoD(j) and R(j), respectively. The split operator takes L(j+1) as input and returns a low-resolution sample L(j) and a high-resolution sample H(j). The merge operator takes L(j) and H(j) and returns L(j+1). The prediction transform is implemented in Figure 8 Initially, the attribute signal L(N+1) representing the entire point cloud is split into H(N) and L(N). H(N) is then predicted using L(N) and the residual D(N) is calculated. This process is then repeated recursively. The reconstructed attributes are obtained through a cascade of merge operations.

[0054] exist Figure 9 The lifting transform shown in the figure is built on the prediction transform. It introduces an update operator and an adaptive quantization strategy. In the LoD prediction scheme, each point is associated with an influence weight. Points in lower LoD are used more often and therefore have a more significant impact on the encoding process. The update operator determines U(j) based on the residual D(j), and then uses U(j) to update the value of L(j), as shown in Figure 9 The updated signal U(j) is a function of the residual D(j), the distance between the predicted point and its neighbors, and their corresponding weights. Finally, to guide the quantization process, the transformed coefficients associated with each point are multiplied by the square root of their respective weights.

[0055] In the octree, the volume of the point cloud in which the point cloud resides is determined, and a cubic bounding box / block is obtained, and the block is then divided into sub-blocks, and then for each iteration, it is determined whether the sub-block contains a voxel, and if the sub-block contains more than one voxel, the sub-block is further divided (e.g., decomposed) until the sub-block consists of a single voxel.

[0056] In previous implementations of G-PCC, a single trisoup node size was used across all slices. Thus, the node size for slice 0 would be the same as the node size for slice 1, and the node size for slice 1 would be the same as the node size for slice N-1.

[0057] However, as described herein for the embodiments of trisoup node size per slice, the node size for one or more slices can be different. For example, the node size can increase or decrease as the number of slices increases or decreases. By way of further example, the node size for slice 0 can be very small, then the node size for slice 1 can be slightly larger, and so on, until the node size for slice N-1 is the largest. In another example, the node size for slice 0 is a first size, and the node sizes for the other slices are a second size. The node size can be doubled each time (from slice to slice), squared, or changed in another size. For example, a user can specify a node size for each slice.

[0058] On a per-slice basis, block / node size is used together with the concept of slices. Slices determine the number of points that are put into a slice. The point cloud can be split into slices as desired - for example, blocks of equal size or regions of interest. For example, a specific region of interest can be specified by the user / device. Further example, using machine learning, face detection or anything else, shapes / objects can be detected in order to be separated into groups / slices. This enables the encoder to have a specific block / node size for each slice. By having different node sizes, it is possible to have different areas, i.e. more important areas with a larger number of triangles to approximate the surface and less important areas with fewer triangles. This makes the encoder / decoder more flexible and efficient.

[0059] In some embodiments, segmentation and node size determination are performed by a person (e.g., in a configuration file), and in some embodiments, these steps are performed using machine learning without human intervention. For example, a user can define 10 slices and 10 or fewer node sizes in a configuration file, and the encoder then uses the first node size for the first slice, and so on. In another example, if the user defines fewer node sizes than slices, the last node size is applied to the remaining slices (e.g., if there are five slices and two node sizes are defined, the first slice uses the first node size, and the second to fifth slices use the second node size). For machine learning, the device / system can be trained to determine regions of interest (e.g., template matching or any other imaging process for detecting faces, humans, specific objects (e.g., vehicles), animals, and / or any specified objects). The device / system can also be trained to determine which node size to use for each slice. For example, if a slice contains a region of interest, the device / system learns that the slice should use a smaller node size than if the slice does not contain a region of interest. Moreover, regions of interest at different levels can be developed / learned. For example, the face can be designated as the highest level region of interest, while the rest of the body is the second highest level region of interest, and so on, to the lowest level region of interest (such as background information).

[0060] In G-PCC, the trisoup node size is indicated in the Geometry Parameter Set (GPS). Additionally, in the geometry header, the trisoup node size parameter defined in the GPS is used to indicate the remaining parameters for the trisoup codec (such as the number of sampling values and unique fragments). If it has a value other than 0, this indicates that a trisoup with a tree level defined in the GPS will be used. If you want to use slices with trisoup, the current representation does not allow the node size to be changed on a slice basis. This document describes a high-level syntax modification to allow control of node size on a slice basis in the trisoup codec. The enable flag GPS can be sent, and the node size value can be sent in the GDU header. The high-level syntax modification is presented below.

[0061] The following is example code for signaling with trisoup as described herein:

[0062]

[0063]

[0064]

[0065] In this solution, trisoup_enabled_flag equal to 1 specifies that the geometry data unit header may include trisoup codec syntax, including log2_tisoup_node_size; trisoup_enabled_flag equal to 0 specifies that the geometry data unit header includes only octree codec syntax. When trisoup_enabled_flag is 1, the bitstream conformance requirements are: a) inferred_direct_coding_mode_enabled_flag must be equal to 0, and b) unique_geometry_points_flag must be equal to 1. The log2_trisoup_node_size element specifies the variable TrisoupNodeSize as the size of the triangle node as follows: TrisoupNodeSize = (1 < <log2_trisoup_node_size-1)。

[0066] Another possibility is to send the base node size in the GPS, a flag to enable incremental offsets, and then send that incremental offset in the GDU header.

[0067] Geometry parameter set syntax

[0068]

[0069]

[0070]

[0071] In this solution, log2_trisoup_node_size_offset_present_flag equal to 1 specifies that the trisoup node size offset indicated by log2_trisoup_node_size_offset is present in the geometry data unit header; log2_tisoup_node_size_offset_present_flag equal to 0 specifies that no such offset is present. The element log2_trisoup_node_size_offset specifies the offset relative to log2_tisoup_node_size used in the trisoup codec syntax.

[0072] Figure 10 A flow chart illustrating a method for encoding a point cloud using a trisoup node size per slice according to some embodiments is shown. In step 1000, a device receives point cloud information. For example, the point cloud information is received and / or downloaded directly from a camera device.

[0073] In step 1002, the point cloud information is segmented / partitioned into slices. Segmentation can be performed manually or via machine learning. For example, a user indicates / selects a slice. In another example, the device / system utilizes machine learning to indicate / select a slice, such as by determining regions of interest and selecting those regions as specific slices. Regions of interest can be determined by a machine using any image processing technique, such as facial recognition, body recognition, and / or other object detection / recognition.

[0074] In step 1004, the node / block size is determined for the slice. The node size can be determined by a human or via machine learning. For example, a user can edit a configuration file to indicate the size of each node based on the slice. This information can include details such as slice 0 is the first specified node size, slice 1 is the second specified node size, etc., or more general information such that the node size increases or decreases as the number of slices increases. In another example, the node size is determined by machine learning, such as the device / system learns that a particular slice (e.g., based on a determined region) has a smaller node size when compared to a slice that does not include the region of interest. For example, using classification, the device / system uses the smallest node size for slices with faces, the second smallest node size for slices with bodies (non-faces), and the largest node size for other slices. The node size can be based on voxels or any other unit (e.g., the smallest is 1 voxel, the second smallest is 2 voxels, and the largest is 4 voxels).

[0075] In step 1006, the encoder encodes the point cloud information based on the slice and node size. The encoding is described in the G-PCC standard and modified as described herein based on the slice and node size information.

[0076] In some embodiments, fewer or additional steps can be implemented. For example, the decoder decodes point cloud information based on varying node sizes and slices. In some embodiments, the order of the steps is modified. For example, the order of the steps of selecting slices and determining node sizes can be switched.

[0077] Figure 11 A diagram illustrates a per-slice trisoup node size approach, according to some embodiments. As shown, the geometric codec flexibility of G-PCC can be increased by allowing it to operate using different trisoup node sizes for each slice. The previous G-PCC implementation 1110 used a single trisoup node size across all slices. In the per-slice trisoup node size approach, the node size for slice 0 1100 is smaller than the node size for slice 1 1102, and so on, up to a maximum node size for slice N-1 1104.

[0078] Figure 12A block diagram of an exemplary computing device configured to implement the per-slice trisoup node size method according to some embodiments is illustrated. Computing device 1200 can be used to acquire, store, calculate, process, transmit, and / or display information (such as images and videos including 3D content). Computing device 1200 can implement any of aspects of G-PCC, point cloud codec, and / or the per-slice trisoup node size method. Generally speaking, a hardware structure suitable for implementing computing device 1200 includes a network interface 1202, a memory 1204, a processor 1206, (one or more) I / O devices 1208, a bus 1210, and a storage device 1212. The choice of processor is not important, as long as a suitable processor with sufficient speed is selected. Memory 1204 can be any conventional computer memory known in the art. Storage device 1212 can include a hard drive, CDROM, CDRW, DVD, DVDRW, high-definition disk / drive, ultra-high-definition drive, flash memory card, or any other storage device. Computing device 1200 can include one or more network interfaces 1202. Examples of network interfaces include network cards connected to an Ethernet or other type of LAN. I / O device(s) 1208 can include one or more of the following: a keyboard, a mouse, a monitor, a screen, a printer, a modem, a touch screen, a button interface, and other devices. The per-slice trisoup node size application(s) 1230 for implementing the per-slice trisoup node size method are likely stored in storage device 1212 and memory 1204 and processed during normal application processing. Figure 12 More or fewer of the components shown in can be included in computing device 1200. In some embodiments, trisoup node size per slice hardware 1220 is included. Although Figure 12 The computing device 1200 in FIG. 1 includes an application 1230 and hardware 1220 for the per-slice trisoup node size method, but the per-slice trisoup node size method can be implemented on the computing device in hardware, firmware, software, or any combination thereof. For example, in some embodiments, the per-slice trisoup node size application 1230 is programmed in a memory and executed using a processor. In another example, in some embodiments, the per-slice trisoup node size hardware 1220 is programmed hardware logic including gates specifically designed to implement the per-slice trisoup node size method.

[0079] In some embodiments, (one or more) per-slice trisoup node size applications 1230 include several applications and / or modules. In some embodiments, a module also includes one or more sub-modules. In some embodiments, fewer or additional modules can be included.

[0080] In some embodiments, the per-slice trisoup node size hardware 1220 includes a camera component (such as a lens, an image sensor, and / or any other camera components).

[0081] Examples of suitable computing devices include personal computers, laptops, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants, cellular / mobile phones, smart appliances, game consoles, digital cameras, digital video cameras, camera phones, smartphones, portable music players, tablet computers, mobile devices, video players, video disc recorders / players (e.g., DVD recorders / players, high-definition disc recorders / players, ultra-high-definition disc recorders / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smart watches), vehicles (e.g., self-driving vehicles), or any other suitable computing device.

[0082] Figure 13 The figure shows the experimental results of a constant node size. Specifically, in the experiment, trisoup_node_size_log2 is equal to 4 for all nodes.

[0083] Figure 14 The figure shows experimental results with different node sizes according to some embodiments. Specifically, in the experiment, trisoup_node_size_log2 is equal to 4 for partitions (e.g., slices) 1, 2, and 3, while trisoup_node_size_log2 is equal to 1 for partition 0. In this result, the quality of the hair (partition 0) improves as the node size decreases.

[0084] Figure 15 1. A comparison of experimental results with constant node size versus different node sizes according to some embodiments is illustrated. Specifically, in the left image, trisoup_node_size_log2 is equal to 3 and is constant for all partitions. In the right image, trisoup_node_size_log2 is equal to 3 for partitions 1, 2, and 3, while trisoup_node_size_log2 is equal to 1 for partition 0. In this result, the quality of the right image in partition 0 (above the dividing line) improves as the node size decreases when compared to partition 0 of the left image and the other partitions of both images.

[0085] Figure 16 FIGURE 1 illustrates a PSNR graph according to some embodiments. A comparison of Peak Signal-to-Noise Ratio (PSNR) is shown for a constant node size and for different node sizes.

[0086] To utilize the per-slice trisoup node size method described herein, a device acquires or receives 3D content and processes and / or transmits the content in an optimized manner to enable correct and efficient display of the 3D content. The per-slice trisoup node size method can be implemented with user assistance or automatically without user involvement.

[0087] In operation, the per-slice trisoup node size approach encodes 3D content more efficiently. The per-slice trisoup node size approach allows for flexibility when encoding point clouds. Instead of having every block / node be the same size, the user or machine can dictate the block / node size, allowing regions of interest to have smaller node sizes for greater specificity in that region.

[0088] Some Examples of TRISOUP Node Sizes Per Slice

[0089] 1. A method of programming a non-transitory memory of a device, comprising:

[0090] Receive point cloud information;

[0091] Split the point cloud information into multiple slices;

[0092] determining a plurality of node sizes; and

[0093] The point cloud is encoded using a node size from the plurality of node sizes for each slice from the plurality of slices.

[0094] 2. The method of clause 1, wherein the node size is determined based on a region of interest criterion.

[0095] 3. The method of clause 2, wherein the node size is reduced for the region of interest.

[0096] 4. The method of clause 3, wherein the plurality of node sizes comprises a smaller node size for slices representing a region of interest and a larger node size for remaining slices.

[0097] 5. The method of clause 1, wherein the amount of the plurality of slices need not be equal to the amount of the plurality of node sizes.

[0098] 6. The method of clause 1, wherein the plurality of node sizes are programmable by a user by specifying the node sizes in a configuration file.

[0099] 7. The method of clause 1, wherein the plurality of node sizes are determined using machine learning.

[0100] 8. The method of clause 1, wherein the plurality of node sizes are determined based on any arbitrary criteria other than region of interest.

[0101] 9. A device comprising:

[0102] Non-volatile memory used to store applications used for:

[0103] Receive point cloud information;

[0104] Split the point cloud information into multiple slices;

[0105] determining a plurality of node sizes; and

[0106] encoding a point cloud using a node size from the plurality of node sizes for each slice from the plurality of slices; and

[0107] A processor coupled to the memory, the processor configured to process the application.

[0108] 10. The apparatus of clause 9, wherein the node size is determined according to a region of interest criterion.

[0109] 11. The apparatus of clause 10, wherein the node size is reduced for the region of interest.

[0110] 12. The apparatus of clause 11, wherein the plurality of node sizes comprises a smaller node size for slices representing a region of interest and a larger node size for remaining slices.

[0111] 13. The apparatus of clause 9, wherein the amount of the plurality of slices need not be equal to the amount of the plurality of node sizes.

[0112] 14. The apparatus of clause 9, wherein the plurality of node sizes are programmable by a user by specifying the node sizes in a configuration file.

[0113] 15. The apparatus of clause 9, wherein the plurality of node sizes are determined using machine learning.

[0114] 16. The apparatus of clause 9, wherein the plurality of node sizes are determined based on any arbitrary criteria other than region of interest.

[0115] 17. A system comprising:

[0116] Encoder, configured to:

[0117] Receive point cloud information;

[0118] Split the point cloud information into multiple slices;

[0119] determining a plurality of node sizes; and

[0120] encoding a point cloud using a node size from the plurality of node sizes for each slice from the plurality of slices; and

[0121] A decoder is configured to decode the encoded point cloud information.

[0122] 18. The system of clause 17, wherein the node size is determined based on a region of interest criterion.

[0123] 19. The system of clause 18, wherein the node size is reduced for the region of interest.

[0124] 20. The system of clause 19, wherein the plurality of node sizes comprises a smaller node size for slices representing a region of interest and a larger node size for remaining slices.

[0125] 21. The system of clause 17, wherein the amount of the plurality of slices need not be equal to the amount of the plurality of node sizes.

[0126] 22. The system of clause 17, wherein the plurality of node sizes are programmable by a user by specifying the node sizes in a configuration file.

[0127] 23. The system of clause 17, wherein the plurality of node sizes are determined using machine learning.

[0128] 24. The system of clause 17, wherein the plurality of node sizes are determined based on any arbitrary criteria other than region of interest.

[0129] The present invention has been described with reference to specific embodiments that incorporate details to facilitate an understanding of the principles of construction and operation of the invention. This reference herein to specific embodiments and details thereof is not intended to limit the scope of the appended claims. It will be apparent to those skilled in the art that various other modifications may be made in the embodiments chosen for illustration without departing from the spirit and scope of the invention as defined by the claims.

Claims

1. A method of programming in a non-transitory memory of a device, comprising: Receive point cloud information; Split the point cloud information into multiple slices; Determine multiple node sizes; as well as The point cloud is encoded using a node size from the plurality of node sizes for each slice in the plurality of slices, wherein the node size doubles as the slice number of the plurality of slices increases, and wherein when fewer node sizes than the number of slices are defined, a last node size is applied to a remaining number of slices greater than the defined node sizes. The method of claim 1 , wherein the node size is determined based on a region of interest criterion. The method of claim 2 , wherein the node size is reduced for the region of interest. 4 . The method of claim 3 , wherein the plurality of node sizes comprises a smaller node size for slices representing a region of interest and a larger node size for remaining slices. The method of claim 1 , wherein the amount of the plurality of slices need not be equal to the amount of the plurality of node sizes.

6. The method of claim 1, wherein the plurality of node sizes are programmed by a user by specifying the node sizes in a configuration file.

7. The method of claim 1, wherein the plurality of node sizes are determined using machine learning.

8. The method of claim 1, wherein the plurality of node sizes are determined based on any arbitrary criteria other than region of interest.

9. A device comprising: Non-transitory memory for storing applications for: Receive point cloud information; Split the point cloud information into multiple slices; Determine multiple node sizes; as well as encoding the point cloud using a node size from the plurality of node sizes for each slice in the plurality of slices, wherein the node size doubles as the slice number of the plurality of slices increases, wherein when fewer node sizes than the number of slices are defined, applying a last node size to a number of remaining slices greater than the defined node sizes; as well as A processor coupled to the memory, the processor configured to process the application.

10. The apparatus of claim 9, wherein the node size is determined based on a region of interest criterion. The apparatus of claim 10 , wherein the node size is reduced for the region of interest.

12. The apparatus of claim 11, wherein the plurality of node sizes comprises a smaller node size for slices representing a region of interest and a larger node size for remaining slices.

13. The apparatus of claim 9, wherein the amount of the plurality of slices need not be equal to the amount of the plurality of node sizes.

14. The apparatus of claim 9, wherein the plurality of node sizes are programmable by a user by specifying the node sizes in a configuration file.

15. The apparatus of claim 9, wherein the plurality of node sizes are determined using machine learning.

16. The apparatus of claim 9, wherein the plurality of node sizes are determined based on any arbitrary criteria other than a region of interest.

17. A system comprising: Encoder, configured to: Receive point cloud information; Split the point cloud information into multiple slices; Determine multiple node sizes; as well as encoding the point cloud using a node size from the plurality of node sizes for each slice in the plurality of slices, wherein the node size doubles as the slice number of the plurality of slices increases, wherein when fewer node sizes than the number of slices are defined, applying a last node size to a number of remaining slices greater than the defined node sizes; and A decoder is configured to decode the encoded point cloud information.

18. The system of claim 17, wherein node size is determined based on a region of interest criterion.

19. The system of claim 18, wherein for regions of interest, the node size is reduced.

20. The system of claim 19, wherein the plurality of node sizes includes a smaller node size for slices representing a region of interest and a larger node size for remaining slices.

21. The system of claim 17, wherein the amount of the plurality of slices need not be equal to the amount of the plurality of node sizes.

22. The system of claim 17, wherein the plurality of node sizes are programmable by a user by specifying the node sizes in a configuration file.

23. The system of claim 17, wherein the plurality of node sizes are determined using machine learning.

24. The system of claim 17, wherein the plurality of node sizes are determined based on any arbitrary criteria other than region of interest.

Images