In-tree geometry quantization of point clouds

Abstract

An exemplary device includes one or more processors and a memory storing instructions that, when executed by the processors, cause the processors to receive points representing a point cloud in three-dimensional space and generate a data structure representing the point cloud. Generating the data structure includes: encoding the position of each point in each dimension as a bit sequence according to a tree data structure; partitioning each of the sequences into two or more portions according to a zoom depth; determining that a subset of the points is spatially isolated from a remaining portion of the points; quantizing each of the portions associated with the subset of the points according to a first quantization step size; quantizing each of the portions associated with the remaining portion of the points according to a second quantization step size; and including the quantized portions in the data structure.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application Serial No. 62 / 959,105, filed on January 9, 2020, and U.S. Provisional Application Serial No. 62 / 959,832, filed on January 10, 2020, which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to storing and processing information about a point cloud comprising a plurality of points. Background Art

[0004] Various types of sensors (such as light detection and ranging (LIDAR) systems, 3D cameras, 3D scanners, etc.) can capture data indicating the location of points in three-dimensional space (e.g., positions in the X, Y, and Z planes). Furthermore, such systems can capture attribute information in addition to spatial information for corresponding points, such as color information (e.g., RGB values), intensity attributes, reflectivity attributes, motion-related attributes, modality attributes, or various other attributes. In some cases, additional attributes can be assigned to corresponding points, such as a timestamp when the point was captured. The points captured by such sensors can constitute a "point cloud," which includes a set of points, each with associated spatial information and one or more associated attributes. In some cases, a point cloud can include thousands, hundreds of thousands, millions, or even more points. Furthermore, in some cases, a point cloud can be generated, for example, in software, separate from the point cloud being captured by one or more sensors. In either case, such a point cloud can include a large amount of data, and storing and transmitting these point clouds can be costly and time-consuming. Summary of the Invention

[0005] In an example, a method includes receiving, by a computer system, a plurality of points representing a point cloud in a three-dimensional space; and generating, by the computer system, a data structure representing the point cloud in the three-dimensional space. Generating the data structure includes: for each point, representing a position of the point in each dimension of the three-dimensional space as a sequence of bits, wherein the position of the point is represented according to a tree data structure; partitioning each of the sequence of bits into a first portion of bits and a second portion of bits according to a scaled depth of the tree data structure; determining that the subset of points is spatially isolated from the rest of the points; quantizing each of the second portion of bits associated with the subset of points according to a first quantization step size; quantizing each of the second portion of bits associated with the rest of the points according to a second quantization step size, wherein the first quantization step size is different from the second quantization step size; including the quantized second portion of bits in the data structure; and performing, by the computer system, at least one of: transmitting the data structure or storing the data structure.

[0006] Implementations of this aspect may include one or more of the following features.

[0007] In some implementations, generating the data structure can include including a header in the data structure that indicates the first quantization step size.

[0008] In some implementations, determining that the subset of points is spatially isolated from the rest of the points can include determining that the subset of points is spatially isolated from the rest of the points at a first depth of the tree data structure, where the first depth is less than or equal to a zoom depth. In response to the determination, each of the second portion of bits associated with the subset of points can be quantized according to the first quantization step size.

[0009] In some implementations, determining that at a first depth, a subset of points is spatially isolated from a remainder of the points can include, at the first depth, the subset of points occupying one or more first nodes in a tree data structure, and determining that at the first depth, the remainder of the points occupying one or more second nodes in the tree data structure, wherein the one or more first nodes are not adjacent to any of the one or more second nodes.

[0010] In some implementations, the tree data structure can be an octree.

[0011] In some implementations, the tree data structure may include a plurality of nodes arranged according to a plurality of tree levels. Furthermore, a first node of the plurality of nodes may explicitly signal a location of a point in the point cloud. A quantization parameter and / or a quantization step size may be constrained based on a tree level of a first node of the plurality of tree nodes from the tree data structure.

[0012] In another aspect, a method includes receiving, by a computer system, a data structure representing a point cloud in three-dimensional space, the point cloud having a plurality of points. For each point, the data structure includes a bit sequence encoding a position of the point in each dimension of the three-dimensional space, wherein the positions of the points are encoded according to a tree data structure, and, for a subset of the points, the corresponding bit sequence is quantized according to a first quantization step size. For a remaining portion of the points, the corresponding bit sequence is quantized according to a second quantization step size, wherein the first quantization step size is different from the second quantization step size. The subset of points is spatially isolated from the remaining portion of the points. The method also includes determining, by the computer system, a position of each of the points based on the data structure; and outputting, by the computer system, the determined position of each of the points.

[0013] Implementations of this aspect may include one or more of the following features.

[0014] In some implementations, determining the position of each point in the subset of points based on the data structure may include retrieving an indication of a first quantization step size from a header in the data structure, and determining the position of each point in the subset of points based on the first quantization step size.

[0015] In some implementations, the tree data structure can be an octree.

[0016] In some implementations, the method may include rendering three-dimensional content based on the determined position of each of the points. The three-dimensional content may be at least one of virtual reality content or augmented reality content.

[0017] On the other hand, a method includes receiving, by a computer system, a plurality of points representing a point cloud in a three-dimensional space; and generating, by the computer system, a data structure representing the point cloud in the three-dimensional space. Generating the data structure includes: for each point, representing the position of the point in each dimension of the three-dimensional space as a bit sequence, wherein the position of the point is represented according to a tree data structure. The tree data structure includes a plurality of tree levels, each tree level having a plurality of nodes. Each node in a first tree level has a first number of corresponding child nodes in a second tree level, and each node in the second tree level has a second number of corresponding child nodes in a third tree level, wherein the first number is different from the second number. Generating the data structure also includes including in the data structure a first set of parameters indicating the first number and the first tree level and a second set of parameters indicating the second number and the second tree level.

[0018] Implementations of this aspect may include one or more of the following features.

[0019] In some implementations, the first number is a cube of an integer, and the second number is not a cube of an integer.

[0020] In some embodiments, the method may further include determining the occupancy of each node in the third tree level with respect to adjacent nodes in the third tree level; performing one or more data processing steps on each node based on the determined occupancy of adjacent nodes in the third tree level; and performing one or more quantization or scaling steps relative to the data structure after performing the one or more data processing steps.

[0021] In some implementations, performing one or more quantization or scaling steps with respect to the data structure can include dividing each of the bit sequences into two or more portions of bits according to a scaling depth of the tree data structure.

[0022] In some implementations, performing one or more quantization or scaling steps with respect to the data structure can further include quantizing at least some portion of the bits according to a quantization step size; and including the quantized sequence of bits in the data structure.

[0023] In another aspect, a method includes receiving, by a computer system, a data structure representing a point cloud in three-dimensional space, the point cloud having a plurality of points. For each point, the data structure indicates a position of the point encoded according to a tree data structure. The tree data structure includes a plurality of tree levels, each tree level having a plurality of nodes. Each node in a first tree level has a first number of corresponding child nodes in a second tree level, and each node in the second tree level has a second number of corresponding child nodes in a third tree level, wherein the first number is different from the second number. The data structure includes a first set of parameters indicating the first number and the first tree level, and a second set of parameters indicating the second number and the second tree level. The method also includes determining, by the computer system, a position of each of the points based on the data structure; and outputting, by the computer system, the determined position of each of the points.

[0024] Implementations of this aspect may include one or more of the following features.

[0025] In some implementations, the first number is a cube of an integer, and the second number is not a cube of an integer.

[0026] In some embodiments, determining the location of each point based on the data structure may include retrieving a first set of parameters and a second set of parameters from the data structure; and determining, based on the first set of parameters and the second set of parameters, that each node in the first tree level has a first number of corresponding child nodes in the second tree level, and determining that each node in the second tree level has a second number of corresponding child nodes in the third tree level.

[0027] In some implementations, the tree data structure can be an octree.

[0028] In some implementations, the method may include rendering three-dimensional content based on the determined position of each of the points. The three-dimensional content may be at least one of virtual reality content or augmented reality content.

[0029] Other specific implementations relate to systems, devices, and non-transitory computer-readable media having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform the operations described herein.

[0030] The details of one or more embodiments are set forth in the accompanying drawings and the detailed description below. Other features and advantages will be apparent from the detailed description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A system is shown that includes a sensor that captures information about points of a point cloud and an encoder that compresses attribute and / or spatial information of the point cloud, where the compressed attribute and / or spatial information is sent to a decoder.

[0032] Figure 2 An exemplary recursive partitioning of space and corresponding tree data structure is shown.

[0033] Figures 3A to 3E An exemplary encoding of points according to a tree data structure is shown.

[0034] Figures 4A to 4C An exemplary application of the techniques described herein is shown.

[0035] Figure 5 Another exemplary recursive partitioning of space and corresponding tree data structure is shown.

[0036] Figure 6A An exemplary process for generating a data structure representing a point cloud is shown.

[0037] Figure 6B An exemplary process for rendering three-dimensional content based on a data structure representing a point cloud is shown.

[0038] Figure 7A An exemplary process for generating a data structure representing a point cloud is shown.

[0039] Figure 7B An exemplary process for rendering three-dimensional content based on a data structure representing a point cloud is shown.

[0040] Figure 8 Shows compressed point cloud information being used in 3D applications.

[0041] Figure 9 Compressed point cloud information is shown being used in a virtual reality application according to some embodiments.

[0042] Figure 10 An exemplary computer system is shown that may implement an encoder or decoder.

[0043] This specification includes references to "one embodiment" or "an embodiment." The appearance of the phrase "in one embodiment" or "in an embodiment" does not necessarily refer to the same embodiment. The particular features, structures, or characteristics may be combined in any suitable manner consistent with the present disclosure.

[0044] The term "comprising" is open-ended. As used in the appended claims, the term does not exclude additional structures or steps. Consider the following recited claim: "An apparatus comprising one or more processor units..." Such a claim does not exclude the apparatus from including additional components (e.g., a network interface unit, graphics circuitry, etc.).

[0045] Various units, circuits, or other components may be described or stated as being “configured to” perform one or more tasks. In such contexts, “configured to” is used to imply a structure (e.g., circuitry) that performs the one or more tasks during operation by indicating that the unit / circuit / component includes that structure. Thus, the unit / circuit / component is said to be configured to perform the task even when the specified unit / circuit / component is not currently operational (e.g., not turned on). Units / circuits / components used with the “configured to” language include hardware—e.g., circuitry, memory storing program instructions executable to implement the operation, etc. Reference to a unit / circuit / component being “configured to” perform one or more tasks is expressly intended not to invoke 35 U.S.C. §112(f) with respect to that unit / circuit / component. Furthermore, “configured to” may include a general structure (e.g., general circuitry) manipulated by software and / or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in a manner capable of performing the one or more tasks to be solved. "Configured to" may also include adapting a manufacturing process (eg, a semiconductor fabrication facility) to produce a device (eg, an integrated circuit) suitable for implementing or performing one or more tasks.

[0046] "First," "Second," etc. As used herein, these terms act as labels for the nouns that precede them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing a write operation for a "first" value and a "second" value. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0047] "Based on." As used herein, this term is used to describe one or more factors that influence a determination. This term does not exclude additional factors that influence the determination. That is, the determination may be based solely on these factors, or at least in part on these factors. Consider the phrase "A is determined based on B." In this case, B is a factor that influences the determination of A, and such a phrase does not exclude that the determination of A may also be based on C. In other examples, A may be determined based solely on B. DETAILED DESCRIPTION

[0048] As data acquisition and display technologies become more advanced, the ability to capture point clouds comprising tens of thousands of points in 2D or 3D space has increased (such as via LIDAR systems). Furthermore, the development of advanced display technologies (such as virtual reality or augmented reality systems) has increased the potential uses of point clouds. However, point cloud files are typically very large, and storing and transmitting these point cloud files can be costly and time consuming. For example, communication of a point cloud over a private network or a public network (such as the Internet) can require a considerable amount of time and / or network resources, such that some uses of the point cloud data (such as real-time uses) may be limited. Additionally, the storage requirements of the point cloud files may consume a significant amount of storage capacity of the device storing the point cloud files, which may also limit potential applications using the point cloud data.

[0049] In some embodiments, an encoder can be used to generate a compressed point cloud to reduce the cost and time associated with storing and transmitting large point cloud files. In some embodiments, a system can include an encoder that compresses attribute information or spatial information of a point cloud file, allowing the point cloud file to be stored and transmitted more quickly and in a manner that occupies less storage space than an uncompressed point cloud. In some embodiments, compressing the attributes of points in a point cloud can enable the point cloud to be transmitted over a network in real time or near real time. For example, a system can include a sensor that captures attribute information about points in an environment in which the sensor is located, where the captured points and corresponding attributes constitute a point cloud. The system can also include an encoder that compresses the attribute information of the captured point cloud. The compressed attribute information of the point cloud can be transmitted over a network in real time or near real time to a decoder that decompresses the compressed attribute information of the point cloud. The decompressed point cloud can be further processed, for example, to make control decisions based on the surrounding environment at the sensor location. The control decisions can then be transmitted back to a device at or near the sensor location, where the device receiving the control decisions implements the control decisions in real time or near real time. In some embodiments, the decoder can be associated with an augmented reality system, and the decompressed attribute information can be displayed or otherwise used by the augmented reality system. In some embodiments, compressed attribute information about the point cloud can be sent along with compressed spatial information about the points in the point cloud. In other embodiments, the spatial information and attribute information can be encoded separately and / or sent separately to the decoder.

[0050] In some embodiments, the system may include a decoder that receives one or more point cloud files including compressed attribute information from a remote server or other storage device storing the one or more point cloud files via a network. For example, a 3D display, a holographic display, or a head-mounted display may be manipulated in real time or near real time to display different portions of a virtual world represented by a point cloud. To update the 3D display, holographic display, or head-mounted display, a system associated with the decoder may request point cloud files from a remote server based on user manipulation of the display, and these point cloud files may be transmitted from the remote server to the decoder and decoded by the decoder in real time or near real time. The display may then be updated with updated point cloud data (such as updated point attributes) in response to the user manipulation.

[0051] In some embodiments, a system may include one or more LIDAR systems, 3D cameras, 3D scanners, etc., and such sensor devices may capture spatial information, such as the X, Y, and Z coordinates of points in the sensor device's field of view. In some embodiments, the spatial information may be relative to a local coordinate system or may be relative to a global coordinate system (e.g., a Cartesian coordinate system may have fixed reference points, such as fixed points on the Earth, or may have non-fixed local reference points, such as sensor locations).

[0052] In some embodiments, such sensors may also capture attribute information about one or more points, such as color attributes, reflectivity attributes, velocity attributes, acceleration attributes, time attributes, modality, and / or various other attributes. In some embodiments, in addition to LIDAR systems, 3D cameras, 3D scanners, etc., other sensors may capture attribute information to be included in the point cloud. For example, in some embodiments, a gyroscope or accelerometer may capture motion information to be included in the point cloud as an attribute associated with one or more points of the point cloud. For example, a vehicle equipped with a LIDAR system, a 3D camera, or a 3D scanner may include the vehicle's direction and velocity in the point cloud captured by the LIDAR system, 3D camera, or 3D scanner. For example, when points in the vehicle's field of view are captured, these points may be included in a point cloud, wherein the point cloud includes the captured points and associated motion information corresponding to the state of the vehicle at the time the points were captured.

[0053] Figure 1 A system is shown that includes a sensor that captures information about points of a point cloud and an encoder that compresses attribute information of the point cloud, where the compressed attribute information is sent to a decoder.

[0054] System 100 includes a sensor 102 and an encoder 104. Sensor 102 captures a point cloud 110, which includes points representing structures 106 in a view 108 of sensor 102. For example, in some embodiments, structure 106 may be a mountain, a building, a sign, the surroundings of a street, or any other type of structure. In some embodiments, a captured point cloud, such as captured point cloud 110, may include spatial information and attribute information about the points included in the point cloud. For example, point A in captured point cloud 110 includes X, Y, and Z coordinates, as well as attributes 1, 2, and 3. In some embodiments, the attributes of a point may include attributes such as R, G, and B color values, velocity at the point, acceleration at the point, reflectivity of the structure at the point, a timestamp indicating when the point was captured, a string value indicating the modality at the time the point was captured, such as "walking," or other attributes. Captured point cloud 110 may be provided to encoder 104, which generates a compressed version of the point cloud (compressed attribute information 112), which is transmitted to decoder 116 via network 114. In some embodiments, a compressed version of a point cloud (such as compressed attribute information 112) may be included in a common compressed point cloud that also includes compressed spatial information for the points of the point cloud, or in some embodiments, the compressed spatial information and compressed attribute information may be transmitted as separate files.

[0055] In some embodiments, encoder 104 can be integral to sensor 102. For example, encoder 104 can be implemented in hardware or software included in a sensor device, such as sensor 102. In other embodiments, encoder 104 can be implemented on a separate computing device adjacent to sensor 102.

[0056] In some cases, the number of bits required to encode the attribute information comprises a significant portion of the bitstream for the point cloud.

[0057] In some embodiments, the locations of points in point cloud 110 can be encoded according to a tree data structure. For example, the three-dimensional space can be recursively subdivided into successively smaller parts. Each of these parts can be represented by a corresponding node in the tree data structure. The presence of a point in each part can be indicated in the corresponding node of the tree data structure.

[0058] As an example, refer to Figure 2, the positions of points in the point cloud 110 can be encoded according to an octree 200, where each node 202 of the octree 200 has exactly eight child nodes 202. The three-dimensional space 204 can be recursively subdivided into successively smaller eighths. Each of these eighths can be represented by a corresponding node 202 of the octree 200. The presence of a point in each eighth can be indicated in the corresponding node 202 of the octree (e.g., using a bit "1" to indicate the presence of a point in the corresponding eighth, or using a bit "0" to indicate the absence of a point in the corresponding eighth).

[0059] Figures 3A to 3E An example of encoding the positions of points according to a tree data structure is shown in FIG. In this example, a plurality of points 300 on a single two-dimensional plane are encoded according to a quadtree (a tree data structure in which each node has exactly four children). However, it should be understood that points in a three-dimensional point cloud can be encoded in a similar manner according to an octree.

[0060] like Figure 3A As shown, a plurality of points 300 are located on a two-dimensional plane. The presence of a point 300 at a particular location is indicated by a white box in a grid 302, while the absence of a point 300 at a particular location is indicated by a black box in the grid 302. The rows of the grid 302 are indexed according to corresponding binary sequences 304 (shown on the right side of the grid 302, where the black boxes indicate bit "0" and the colored boxes indicate bit "1"). Similarly, the columns of the grid 302 are indexed according to corresponding binary sequences 306 (shown at the bottom of the grid 302, where the black boxes indicate bit "0" and the colored boxes indicate bit "1").

[0061] like Figure 3B As shown, the grid 302 can be divided into four equal-sized portions 308a-308d (e.g., squares). The division can be represented by a tree data structure 310 having a parent node and four child nodes branching from the parent node. The presence of one or more points 300 in each portion 308a-308d can be indicated by a bit "1" in the corresponding child node. Here, since each portion 308a-308d includes at least one point 300, each child node is assigned a bit "1."

[0062] like Figure 3CAs shown, each portion 308a-308d can be further subdivided into four equally sized sub-portions (e.g., squares). For example, portion 308a can be subdivided into portions 310a-310d. This division can be represented by a tree data structure 310, which has four child nodes branching from the node corresponding to portion 308a. The presence of a point or multiple points 300 in each sub-portion 310a-310d can be represented by a bit "1" in the corresponding child node. Here, since each sub-portion in sub-portions 310a, 310b, and 310d includes at least one point 300, the child nodes associated with those sub-portions are each assigned a bit "1." The child node associated with sub-portion 310c is assigned a bit "0." Similarly, other portions 308c-308d can also be subdivided into four equal sub-portions, and the presence of a point in each sub-portion can be indicated by the tree data structure 310 in a manner similar to that described above.

[0063] like Figure 3D As shown, each sub-portion 310a-310d can be further subdivided into four equally sized sub-portions (e.g., squares). For example, sub-portion 310a can be subdivided into sub-portions 312a-312d. This division can be represented by a tree data structure 310, which has four child nodes branching from the node corresponding to sub-portion 310a. The presence of a point or multiple points 300 in each sub-portion 312a-312d can be indicated by a bit "1" in the corresponding child node. Here, since each sub-portion in sub-portions 312a and 312b includes at least one point 300, the child nodes associated with those sub-portions are each assigned a bit "1". The child nodes associated with sub-portions 310c and 310d are each assigned a bit "0". Similarly, the other sub-portions of the grid can each be subdivided into four equal sub-portions, and the presence of a point in each sub-portion can be indicated by the tree data structure 310 in a manner similar to that described above.

[0064] The spatial position of a particular point 300 may be represented by one or more binary sequences. For example, the spatial position of point 300a relative to the horizontal axis may be represented by the binary sequence "011" (e.g., corresponding to its column index), and the spatial position of point 300a relative to the vertical axis may be represented by the binary sequence "101" (e.g., corresponding to its row index).

[0065] although Figures 3A to 3D 3. The tree data structure 310 is shown as having four levels, but in practice, the tree data structure may include any number of levels (corresponding to any number of spatial subdivisions). Figures 3A to 3DA tree data structure 310 is shown having four child nodes branching from each parent node (e.g., a quadtree), but in practice, the tree data structure can have any number of child nodes branching from each parent node (e.g., 8 child nodes branching from each parent node, as with an octree).

[0066] Quantization with multiple step sizes :

[0067] In some implementations, during the encoding process, the positions of the points can be partially quantized or scaled so that the positions (or approximate positions) of the points can be represented using a reduced number of bits. As an example, the encoder can divide each of the binary sequences representing the positions of the points into two parts: (i) a first part comprising the first d bits of the binary sequence, where d is a particular depth (e.g., representing bits that have already been encoded by the encoder), and (ii) a second part comprising the remaining bits of the binary sequence. The second part can be quantized according to a selected quantization parameter.

[0068] To illustrate, the binary representation of the x position of a point can be represented as {b4, b3, b2, b1, b0}, where x is between 0 and 31 (inclusive). For example, x=21=0b10101 (i.e., the binary sequence "10101"). After encoding the first two tree levels, at a scaled depth of depth d=2, the first two bits of 0b10101 are discarded, leaving 0b101 (i.e., decimal value 5). The remainder can be quantized according to a step size of 2 (e.g., 0b101 / 2=0b10). This has the effect of reducing the range of the remainder from decimal values 0 to 7 to decimal values 0 to 3. Since the range has been reduced by a factor of two, only two tree levels are required to represent the remainder, rather than the three levels of the remainder before quantization. The level reduction can be expressed as Floor(Log2(QS)), where QS is the quantization step size. A decoder informed of quantization with a step size of 2 at tree depth 2 will decode four bits of the corresponding point (5-Floor(Log2(2))=4). The decoder will scale the quantized portion of the position information (the two least significant bits, 0b10) by the quantization step size QS (0b10*2=0b100) and append the scaled value to the two most significant bits (0b10:0b100), resulting in a reconstructed position of 0b10100.

[0069] In some implementations, the quantitative step size QS may be determined based on an integer quantization parameter QP of an exponential function f(QP)=QS. For example, the relationship QS=2 may be used. (QP-4) / 6 The quantitative step size QS is determined. In this example, increasing QP by 6 (eg, from 4 to 10) doubles the quantitative step size QS.

[0070] The performance of this "in-tree" geometric quantization has various effects. For example, this quantization eliminates levels from the subtree of fully quantized nodes, thereby avoiding signaling occupation of the eliminated nodes. In addition, this quantization reduces the number of coded points in the tree data structure. Moreover, this quantization compresses sparse surface combinations into a less sparse form. As a result, information about a point cloud (e.g., the presence and / or location of one or more points in the point cloud) can be stored using less data (e.g., compared to techniques that do not utilize quantization).

[0071] In some implementations, all points in a point cloud can be encoded according to a tree data structure (e.g., an octree). This can be beneficial, for example, because it can encode and store information about points in a consistent manner and with consistent accuracy and / or precision.

[0072] However, in some implementations, a portion of the points in the point cloud can be encoded entirely according to a tree data structure, and at least some of the remaining points can be encoded in part according to the same tree data structure and in part according to another technique. In some implementations, at least some of the remaining points can be partially encoded by explicitly signaling each of their positions relative to the positions of the tree nodes (e.g., explicitly specifying the position of each point in one or more dimensions). This can be beneficial, for example, in reducing the computational resources required to encode the points according to the tree data structure (which in some cases can be higher than using explicit signaling techniques).

[0073] In some implementations, the encoder can distinguish between different subsets of points, depending on their proximity to other subsets of points. For example, the encoder can identify one or more subsets of points that are close to each other and fully encode those points according to the tree data structure. The encoder can also identify one or more subsets of points that are isolated from other subsets of points and partially encode each of those points according to explicit signaling techniques. This can be beneficial, for example, because information about isolated points can be more efficiently encoded and stored using explicit signaling techniques rather than using a tree data structure.

[0074] In some implementations, an isolated subset of points can be identified during encoding of a tree data structure (e.g., during recursive partitioning of space into successively smaller portions) by identifying a portion of space occupied by one or more points (e.g., one-eighth), but not surrounded by a portion of space occupied by any points (e.g., another one-eighth). The encoder can designate the points within the occupied portion of space as "isolated" points and encode these points based, in part, on explicit signaling techniques. The encoder can encode the remaining points based on a tree data structure (e.g., an octree). In some implementations, the set of isolated points can be referred to as an inferred direct coding mode (IDCM) node.

[0075] In some applications, the precise location of isolated points may not be as important as that of a dense, non-isolated subset of points. For example, due to sampling noise, isolated points may have greater position uncertainty than a dense, non-isolated subset of points, where noise reduction techniques based on local point position correlations can be used. When encoding with tree data structures (such as octrees or quadtrees), this excess precision and noise can lead to lower compression efficiency even when using explicit signaling techniques.

[0076] In some implementations, an encoder may specify a quantization parameter QP (and correspondingly a quantization step size QS) for quantizing isolated points (e.g., IDCM nodes) during encoding. In some implementations, the quantization parameter QP may be signaled in a "geometry slice" (e.g., a data structure representing a point). In some implementations, the quantization parameter QP may be signaled in a header of the geometry slice. In other implementations, the quantization parameter may be signaled in a parameter set. In another implementation, the quantization parameter may be derived from a base value signaled in the parameter set and an offset signaled in the header of the geometry slice. In another implementation, the quantization parameter may be derived from a base value and an offset signaled in a corresponding tree node at a scaled depth of the geometry slice.

[0077] Each node that partially represents an explicitly encoded point position is quantized by the encoder according to a corresponding quantization step size for isolated points. In some implementations, the quantization step size is an integer power of two. In other implementations, the step size is an exponential function of the quantization parameter. In some implementations, the benefit of using a non-quadratic quantization step size may be limited in practicality in cases where isolated points are uncorrelated with other geometric information, because two step sizes of 2 n and 2 n The cost difference of a coded bit of -1 may not decrease, while the distortion caused by a larger step size may be larger; therefore, the larger of the two step sizes may be undesirable for compression in terms of rate-distortion performance and may be more complex to implement.

[0078] In some implementations, the quantization node size (the number of bits representing the position information to be quantized) is the same as the size of the tree node containing the isolated point (e.g., IDCM node). In other implementations, the quantization node size can be smaller than the tree node and can be indicated by other means. In some implementations, the other means is the quantization tree depth or node size signaled in the slice header or parameter set.

[0079] After quantization, the encoder uses an explicit signaling method to encode the quantization point location. The number of bits to encode is derived from the node size minus any bits eliminated by quantization, such as Floor(Log2(QS)).

[0080] A decoder that encounters a node with an explicit codepoint location uses the quantization parameter information signaled in the bitstream to determine the step size and number of bits to decode using the explicit signaling method. The number of bits to decode is derived from the node size minus any bits eliminated by quantization, e.g., Floor(Log2(QS)).

[0081] The decoder scales the quantized portion of the position information according to the step size and combines it with the node position to determine the reconstructed point position.

[0082] Since nodes with explicitly encoded point positions may appear at any depth within the tree data structure, decoders must be careful about cases where the step size is larger than the node size (e.g., when all position bits are eliminated by quantization). Decoders that typically expect to use the value of Floor(Log2(QS)) to determine the difference in bits between the reconstructed point representation and the number of decoded bits (including any contribution from node positions) must restrict this value to the log2 quantized node size.

[0083] In some implementations, to simplify the implementation, the quantization parameter or quantization step size can be constrained based on the level of the tree data structure where the node with the explicitly encoded point position is located. The quantization parameter (or step size) can be constrained so that Floor(Log2(QS)) is no larger than the node size. In other implementations, the constraint can be such that Ceiling(Log2(QS)) is no larger than the node size.

[0084] In some implementations, the quantization parameter and the resulting quantization step size used for quantization / scaling of explicit coding point positions can be constrained based on the node size of the current tree level. In these implementations, the node size need not be cubic, and the constraint value is derived using the smallest node size dimension. Example code for these implementations is shown below:

[0085]

[0086] In some implementations, the encoder may specify a first "explicit" quantization parameter QP to be used during the encoding process. e (and correspondingly using the first quantization step size QS e ) to quantize isolated points (e.g., IDCM nodes), while a second "internal" quantization parameter QP is used during encoding i (and correspondingly using the second quantization step size QS i) to quantize non-isolated points (e.g., non-IDCM nodes). In some implementations, explicit quantization parameters QP may be signaled in a “geometry slice” (e.g., a data structure representing a point). e In some implementations, an explicit quantization parameter QP may be signaled in the header of the geometry slice. e In other implementations, the explicit quantization parameter QP may be signaled in the parameter set. e In another embodiment, the explicit quantization parameter QP e It can be derived from the base value signaled in the parameter set and the offset signaled in the header of the geometry slice. In another specific implementation, the explicit quantization parameter QP e It can be derived from the second (explicit quantization parameter) by an offset signaled in the header of the geometry slice or in the parameter set. In some implementations, QP e It can be called IDCM QP.

[0087] To illustrate, an exemplary syntax for a data structure (eg, a "geometry slice") is shown below.

[0088]

[0089] Table 1: Example syntax for geometry slicing.

[0090] As shown above, the quantization parameter QP used to quantize isolated points (e.g., IDCM nodes) can be signaled using the variable geom_direct_mode_qp_offset in the geometry parameter set. e In this example, geom_direct_mode_qp_offset will be the increment of geom_base_qp (e.g., the second quantization parameter QP used to quantize non-isolated points). i increment or offset).

[0091] As another example, the quantization parameter QP used to quantize isolated points can be signaled using the variable geom_direct_mode_qp_offset in the geometry slice header as shown below. e , where sliceQp=geom_base_qp+slice_qp_offset, and geom_direct_mode_qp_offset is the increment of the derived sliceQp, or the increment of geom_base_qp.

[0092]

[0093] Table 2: Example syntax of a geometry slice header.

[0094] In some implementations, the depth at which tree nodes can be quantized is signaled. This depth can be referred to as the scaling depth. In some implementations, these nodes can also contain a quantization parameter offset that indicates the degree of scaling applied to points reconstructed from the subtree represented by the node. However, because nodes containing explicitly encoded point positions may appear earlier in the tree, these nodes may escape quantization and result in reduced compression efficiency before the appropriate quantization parameter offset is signaled.

[0095] In some implementations, nodes containing explicitly coded point positions that occur before the signaled in-tree quantization depth can be quantized (or scaled) according to a first quantization parameter (eg, derived from geom_direct_mode_qp_offset).

[0096] In some implementations, if the node is a derivative of a subtree that has not been quantized (or quantization has served as an identity function), a node containing explicitly encoded point positions occurring at a depth greater than or equal to the quantization depth can be quantized using a first quantization parameter.

[0097] For nodes containing explicitly coded point positions occurring at a depth greater than or equal to the quantization depth, some implementations will not apply further quantization or adjust scaling in the decoder, resulting in these nodes using the second quantization parameter. In other words, since at the quantization point the node is determined not to contain an explicitly coded point position, the node has already been quantized using the second quantization parameter. This avoids the burden of performing two rounds of quantization or scaling; for example, inversely scaling the explicitly coded portion of the point position using the second (quantization parameter), and then inversely scaling the resulting remaining quantized portion (including the scaled explicitly coded portion).

[0098] In some implementations, if the first quantization parameter is greater than the node quantization parameter, nodes containing explicit coding point positions occurring at greater than or equal to the quantization depth are further quantized.

[0099] In other embodiments where nodes containing explicitly coded point positions are restricted to using quantization step sizes that are powers of two, nodes containing explicitly coded point positions that occur at depths greater than or equal to the quantization depth are further quantized. In some embodiments, the further quantization is based on the difference between the first quantization step size and the second quantization step size. For example, a subtree that has been quantized to a quantization node size of 2^7 and a step size of 3 (implying a quantization volume of 2^6 and therefore an effective subtree height of 6) contains an explicitly coded point position node at a relative depth of 2 (e.g., representing a quantization volume of 2^4), and where the quantization step size for such a node is 8, the node is further quantized with a step size of Floor(8 / 3)=2, resulting in a cascaded quantization step size of 6. In another embodiment, the step size for the further quantization is derived from 2^(Log2(8)-Ceiling(Log2(3)))=2. In another embodiment, the step size for further quantization is derived from 2̂(Log2(8)-Floor(Log2(3)))=4, resulting in a cascaded quantization step size of 12.

[0100] In another implementation, the third quantization parameter is signaled in a slice header or parameter set for use by nodes containing explicit coding point positions occurring at greater than or equal to the quantization depth.

[0101] Figures 4A to 4C An exemplary application of the techniques described herein is shown. Figure 4A An example of a recursively partitioned three-dimensional space 400 is shown, which includes a plurality of points (represented by circles) and a corresponding tree data structure 402. In the tree data structure, filled squares represent occupied internal tree nodes, empty squares represent unoccupied child nodes, and circles represent leaf nodes. In this example, the point labeled "a" is an isolated point. Although Figures 4A to 4C The tree data structure 402 shown in FIG. 4 is a quadtree (eg, encoding points in two dimensions), but in some implementations, the tree data structure 402 may be an octree (eg, encoding points in three dimensions).

[0102] like Figure 4B As shown, the tree data structure 402 can encode the isolated point "a" using IDCM, so that the tree terminates early with respect to the point (e.g., Figure 4B There is no path to Figure 4A This determination is performed at the location marked "a" in the tree.

[0103] like Figure 4CAs shown, the tree is divided into two parts by a line indicating the size of the quantization node. The nodes immediately after the line (e.g., to the right of the line) are allowed to signal the QP offset (e.g., Figure 4C denoted by exemplary derived QP values). The subtree labeled qp=10 terminates early because quantization eliminates one level of the tree. Node "a" appears before the quantization node size line and is therefore typically not quantized. However, as described herein, node "a" can be quantized according to explicit signaling methods.

[0104] Encoding point cloud information using non-cubic tree nodes :

[0105] exist Figure 2 In the example shown, a tree data structure has multiple nodes where each parent node has exactly eight children (e.g., an octree), corresponding to a recursive partitioning of space into successively smaller cubic subdivisions. This may be useful, for example, for encoding points in a general use case (e.g., when the points are relatively evenly dispersed in three-dimensional space).

[0106] However, this is not necessarily always the case. For example, in some implementations, the tree data structure can have many different levels. For at least one of the levels, each of the parent nodes of the level can have a corresponding first number of child nodes. Furthermore, for at least some other levels, each of the parent nodes of the level can have a corresponding second and different number of child nodes. This configuration corresponds to recursively partitioning the space into successively smaller non-cubic and / or cubic sub-portions. This is useful, for example, for encoding points that are distributed in three-dimensional space but are relatively concentrated with respect to a particular dimension relative to other dimensions. For example, the points can be distributed along an elongated rectangular prism (e.g., a point cloud representing the surface of a generally planar object). Therefore, recursively partitioning the space into elongated rectangular prisms (at least in part) can be more beneficial than partitioning the space into cubic portions (which can result in many cubic portions having few points or no points at all).

[0107] For example, reference Figure 5, the positions of points in the point cloud 110 can be encoded according to a quadtree plus binary tree (QTBT) 500, where each node 502 of the first level of the QTBT 500 has exactly four child nodes 502 (e.g., a quadtree), and each node of the second level of the QTBT 500 has exactly two child nodes 502 (e.g., a binary tree). Successive levels can alternate between the quadtree arrangement and the binary tree arrangement. Thus, the three-dimensional space 504 can be recursively subdivided into smaller elongated quadrants, then subdivided into smaller cubic portions, then subdivided into smaller elongated quadrants, then subdivided into smaller cubic portions, and so on. Each of these portions can be represented by a corresponding node 502 in the octree 500. The presence of a point in each portion can be indicated in the corresponding node 502 of the QTBT 500 (e.g., using a bit "1" to indicate the presence of a point in the corresponding portion, or using a bit "0" to indicate the absence of a point in the corresponding portion).

[0108] although Figure 5 The QTBT arrangement is shown in FIG, but in practice, the data tree structure can have other arrangements. For example, the data tree structure can have multiple different levels, where the parent node at each level has any number of child nodes (for example, the parent node at each level can have two, four, eight, sixteen, or any number of child nodes).

[0109] In some implementations, a tree data structure can initially have a non-cubic arrangement (e.g., a QTBT arrangement) for one or more upper levels, but can have a cubic arrangement (e.g., an octree arrangement) for the remaining lower levels.

[0110] In some implementations, an encoder or decoder processes a node in a particular way depending on whether the nodes that are spatially adjacent to that node are occupied. For example, the encoded occupancy information of a node can be contextualized based on whether the adjacent nodes are occupied. In the case where points are encoded according to an octree arrangement, it may not be necessary to explicitly indicate this arrangement at each level of the tree data structure. For example, since each parent node is always subdivided into exactly eight child nodes, the encoder or decoder can easily determine the size, shape, and orientation of each node and its adjacent nodes without explicitly signaling such information.

[0111] However, in non-cubic arrangements (e.g., QTBT arrangements), the arrangement of the tree data structure can be explicitly indicated so that the encoder and decoder can accurately determine the size, shape, and orientation of each node and its neighboring nodes. For example, for each level of the tree data structure, the tree data structure can explicitly signal that each node at that level has a specific number of child nodes (e.g., two, four, eight, etc.) and their spatial correspondence. This can be referred to as the "partition state" of the nodes in that level. During the encoding or decoding process, the encoder or decoder can refer to this parameter to determine the size, shape, and orientation of each node.

[0112] In some implementations, the arrangement of the tree data structure can be explicitly dictated by a parameterization process that is performed by both the encoder and decoder to determine the partitioning state of nodes in a given level of the tree. For example, one such parameterization might cause levels of four and two children to appear before any eight children; conversely, another parameterization might cause all eight children to appear before any four or two children.

[0113] As an example, the structure of a node can be defined according to the following pseudocode:

[0114]

[0115] Table 3: Pseudocode for defining the structure of a node.

[0116] In this example, the partition state of the adjacent nodes is represented by three bits, which indicate which of the x, y, and z components are encoded according to the partitioning scheme. For example, for an octree partition (e.g., each node in a level is divided into eight corresponding child nodes), all three bits are encoded. As another example, for a quadtree partition (e.g., each node in a level is divided into a fourth corresponding child node), only two of the three bits are encoded. As another example, for a binary tree partition (e.g., each node in a level is divided into two corresponding child nodes), only one bit is encoded.

[0117] In some implementations, to efficiently determine spatially neighboring nodes, a three-dimensional sparse occupancy map can be constructed. Memory-efficient implementations may choose to store occupancy information for an entire octree node in each byte of memory, comprising eight bits (one byte) in a linearly addressable array of memory locations. In some implementations, the memory locations are based on a Morton code representation of the node's spatial location information, with the least significant bit of each component subtracted, with each component forming a three-bit Morton code index into an eight-bit occupancy word. The three-dimensional sparse occupancy map can be constructed by scanning a sequence of nodes in tree traversal order.

[0118] In some implementations, a three-dimensional sparse occupancy map can be constructed from non-cubic nodes in a non-cubic tree arrangement by changing the configuration of the memory locations of the nodes. In some embodiments, only the least significant bits of spatial position information corresponding to the bits encoded in a single layer are removed. For example, if a node encodes four child nodes, each child node may not encode the position of the z0 bit (x2.x1.x0, y2.y1.y0, z2.z1). The Morton code can be constructed as x2.y2.z2.x1.y1.z1, discarding the x0 and y0 bits of position information.

[0119] In some implementations, nodes in a non-cubic tree arrangement can be quantized according to a quantization parameter (QP). In some implementations, the degree of quantization can vary between nodes within the same tree level at a specific quantization or scaling depth. Because quantization can change the height of the subtree below the quantized node, in some implementations, the partition state of a node in a first subtree with a first QP and a specific height at a level may necessarily be different from the partition state of a second subtree with a second, different QP and a different specific height. For example, two nodes in the same level may have different partition states because they belong to subtrees with different QPs.

[0120] In some embodiments, information may be stored in each node to identify the partition state of the node (e.g., the partition state of the node determined when the node or parent node is decoded). In some embodiments, this information may include a three-bit value that identifies which of the x-bit, y-bit, and / or z-bit is the most recent common bit. When constructing a three-dimensional sparse occupancy map, the node's location information and the node's partition state are used to determine the node's memory location. To continue with the previous example, a child node with location (x2.x1.x0, y2.y1.y0, z2.z1) has three bits of partition state information 0b110 stored next to it, indicating that the x and y bits were most recently encoded.

[0121] In some embodiments, information can be stored on a per-level basis to identify the partition state of the nodes at each level. For example, when constructing a three-dimensional sparse occupancy map, information about the partition state of the nodes at each level can be stored in a memory location specific to that level. Information can be selectively retrieved from the occupancy map on a per-level basis (e.g., based on an indication of the tree at a particular level, such as the encoding axis of the current level of the tree and / or a previous level of the tree).

[0122] As described above, in some implementations, the tree data structure may initially have a non-cubic arrangement (e.g., a QTBT arrangement) for one or more upper levels, but may have a cubic arrangement (e.g., an octree arrangement) for the remaining lower levels. In some implementations, the encoder may generate the tree data structure so that it has a non-cubic arrangement only for levels above the quantization or scaling depth (e.g., depth). For levels below the quantization or scaling depth, the encoder may generate the tree data structure so that it has a cubic arrangement (e.g., an octree) only for those levels.

[0123] Exemplary Process

[0124] Figure 6A An exemplary process 600 for generating a data structure representing a point cloud in three-dimensional space is shown. Process 600 may be performed at least in part using one or more devices (e.g., Figure 10 One or more computer systems shown).

[0125] According to process 600 , a computer system receives a plurality of points representing a point cloud in three-dimensional space (block 602 ).

[0126] The computer system generates a data structure representing a point cloud in three-dimensional space (block 604 ).

[0127] Generating the data structure includes, for each point, representing the position of the point in each dimension of the three-dimensional space as a bit sequence (block 604a). The position of the point is represented according to a tree data structure. In some implementations, the tree data structure can be an octree.

[0128] Additionally, the bit sequence is divided into a first portion of bits and a second portion of bits according to a zoom depth of the tree data structure (block 604b).

[0129] It is determined that a subset of points is spatially isolated from the rest of the points (block 604c).

[0130] In some implementations, determining that the subset of points is spatially isolated from the rest of the points can include determining that the subset of points is spatially isolated from the rest of the points at a first depth of the tree data structure, where the first depth is less than or equal to a zoom depth. In response to the determination, each of the second portion of bits associated with the subset of points can be quantized according to the first quantization step size.

[0131] In some implementations, determining that at a first depth, a subset of points is spatially isolated from a remainder of the points can include, at the first depth, the subset of points occupying one or more first nodes in a tree data structure, and determining that at the first depth, the remainder of the points occupying one or more second nodes in the tree data structure, wherein the one or more first nodes are not adjacent to any of the one or more second nodes.

[0132] Each of the second portion of bits associated with the subset of points is quantized according to the first quantization step size (block 604d).

[0133] Each of the second portion of bits associated with the remainder of the points (e.g., points not associated with the subset of points) is quantized according to a second quantization step size, where the first quantization step size is different from the second quantization step size (block 604e).

[0134] The quantized bit portions are included in a data structure (block 604f).

[0135] The data structure is transmitted and / or output (block 606).

[0136] In some implementations, generating the data structure can further include including a header indicating the first quantization step size in the data structure.

[0137] In some implementations, the tree data structure may include a plurality of nodes arranged according to a plurality of tree levels. Furthermore, a first node of the plurality of nodes may explicitly signal a location of a point in the point cloud. A quantization parameter and / or a quantization step size may be constrained based on a tree level of a first node of the plurality of tree nodes from the tree data structure.

[0138] Figure 6B An exemplary process 620 for rendering three-dimensional content based on a data structure representing a point cloud is shown. The process 620 may be performed at least in part using one or more devices (e.g., Figure 10 One or more computer systems shown).

[0139] According to process 620, a computer system receives a data structure representing a point cloud in three-dimensional space (block 622). The point cloud has a plurality of points. For each point, the data structure includes a bit sequence encoding the position of the point in each dimension of the three-dimensional space, wherein the position of the point is encoded according to a tree data structure, and for a subset of the points, the corresponding bit sequence is quantized according to a first quantization step size. For the remaining points, the corresponding bit sequence is quantized according to a second quantization step size, wherein the first quantization step size is different from the second quantization step size. The subset of points is spatially isolated from the remaining points. In some implementations, the tree data structure can be an octree.

[0140] The computer system determines a position of each point based on the data structure (block 624). In some implementations, this may include retrieving an indication of a first quantization step size from a header in the data structure and determining a position of each point in the subset of points based on the first quantization step size.

[0141] The computer system outputs the determined position of each point (block 626).

[0142] In some implementations, the computer system may further render three-dimensional content based on the determined position of each point. The three-dimensional content may be at least one of virtual reality content or augmented reality content.

[0143] Figure 7A An exemplary process 700 for generating a data structure representing a point cloud in three-dimensional space is shown. Process 700 may be performed at least in part using one or more devices (e.g., Figure 10 One or more computer systems shown).

[0144] According to process 700 , a computer system receives a plurality of points representing a point cloud in three-dimensional space (block 702 ).

[0145] The system generates a data structure representing a point cloud in three-dimensional space (block 704 ).

[0146] Generating a data structure includes, for each point, representing the position of the point in each dimension of the three-dimensional space as a bit sequence (block 704a). The position of the point is represented according to a tree data structure. The tree data structure includes a plurality of tree levels, each tree level having a plurality of nodes. Each node in a first tree level has a first number of corresponding child nodes in a second tree level. Each node in a second tree level has a second number of corresponding child nodes in a third tree level, wherein the first number is different from the second number. In some implementations, the first number is a cube of an integer, and the second number is not a cube of an integer.

[0147] Generating the data structure may include including in the data structure a first set of parameters indicating a first quantity and a first tree level and a second set of parameters indicating a second quantity and a second tree level (block 704b).

[0148] In some embodiments, the process may include determining the occupancy of each node in the third tree level with respect to adjacent nodes in the third tree level; performing one or more data processing steps on each node based on the determined occupancy of adjacent nodes in the third tree level; and performing one or more quantization or scaling steps relative to the data structure after performing the one or more data processing steps.

[0149] In some implementations, performing one or more quantization or scaling steps with respect to the data structure can include dividing each of the bit sequences into two or more portions of bits according to a scaling depth of the tree data structure.

[0150] In some implementations, performing one or more quantization or scaling steps with respect to the data structure can further include quantizing at least some portion of the bits according to a quantization step size; and including the quantized portion of the bits in the data structure.

[0151] Figure 7B An exemplary process 720 for rendering three-dimensional content based on a data structure representing a point cloud is shown. The process 720 may be performed at least in part using one or more devices (e.g., Figure 10 One or more computer systems shown).

[0152] According to process 720, a computer system receives a data structure representing a point cloud in three-dimensional space (box 722). The point cloud has a plurality of points. For each point, the data structure indicates the location of the point encoded according to a tree data structure. The tree data structure includes a plurality of tree levels, each tree level having a plurality of nodes. Each node in a first tree level has a first number of corresponding child nodes in a second tree level, and each node in the second tree level has a second number of corresponding child nodes in a third tree level, wherein the first number is different from the second number. The data structure includes a first set of parameters indicating the first number and the first tree level, and a second set of parameters indicating the second number and the second tree level. In some embodiments, the first number is a cube of an integer, and the second number is not a cube of an integer. In some embodiments, the tree data structure can be an octree.

[0153] The computer system determines a location of each point based on the data structure (block 724). This may include retrieving a first set of parameters and a second set of parameters from the data structure, and determining, based on the first set of parameters and the second set of parameters, that each node in the first tree level has a first number of corresponding child nodes in the second tree level, and determining that each node in the second tree level has a second number of corresponding child nodes in the third tree level.

[0154] The computer system outputs the determined position of each point (block 726).

[0155] In some implementations, the computer system may further render three-dimensional content based on the determined position of each point. The three-dimensional content may be at least one of virtual reality content or augmented reality content.

[0156] Figure 8 Shown is a compressed point cloud being used in a 3D application.

[0157] In some embodiments, a sensor (such as sensor 102), an encoder (such as encoder 104), and a decoder (such as decoder 116) may be used to transmit a point cloud in a 3D application. For example, at block 802, a sensor (such as sensor 102) may capture a 3D image, and at block 804, the sensor or a processor associated with the sensor may perform 3D reconstruction based on the sensed data to generate a point cloud.

[0158] At block 806, an encoder (such as encoder 104) may compress the point cloud, and at block 808, the encoder or post-processor may package the compressed point cloud and transmit it via network 810. At block 812, the data packet may be received at a destination location including a decoder (such as decoder 116). At block 814, the decoder may decompress the point cloud, and at block 816, the decompressed point cloud may be rendered. In some embodiments, the 3D application may transmit the point cloud data in real time, such that the display at block 816 represents the image being observed at block 802. For example, at block 816, a camera in a canyon may allow a remote user to experience walking through a virtual canyon.

[0159] Figure 9 A compressed point cloud is shown being used in a virtual reality (VR) or augmented reality (AR) application.

[0160] In some embodiments, the point cloud may be generated in software (e.g., as opposed to being captured by a sensor). For example, at block 902, virtual reality or augmented reality content is generated. The virtual reality or augmented reality content may include point cloud data and non-point cloud data. For example, a non-point cloud character may traverse terrain represented by a point cloud. At 904, the point cloud data may be compressed, and at block 906, the compressed point cloud data and the non-point cloud data may be packaged and transmitted via a network 908. For example, the virtual reality or augmented reality content generated at block 902 may be generated at a remote server and transmitted to a VR or AR content consumer via network 908. At block 910, the data packets may be received and synchronized at the VR or AR consumer's device. At block 912, a decoder operating at the VR or AR consumer's device may decompress the compressed point cloud, and the point cloud and non-point cloud data may be rendered in real time, for example, on a head-mounted display of the VR or AR consumer's device. In some embodiments, the point cloud data may be generated, compressed, decompressed, and rendered in response to the VR or AR consumer manipulating the head-mounted display to look in different directions.

[0161] In some embodiments, point cloud compression as described herein can be used in various other applications such as geographic information systems, live sports broadcasts, museum displays, autonomous navigation, etc.

[0162] Exemplary Computer System

[0163] Figure 10 An exemplary computer system 1000 is shown that can implement an encoder or decoder or any other of the components described herein (e.g., as described above with reference to FIG. Figures 1 to 9 ). The computer system 1000 can be configured to perform any or all of the embodiments described above. In various embodiments, the computer system 1000 can be any of various types of devices, including, but not limited to, a personal computer system, a desktop computer, a laptop computer, a notebook computer, a tablet computer, an all-in-one computer, a tablet or netbook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a camera, a set-top box, a mobile device, a consumer device, a video game controller, a handheld video game device, an application server, a storage device, a television, a video recording device, a peripheral device (such as a switch, a modem, a router), or generally any type of computing or electronic device.

[0164] Various embodiments of the point cloud encoder or decoder as described herein may be executed in one or more computer systems 1000, which may interact with various other devices. Figures 1 to 9 Any component, action, or functionality described may be implemented in a configuration such that Figure 10 1000. In the illustrated embodiment, the computer system 1000 includes one or more processors 1010 coupled to a system memory 1020 via an input / output (I / O) interface 1030. The computer system 1000 also includes a network interface 1040 coupled to the I / O interface 1030, and one or more input / output devices 1050, such as a cursor control device 1060, a keyboard 1070, and one or more displays 1080. In some cases, it is contemplated that embodiments may be implemented using a single instance of the computer system 1000, while in other embodiments, multiple such systems or multiple nodes comprising the computer system 1000 may be configured to host different portions or instances of the embodiments. For example, in one embodiment, some elements may be implemented via one or more nodes of the computer system 1000 that are different from those nodes that implement other elements.

[0165] In various embodiments, computer system 1000 can be a uniprocessor system including one processor 1010, or a multiprocessor system including a plurality of processors 1010 (e.g., two, four, eight, or another suitable number). Processor 1010 can be any suitable processor capable of executing instructions. For example, in various embodiments, processor 1010 can be a general-purpose or embedded processor that implements any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISAs. In a multiprocessor system, each of processors 1010 can typically, but not necessarily, implement the same ISA.

[0166] The system memory 1020 can be configured to store point cloud compression or point cloud decompression program instructions 1022 and / or sensor data that can be accessed by the processor 1010. In various embodiments, the system memory 1020 can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory. In the illustrated embodiment, the program instructions 1022 can be configured to implement an image sensor control application that incorporates any of the functionalities described above. In some embodiments, the program instructions and / or data can be received, sent, or stored on different types of computer-accessible media or similar media separate from the system memory 1020 or the computer system 1000. Although the computer system 1000 is described as implementing the functionality of the functional blocks of the preceding figures, any functionality described herein can be implemented by such a computer system.

[0167] In one embodiment, I / O interface 1030 can be configured to coordinate I / O communications between processor 1010, system memory 1020, and any peripheral devices in the device (including network interface 1040 or other peripheral device interfaces, such as input / output devices 1050). In some embodiments, I / O interface 1030 can perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., system memory 1020) into a format suitable for use by another component (e.g., processor 1010). In some embodiments, I / O interface 1030 can include support for devices attached, for example, via various types of peripheral buses (such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard). In some embodiments, the functionality of I / O interface 1030 can be divided into two or more separate components, such as a north bridge and a south bridge, for example. Furthermore, in some embodiments, some or all of the functionality of I / O interface 1030 (such as the interface to system memory 1020) can be incorporated directly into processor 1010.

[0168] The network interface 1040 can be configured to allow data to be exchanged between the computer system 1000 and other devices (e.g., carriers or proxy devices) attached to the network 1085, or between nodes of the computer system 1000. In various embodiments, the network 1085 can include one or more networks, including but not limited to a local area network (LAN) (e.g., an Ethernet or an enterprise network), a wide area network (WAN) (e.g., the Internet), a wireless data network, some other electronic data network, or some combination thereof. In various embodiments, the network interface 1040 can support communication via a wired or wireless general data network (such as any suitable type of Ethernet network), for example; via a telecommunications / telephone network (such as an analog voice network or a digital fiber optic communication network); via a storage area network (such as a Fibre Channel SAN), or via any other suitable type of network and / or protocol.

[0169] In some embodiments, input / output devices 1050 may include one or more display terminals, keyboards, keypads, trackpads, scanning devices, voice or optical recognition devices, or any other device suitable for inputting or accessing data by one or more computer systems 1000. Multiple input / output devices 1050 may be present in computer system 1000 or may be distributed across various nodes of computer system 1000. In some embodiments, similar input / output devices may be separate from computer system 700 and may interact with one or more nodes of computer system 1000 through a wired or wireless connection, such as through network interface 1040.

[0170] like Figure 10As shown, memory 1020 may include program instructions 1022, which may be executed by a processor to implement any of the elements or actions described above. In one embodiment, the program instructions may execute the method described above. In other embodiments, different elements and data may be included. Note that the data may include any of the data or information described above.

[0171] Those skilled in the art will appreciate that computer system 1000 is merely illustrative, and is not intended to limit the scope of the embodiments. Specifically, computer system and equipment may include any combination of hardware or software that can perform the indicated function, including computers, network equipment, internet equipment, personal digital assistants (PDAs), wireless phones, pagers, etc. Computer system 1000 may also be connected to other devices not shown, or may otherwise operate as an independent system. In addition, the functions provided by the shown components may be combined in fewer components or distributed in additional components in some embodiments. Similarly, in some embodiments, the functions of some components in the shown components may not be provided, and / or other additional functions may be available.

[0172] Those skilled in the art will also recognize that, although various projects are shown as being stored in memory or on storage device during use, for the purpose of memory management and data integrity, these projects or parts thereof can be transmitted between memory and other storage devices. Alternatively, in other embodiments, some or all of these software components can be executed in the memory on another device, and communicate with the illustrated computer system via inter-computer communication. Some or all of system components or data structures also can be stored on computer accessible media or portable articles (for example as instructions or structured data) to be read by suitable drivers, and its multiple examples are described above. In some embodiments, the instructions stored on the computer accessible media separated from computer system 1000 can be transferred to computer system 1000 by transmission media or signal (such as electrical signals, electromagnetic signals or digital signals transmitted by communication media such as networks and / or wireless links). Various embodiments can also include receiving, sending or storing instructions and / or data implemented according to the above description on computer accessible media. Generally speaking, computer-accessible media may include non-transitory computer-readable storage media or memory media, such as magnetic or optical media, e.g., disks or DVD / CD-ROMs, volatile or non-volatile media, such as RAM (e.g., SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc. In some embodiments, computer-accessible media may include transmission media or signals, such as electrical, electromagnetic, or digital signals transmitted via a communication medium, such as a network and / or wireless link.

[0173] In different embodiments, the methods described herein can be implemented in software, hardware, or a combination thereof. In addition, the order of the method frames can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. For those skilled in the art who benefit from this disclosure, it is obvious that various modifications and changes can be made. The various embodiments described herein are intended to be illustrative and not restrictive. Many variations, modifications, additions, and improvements are possible. Therefore, multiple examples can be provided for the components described as a single example in this article. The boundaries between various components, operations, and data repositories are arbitrary to a certain extent, and specific operations are shown in the context of specific exemplary configurations. Other allocations of functions are contemplated and may fall within the scope of the appended claims. Finally, the structure and function presented as discrete components in the exemplary configuration may be implemented as a combined structure or component. These and other variations, modifications, additions, and improvements may fall within the scope of the embodiments defined in the following claims.

Claims

1. A method comprising: receiving, by a computer system, a data structure representing a point cloud in three-dimensional space, the point cloud having a plurality of points, wherein the data structure comprises, for each point, a sequence of bits encoding the position of the point in each dimension of the three-dimensional space, wherein the position of the point is encoded according to a tree data structure representing a plurality of octants, wherein for the subset of points, a corresponding bit sequence represents an explicit signaling of the position of the subset of points relative to one or more reference points in the tree data structure according to a direct coding mode, and the corresponding bit sequence is quantized according to a first quantization step size, wherein, for the remaining portion of the point, the corresponding bit sequence represents the position of the remaining portion of the point according to the tree data structure without the direct encoding mode, and the corresponding bit sequence is quantized according to a second quantization step size, wherein the first quantization step size is different from the second quantization step size, wherein said subset of said points occupies one or more first eighths of said plurality of eighths, wherein each first one-eighth space of the one or more first one-eighth spaces is surrounded by a corresponding unoccupied one-eighth space of the plurality of one-eighth spaces, Determining, by the computer system, the position of each of the points based on the data structure, wherein determining the position of each of the points comprises: For a subset of the points: decoding the corresponding bit sequence according to the direct encoding mode to determine first position information representing the position of the subset of points relative to the one or more reference points, and scaling the first position information by the first quantization step size, and; For the rest of the points: decoding the corresponding bit sequence to determine second position information representing the position of the remaining portion of the point according to the octree, and Dequantizing the second position information according to the second quantization step size; and The determined position of each of the points is output by the computer system.

2. The method of claim 1 , wherein determining the position of each of the points based on the data structure comprises retrieving an indication of the first quantization step size from a header in the data structure, and The position of each of the points in the subset of points is determined based on the first quantization step size. The method of claim 1 , wherein the tree data structure is an octree. 4 . The method of claim 1 , further comprising rendering three-dimensional content based on the determined position of each of the points, wherein the three-dimensional content is at least one of virtual reality content or augmented reality content.

5. A device comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: receiving a data structure representing a point cloud in three-dimensional space, the point cloud having a plurality of points, wherein the data structure comprises, for each point, a sequence of bits encoding the position of the point in each dimension of the three-dimensional space, wherein the position of the point is encoded according to a tree data structure representing a plurality of octants, wherein for the subset of points, a corresponding bit sequence represents an explicit signaling of the position of the subset of points relative to one or more reference points in the tree data structure according to a direct coding mode, and the corresponding bit sequence is quantized according to a first quantization step size, wherein, for the remaining portion of the point, the corresponding bit sequence represents the position of the remaining portion of the point according to the tree data structure without the direct encoding mode, and the corresponding bit sequence is quantized according to a second quantization step size, wherein the first quantization step size is different from the second quantization step size, wherein said subset of said points occupies one or more first eighths of said plurality of eighths, wherein each first one-eighth space of the one or more first one-eighth spaces is surrounded by a corresponding unoccupied one-eighth space of the plurality of one-eighth spaces, Determining the position of each of the points based on the data structure, wherein determining the position of each of the points comprises: For a subset of the points: decoding the corresponding bit sequence according to the direct encoding mode to determine first position information representing the position of the subset of points relative to the one or more reference points, and scaling the first position information by the first quantization step size, and; For the rest of the points: decoding the corresponding bit sequence to determine second position information representing the position of the remaining portion of the point according to the octree, and Dequantizing the second position information according to the second quantization step size; and The determined position of each of the points is output.

6. The apparatus of claim 5, wherein determining the position of each of the points based on the data structure comprises retrieving an indication of the first quantization step size from a header in the data structure, and The position of each of the points in the subset of points is determined based on the first quantization step size. The apparatus of claim 5 , wherein the tree data structure is an octree.

8. The apparatus of claim 5, the operations further comprising rendering three-dimensional content based on the determined position of each of the points, wherein the three-dimensional content is at least one of virtual reality content or augmented reality content.

9. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform operations comprising: receiving a data structure representing a point cloud in three-dimensional space, the point cloud having a plurality of points, wherein, for each point, the data structure comprises a bit sequence encoding a position of the point in each dimension of the three-dimensional space, wherein the position of the point is encoded according to a tree data structure representing a plurality of octants, wherein for a subset of the points, a corresponding bit sequence represents an explicit signaling of the position of the subset of points relative to one or more reference points in the tree data structure according to a direct coding mode, and wherein the corresponding bit sequence is quantized according to a first quantization step size, in, For the remaining portion of the point, the corresponding bit sequence represents the position of the remaining portion of the point according to the tree data structure without the direct encoding mode, and the corresponding bit sequence is quantized according to a second quantization step size, wherein the first quantization step size is different from the second quantization step size, and wherein said subset of said points occupies one or more first eighths of said plurality of eighths, wherein each first one-eighth space of the one or more first one-eighth spaces is surrounded by a corresponding unoccupied one-eighth space of the plurality of one-eighth spaces, Determining the position of each of the points based on the data structure, wherein determining the position of each of the points comprises: For a subset of the points: decoding the corresponding bit sequence according to the direct encoding mode to determine first position information representing the position of the subset of points relative to the one or more reference points, and scaling the first position information by the first quantization step size, and; For the rest of the points: decoding the corresponding bit sequence to determine second position information representing the position of the remaining portion of the point according to the octree, and Dequantizing the second position information according to the second quantization step size; and The determined position of each of the points is output.

10. The non-transitory computer-readable storage medium of claim 9, wherein determining the position of each of the points based on the data structure comprises retrieving an indication of the first quantization step size from a header in the data structure, and determining the position of each of the points in the subset of points based on the first quantization step size.

11. The non-transitory computer-readable storage medium of claim 9, wherein the tree data structure is an octree.

12. The non-transitory computer-readable storage medium of claim 9, the operations further comprising rendering three-dimensional content based on the determined position of each of the points, wherein the three-dimensional content is at least one of virtual reality content or augmented reality content.

Images