Compressing point cloud data frames

Abstract

A method for encoding frames of a sequence of frames of point cloud data for rendering a dynamic three-dimensional scene, each frame including a plurality of points, each point including a three-dimensional position and one or more attributes, the method comprising: obtaining a current point to be encoded in a current frame; identifying a previous point encoded in a previous frame that corresponds to the current point to be encoded in the current frame; encoding at least one attribute of the current point by referencing the previous point using inter-frame predictive coding to generate encoded data for the current point; and generating the current frame including the encoded data for the current point.

Description

[Technical field]

[0001] The present disclosure relates to a method for encoding and decoding frames of a sequence of frames of point cloud data for rendering a three-dimensional dynamic scene, and also to an apparatus and a computer program for implementing such a method. Furthermore, the present disclosure relates to a frame structure of the encoded frames. [Background technology]

[0002] For example, point cloud data is increasingly being used to represent three-dimensional scenes in virtual reality or augmented reality applications, in which the point cloud data is stored on or communicated to a display device, such as a screen or near-eye display, and rendered to display the three-dimensional scene.

[0003] In such a three-dimensional scene, a large number of points may be used to represent various objects or object surfaces in the scene. Typically, each point has a three-dimensional position and one or more attributes, which may include, for example, surface color, transparency value, object size, and surface normal direction. Each attribute may have a value selected from a continuous range or may have a value selected from a discrete set.

[0004] In a dynamic 3D scene, each point can change either its 3D position or its attributes. A sequence of instances of a 3D scene is represented as a sequence of frames, which can be rendered to reconstruct the dynamic 3D scene, including the changes in the 3D position and attributes of each point, as well as the addition or removal of points. In most cases, the sequence of frames represents an evenly spaced time sequence (e.g., some regular frame rate of every second). However, the frames of a sequence may be spaced at unequal time intervals, or may not be linked to a time sequence at all, such as when the sequence is intended to be stepped through manually.

[0005] One particular application of point cloud data is volumetric video, where techniques such as motion capture are used to acquire point cloud data as a three-dimensional recording of a real-world scene, or as a mix of real-world scenes with computer-generated elements. Typically, in volumetric video, the density of points in the point cloud can be high enough to accurately reproduce human movements or facial expressions.

[0006] Uncompressed point cloud data formats contain a complete definition of every point for every frame. This can require large amounts of data storage and can mean that live rendering or streaming of dynamic 3D scenes is not possible. It is therefore desirable to provide an efficient compressed format for point cloud data.

[0007] Known geometry-based point cloud compression techniques use logical structures such as octrees to efficiently describe a sparse set of points in 3D space in a split second, which are then used to predictively code attributes between relatively close points (i.e., intraframe predictive coding), as described in GRAZIOSI et al., "An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC)", DOI:10.1017 / ATSIP.2020.12.

[0008] Another known geometry-based point cloud compression technique uses motion vectors to represent the texture or object movement in successive frames, as described in ROODAKI et al., “G-Arrays: Geometric arrays for efficient point cloud processing”, DOI: 10.1109 / ICASSP39728.2021.9413902. Summary of the Invention

[0009] The present disclosure provides techniques for encoding and decoding point cloud data with improved compression efficiency.

[0010] More specifically, according to a first aspect, the present invention provides a method for encoding a frame of a sequence of frames of point cloud data for rendering a dynamic three-dimensional scene, each frame comprising a plurality of points, each point comprising a three-dimensional position and one or more attributes, the method comprising: obtaining a current point to be encoded in a current frame; identifying a previous point encoded in a previous frame that corresponds to the current point to be encoded in the current frame; and encoding at least one attribute of the current point by referencing the previous point using inter-frame predictive coding to generate encoded data for the current point.

[0011] As used herein, "interframe predictive coding" refers to coding the difference between a characteristic of one frame and the same characteristic of another frame. For example, interframe predictive coding of an attribute of a current point to be coded in a current frame means coding the difference between the value of that attribute of the current point in the current frame and the value of that attribute of a previous point that is coded in a previous frame.

[0012] In a feature of the first aspect, the labels "current" and "previous" are used to identify which instant in a dynamic three-dimensional scene is represented by the corresponding point or frame. As used herein, "previous" does not necessarily mean "immediately before," and "previous frame" may refer to different previous frames for each point, depending on how long (i.e., for how many frames) the particular point has been suitable for inter-frame predictive coding.

[0013] A feature of the first aspect is that it provides inter-frame prediction of attributes other than position for point cloud data, thereby improving compression efficiency when attributes such as lighting and color of a point change over the course of a dynamic scene.

[0014] The method may further include generating a current frame, including the encoded data for the current point. The "generating a current frame" step may simply include recording the encoded data for the current point in memory, or transmitting the encoded data for the current point to a separate device. Once generated, the current frame may be transmitted as a stream, with each current point having encoded data transmitted separately. Alternatively, the entire frame may be recorded in memory, or the entire current frame, having multiple points, may be encoded before being transmitted from the encoder to a separate device.

[0015] Optionally, identifying the previous point corresponding to the current point includes identifying a previous point coded in a previous frame that has the same three-dimensional position as the current point. This provides a method for selecting points suitable for inter-frame prediction of attributes. In this application, "same three-dimensional position" means three-dimensional positions that are similar within a predetermined level of accuracy, where the predetermined level of accuracy may be a relative or absolute level selected according to a particular use case of the method. For example, a characteristic distance between points may be determined based on the total volume of the three-dimensional scene and the number of points in the scene, or based on a minimum distance between points in a previous frame. If the displacement between the previous point and the current point is much smaller than the characteristic distance, it may be acceptable to assume that the previous point corresponds to the current point. On the other hand, if a three-dimensional scene is expected to have points appear and disappear regularly between consecutive frames, it may be necessary to define "same position" as a matched position.

[0016] Optionally, one or more attributes of each point include non-predictive attributes, and identifying the predecessor point corresponding to the current point includes identifying a predecessor point coded in the previous frame that has the same three-dimensional position as the current point and has the same value of the non-predictive attribute as the current point. This further improves the method of selecting points suitable for inter-frame prediction of attributes by using inter-frame prediction when the three-dimensional position of the point as well as another "non-predictive" attribute, such as surface orientation, is the same as the previous frame. Any attribute may be non-predictive. For example, an attribute may be selected to be "non-predictive" if it is not suitable for inter-frame prediction or if it is particularly vulnerable to compression-related errors. In this application, "same value of non-predictive attribute" means similar values ​​within a predetermined level of accuracy. Furthermore, in this method, one or more attributes of each point may include multiple non-predictive attributes. Optionally, different levels of accuracy can be pre-set for each non-predictive attribute.

[0017] Optionally, the method includes obtaining a first portion of the current points to be coded in the current frame, and for each current point of the first portion, identifying whether inter-frame predictive coding is available for the current point, where inter-frame predictive coding is available if there is a previous point coded in the previous frame that corresponds to the current point to be coded in the current frame, and encoding the frame by performing inter-frame predictive coding for each current point for which inter-frame predictive coding is available, and performing intra-frame predictive coding for each current point for which inter-frame predictive coding is not available. In this specification, intra-frame predictive coding is understood to mean "attempt of intra-frame coding", and some "intra-frame predictive coding points" may not actually be changed during coding if there is no associated reference in a point of the frame.

[0018] Optionally, the method further comprises generating encoding type information for each point of said current frame indicating an encoding type, said type being inter-frame predictive encoding or intra-frame predictive encoding. The encoding type information identifies which points are partially defined by one or more previous frames, thereby aiding a decoder in rendering the compressed point cloud data. The encoding type information may be included when generating the current frame for recording or transmission.

[0019] Optionally, in a first particular type of embodiment, the method further comprises generating ordering information defining an order of a second portion of the plurality of current points to be coded in the current frame, the current frame being generated including the ordering information and including coded data ordered according to the ordering information. The second portion may be selected independently of the first portion, and the first and second portions may overlap partially or completely. For example, the points of the second portion may be a subset of the points of the first portion to be intraframe predictive coded. In another example, the ordering information may describe a complete order of all points.

[0020] Optionally, in a first particular type of embodiment, the ordering information includes a three-dimensional position of each point in the current frame, which may be defined in an overall space encompassing the entire three-dimensional scene, or may be a position in a smaller local space, such as a node of an octree that divides the overall space.

[0021] Optionally, in a first particular type of embodiment, the ordering information comprises an octree based on the three-dimensional position of each point in the current frame, where a volume is recursively divided along the X, Y and Z axes into eight sub-volumes and a tree search of points associated with each node of the octree is created. An octree has the advantage that it reduces the amount of data required to define the position of each point.

[0022] Optionally, in a first particular type of embodiment, each leaf node of said octree contains at least one three-dimensional location of a point of said current frame. That is, while a volume is recursively divided into sub-volumes, sub-volumes that do not contain points are not further divided and are omitted when storing the octree in memory. This "pruning" of unnecessary branches of the octree reduces the size of the compressed data and the time required to read, decode and render the compressed data.

[0023] Optionally, in a first particular type of embodiment, the points of the current frame are ordered by the first, second, and third dimensions of each point's three-dimensional location. This provides an alternative default way of sorting the points, and can be combined with an octree. For example, an octree can divide the total space up to three levels, reducing the volume per octree node to 1 / 512 of the total space. If an octree node is still associated with multiple points located close to each other in the total space, the multiple points can be ordered by the third dimension of the points' three-dimensional location.

[0024] In a second particular type of embodiment, which can be optionally combined with the first particular type of embodiment, the method further comprises encoding a third portion of the current points of the current frame using intraframe predictive coding, the third portion being defined to be independent of the first and second portions of the current points, although in many cases the third portion may be the same as the second portion.

[0025] Optionally, in a second particular type of embodiment, the method further includes intraframe predictively encoding a first attribute of the one or more attributes, where intraframe predictively encoding the first attribute includes obtaining an average of the first attribute for the current points, and obtaining, for each of the current points, a difference between the average and the value of the first attribute at the current point. This average differencing technique can reduce the amount of bits required to represent the current points. The average differencing technique can be applied to interframe predictive encoding as well.

[0026] Optionally, in a second particular type of embodiment, the plurality of current points is a local group of points corresponding to nodes at a given level of the octree.

[0027] Optionally, in a second particular type of embodiment, the method includes intraframe predictively encoding a second attribute of the one or more attributes, where intraframe predictively encoding the second attribute includes obtaining a raw value of the second attribute for a first current point and obtaining an encoded value of the second attribute for a second current point, the encoded value being equal to a difference between raw values ​​of the second attribute at the second current point and the first current point.

[0028] Optionally, the method comprises predictively coding a third attribute of the one or more attributes for each fourth portion of the plurality of current points of the frame; performing sparse array compression on the coded values ​​of the third attribute of the fourth portion of the plurality of current points, the sparse array compression including dividing the coded values ​​of the third attribute into groups of a predetermined size N and generating a sparse array identifying whether each group contains only coded values ​​below a predetermined threshold; and generating the current frame including the sparse array and the groups storing at least one coded value above the predetermined threshold. Compression is further improved by excluding zero or near-zero groups. Here, the fourth portion shall be considered as independent and independent from the first, second and third portions. This optional feature is equally applicable to intraframe predictive coding as it is to interframe predictive coding. The "predetermined threshold" is zero for attributes that cannot be reasonably decoded lossily. For example, the predetermined threshold for a color attribute may be greater than zero if it does not have a noticeable effect on the rendering of the three-dimensional scene, but should be zero for other example attributes, such as attributes that indicate one of a discrete set of possible lighting positions.

[0029] Optionally, the method further comprises performing run-length compression on the sparse array, which can further improve compression.

[0030] Optionally, the method further includes applying entropy coding, such as VC-6 coding, to at least one of the current frame ordering information, the coding type information, and the coded value of the fourth attribute. Entropy coding can further improve compression. VC-6 is described in related standard specifications, including PCT / GB2018 / 053552 and "ST 2117-1:2020-SMPTE Standard-VC-6 Multiplanar Picture Format-Part 1 Elementary Bitstream" published on July 21, 2020 (DOI10.5594 / SMPTE.ST2117-1.2020 and electronic ISBN: 978-1-68303-219-9), all of which are incorporated herein by reference in their entirety.

[0031] Optionally, the entropy coding method is Huffman coding, range coding, or a method compliant with the VC-6 standard.

[0032] According to a second aspect of the present disclosure, there is provided a method for decoding a frame of a sequence of frames of point cloud data for rendering a dynamic three-dimensional scene, each frame including a plurality of points, each point including a three-dimensional position and one or more attributes, the method comprising: obtaining a current point to be decoded in a current frame; identifying a previous decoded point in a previous frame that corresponds to the current point to be decoded in the current frame, and decoding at least one attribute of the current point by referencing the previous point using inter-frame prediction to generate decoded data for the current point. This decoding method corresponds to the encoding method of the first aspect, and similarly, optional features of the first aspect may also be optionally included in an implementation of the second aspect to provide a decoding equivalent to the corresponding encoding described above.

[0033] More specifically, optionally, identifying the predecessor point that corresponds to the current point includes identifying a predecessor point decoded in a previous frame that has the same three-dimensional position as the current point.

[0034] Further optionally, the one or more attributes of each point include a non-predictive attribute, and identifying the predecessor point corresponding to the current point includes identifying a predecessor point encoded in the previous frame that has the same three-dimensional position as the current point and that has the same value of the non-predictive attribute as the current point.

[0035] Further optionally, the method of decoding a frame includes obtaining a first portion of the plurality of current points to be decoded in the current frame; The method includes: for each current point of the first portion, identifying whether inter-frame predictive coding was used for the current point, where inter-frame predictive coding is usable if there is a previous point coded in the previous frame that corresponds to the current point to be coded in the current frame; and decoding the frame by performing inter-frame predictive decoding for each current point for which inter-frame predictive decoding is usable, and performing intra-frame predictive decoding for each current point for which inter-frame predictive decoding is not usable.

[0036] Further, optionally, the current frame includes coding type information indicating a coding type for each point of said current frame, said type being inter-frame predictive coding or intra-frame predictive coding.

[0037] Furthermore, optionally for the second aspect, the method further comprises decoding ordering information defining an order of a second portion of said plurality of current points encoded within said current frame.

[0038] Additionally, optionally for the second aspect, the method further includes decoding a third portion of the plurality of current points of the current frame using intraframe predictive decoding.

[0039] When the method of the second aspect includes decoding the third portion using intraframe predictive decoding, the method further includes predictively decoding a first attribute of the one or more attributes, and intraframe predictively decoding the first attribute also includes, for each of the plurality of current points, adding an average of the first attribute for the plurality of current points to a difference between the average and the value of the first attribute for the current point.

[0040] Additionally or alternatively, according to a second aspect, the current points may be a local group of points corresponding to nodes at a given level of the octree.

[0041] Optionally, according to a second aspect, the method includes intraframe predictive decoding a second attribute of the one or more attributes, where intraframe predictive decoding of the second attribute includes obtaining a raw value of the second attribute for a first current point and obtaining a decoded value of the second attribute for a second current point, the decoded value being equal to the sum of the encoded value of the second attribute for the second current point and the raw value of the second attribute for the first current point.

[0042] Optionally, according to a second aspect, the method further includes inverting the sparse array compression on the encoded values ​​of the third attribute for each of a fourth portion of the plurality of current points of the frame, where inverting the sparse array compression includes reading the sparse array of encoded values ​​and identifying groups of encoded values ​​to which sparse array compression was applied during encoding, and for each group of encoded values ​​to which sparse array compression was applied, inserting a group of encoded zero values ​​of a predetermined size N at each location to which sparse array compression was applied.

[0043] Optionally, according to a second aspect, the method further comprises inverting the run-length compression.

[0044] According to a third aspect, the present disclosure provides an encoder configured to perform an encoding method as described above in the first aspect.

[0045] According to a fourth aspect, the present disclosure provides a decoder configured to perform a decoding method as described above in the second aspect.

[0046] According to a fifth aspect, the present disclosure provides a computer program comprising instructions which, when executed, perform an encoding method as described above in the first aspect.

[0047] According to a sixth aspect, the present disclosure provides a computer program comprising instructions which, when executed, perform a decoding method as described above in the second aspect.

[0048] The computer program of the fifth or sixth aspect may be stored as instructions on a non-transitory computer readable medium or in a signal transmission.

[0049] According to a seventh aspect, the present disclosure provides frames of a sequence of frames of point cloud data for rendering a dynamic three-dimensional scene, each frame comprising a plurality of points, each point comprising a three-dimensional position and one or more attributes, the frames comprising encoded data for each point of the frames, the encoded data for at least one point being encoded by inter-frame predictive coding.

[0050] The frame of the seventh aspect may be generated according to the method of the first aspect.

[0051] Although the above aspects describe techniques applied to point cloud data, the techniques described herein are quite broadly applicable to any data in the form of a sequence of frames, where each frame includes multiple data elements, each data element including a two-dimensional or three-dimensional position and one or more attributes. For example, the described techniques can be applied to volumetric video recorded as mesh data rather than point cloud data. [Brief description of the drawings]

[0052] [Figure 1]FIG. 2 shows a schematic example of a sequence of frames of point cloud data for rendering a dynamic three-dimensional scene. [Diagram 2] FIG. 1 is a schematic block diagram of a point cloud data communication system. [Diagram 3] 1 is a flow chart that generally illustrates a method for encoding a frame of a sequence of frames of point cloud data. [Figure 4] FIG. 2 is a schematic diagram of a sequence of encoded frames; [Diagram 5] 1 is a flow chart that generally illustrates a method for encoding a frame of a sequence of frames of point cloud data. [Figure 6] 1 is a flow chart that generally illustrates a method for decoding a frame of a sequence of frames of point cloud data. [Figure 7] FIG. 2 is a schematic block diagram of an encoder or a decoder. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0053] Figure 1 is a highly simplified schematic example of the kind of point cloud data to which the described techniques may be applied. More specifically, Figure 1 shows a schematic example of a sequence of frames of point cloud data for rendering a dynamic three-dimensional scene.

[0054] The sequence of frames in Figure 1 contains only two frames, and although in principle the described technique can be applied to only two frames, the described technique offers greater compression efficiency for longer frame sequences.

[0055] Referring to FIG. 1, the sequence of frames includes a first frame 10-1 and a second frame 10-2.

[0056] In a first frame 10-1, the three-dimensional scene includes six points 20-A to 20-F, located at respective three-dimensional positions within the volume of the scene.

[0057] In this simple example, the six points 20-A through 20-F are separate solid objects, more specifically, cubes. More generally, groups of points in a three-dimensional scene may be associated with different surfaces of a single solid object, or may be associated with various objects appearing in the scene.

[0058] Each of the six points 20-A-20-F represents some attribute that may be included in a sequence of frames representing a scene. Each cube has a size and a color, which may be viewed as being represented by the grayscale shaded face of the cube, with darker shading representing a darker color. Additionally, the shaded face of each cube represents the location of a light source, which points to the opposite side of the cube. More generally, the attributes may include any property of a point cloud 3D scene that needs to be displayed when the point cloud data is rendered for a viewer.

[0059] In the second frame 10-2, the three-dimensional scene has changed. The scene now contains seven points 20-A through 20-G. The three-dimensional positions of points 20-E and 20-F have changed, and point 20-G has been added. Also, the light source has moved so that the shaded facing portion of each object is now the "front" surface, as shown in the figure. Additionally, point 20-F has also decreased in size.

[0060] FIG. 2 is a schematic block diagram of a point cloud data communication system that may employ the following computer-implemented methods provided herein.

[0061] Referring to FIG. 2, the system includes a point cloud data generator 210, an encoder 220, a decoder 240, a renderer 250, and a display 260.

[0062] The point cloud data generator 210 may be a known system for generating point cloud data in the format of a sequence of frames, where each frame includes a number of points, and where each point includes a three-dimensional position and one or more attributes. For example, the point cloud data generator 210 may generate a sequence of frames, such as that shown in FIG.

[0063] The sequence of frames is sent to an encoder 220, which performs a method for encoding a frame (usually multiple frames) of the sequence of frames.

[0064] Since the sequence of encoded frames is compressed by encoding, the encoded sequence can be transmitted over a network connection 231 and / or stored on a physical medium 232 (such as a DVD, a flash drive, or a server). The network connection 231 and the physical medium 232 can include any content-independent data communication mode.

[0065] Finally, when a user wishes to view the dynamic three-dimensional scene, the sequence of encoded frames is provided to a decoder 240 via a network connection 231 or a physical medium 232 .

[0066] The decoder 240 performs a method for decoding a frame (usually multiple frames) of a sequence of frames.

[0067] The decoded frame(s) are then passed to a renderer 250, which renders the dynamic three-dimensional scene for display on a display 260. The renderer 250 and display 260 can be any renderer and display currently used for point cloud data, and the renderer 250 is configured to receive the point cloud data in the format of a sequence of frames, where each frame includes a number of points, and where each point includes a three-dimensional position and one or more attributes.

[0068] As can be seen from Fig. 2 and the above description, the present invention mainly relates to encoding and decoding methods, encoders and decoders configured to perform the methods, and encoded data produced by the encoding method and used by the decoding method. Meanwhile, the initial point cloud data received by the encoder and reproduced by the decoder may be conventional point cloud data. Therefore, the generator 210 and the renderer 250 will not be described in detail.

[0069] Figure 3 is a flow diagram illustrating generally a method of encoding a frame of a sequence of frames of point cloud data, and Figure 4 is a schematic diagram of a sequence of encoded frames. It is useful to switch between Figures 3 and 4 to explain the encoding method applied to the sequence of frames of Figure 1.

[0070] Referring to FIG. 3, a key concept of the encoding method disclosed herein is to distinguish between some point cloud points that should be predicted using inter-frame predictive coding and, more preferably, other point cloud points that are predicted using intra-frame predictive coding.

[0071] There are many reasons to choose between interframe coding and intraframe coding.

[0072] For example, interframe coding of moving points can be quite resource intensive due to the need to determine which points in successive frames are in fact the same points. As a result, the inventors have found that it is preferable to perform interframe coding only on point cloud points that are stationary in and before the current frame.

[0073] On the other hand, lighting and color changes are very common in dynamic 3D scenes and can affect a large number of points while being relatively easy to track between frames. As a result, the inventors have found that it is preferable to perform interframe coding on the color attributes.

[0074] 3, in step S310, the encoder 220 evaluates the points of the current frame to be encoded and identifies what type of encoding to use for each point. That is, step S310 includes identifying, for each of a plurality of current points of the frame, whether inter-frame predictive encoding should be used for the current point.

[0075] For example, the encoder 220 may be configured to compare a current frame to a previous frame and determine whether each point in the current frame was present in the previous frame.

[0076] In this case, if the previous frame has a previous point at the same position as a point in the current frame, the encoder 220 determines that it should perform interframe predictive coding on the current point in the current frame by referencing the previous point in order to generate encoded data for the current point in the current frame.

[0077] In alternative embodiments, the encoder 220 may have stricter requirements for identifying a previous point in a previous frame as corresponding to a current point in a current frame. For example, the encoder 220 may require not only that the current point and the previous point have the same three-dimensional location, but also that they have one or more other attributes in common, such as the same size, or the same surface orientation. Such additional required attributes may be designated as "fixed attributes" or "non-predictive attributes" for a given application of the encoding.

[0078] This alternative embodiment works similarly, and if the strict requirements are met, the encoder 220 similarly determines that interframe predictive coding should be performed on the current point in the current frame, with reference to the previous point, to generate encoded data for the current point in the current frame.

[0079] If the encoder 220 does not identify a previous point coded in a previous frame that corresponds to the current point in the current frame, the encoder 220 determines that it should either code the current point with intraframe predictive coding or omit predictive coding of that point entirely. For the purposes of this description, we will assume that the case without predictive coding is a special case of intraframe predictive coding.

[0080] Applying step S310 to the sequence of frames of FIG. 1 would cause encoder 220 to treat second frame 10-2 as the current frame and first frame 10-1 as the previous frame.

[0081] Given the requirement that there must be a predecessor point corresponding to the current point, encoder 220 would determine that points 20-A to 20-D should be coded using interframe predictive coding, and points 20-E to 20-G should be coded using intraframe predictive coding.

[0082] In step S320, the encoder generates coding type information indicating the coding type for each point. Referring to FIG. 4, the coding type information 460 for frame 10-2 may include one bit for each point in the point cloud of the current frame 10-2. In fact, the coding type information may more efficiently omit points that were not present in the previous frame, since such points cannot perform interframe predictive coding, regardless of any relaxation of the rules. As a result, the bit array 460 includes only six bits for the six points 20-A to 20-F that appeared in the previous frame (first frame 10-1).

[0083] As described above, when applying step S310 to the example of Fig. 1, only points 20-A to 20-D need to be coded using interframe prediction. Therefore, as can be seen in Fig. 4, coding type information 460 of frame 10-2 has four 1's (indicating that points 20-A to 20-D are coded using interframe prediction) and two 0's (indicating that points 20-E and 20-F are coded using intraframe prediction).

[0084] In step S330, the encoder 220 generates ordering information 410 that defines the order of points to be encoded in the current frame. Because this ordering information 410 is included only once, additional information about each point can be collected within a layer for each attribute without having to identify the points again for each layer, which would otherwise add complexity to both the amount of memory required to store the encoded data and the encoding and decoding processes.

[0085] It may be most efficient to restrict the ordering information 410 to omit the inter-predictively coded points, since they necessarily already have an order defined in the previous frame. Also, since the previous frame is available to the decoder, including the ordering of the inter-predictively coded points would be redundant. Thus, as seen in the particular example of Figure 4, the coded frame 10-2 may simply have ordering information 410 for the intra-predictively coded points 20-E, 20-F, and 20-G.

[0086] Of course, the format of the ordering information and the method for selecting the order must be specified.

[0087] The simplest way to order the points may be an XYZ sort. More specifically, each point is associated with a three-dimensional location that can be defined numerically in Cartesian dimensions. To sort the points into a single list based on their numerical three-dimensional location, the order of the dimensions can be chosen arbitrarily.

[0088] However, a single list may not always be the most efficient strategy: in particular, it is not the fastest way to compare both frames to identify points that occur in both frames (as required in step S310).

[0089] The solution, the Octree, is known as a structured storage of 3D points that allows efficient lookup of points in 3D space. In an Octree, a volume is bisected recursively along the X, Y, and Z axes, resulting in 8 subvolumes. The volumes and subvolumes are traversed in a tree structure, allowing the lookup of each node of the Octree, i.e. the points associated with each volume and subvolume.

[0090] Nevertheless, at lower levels of the octree structure, the number of points per tree node may be small, at which point it may be more efficient to stop the recursive splitting and use only a single list. For this Cartesian sort, the ordering information 410 may include the absolute or relative three-dimensional position of each point that is subject to intraframe predictive coding, either globally or within a leaf node of the octree.

[0091] To store the octree in memory or to generate a data stream that includes the octree, a linearization method must be selected, such as depth-first or breadth-first linearization of the nodes in the octree. The linearization method may be a predefined method known to both the encoder 220 and the decoder 240 for encoding and reading the octree. Alternatively, the linearization method may be included as a parameter of an individual frame or a sequence of frames.

[0092] To further improve the efficiency of the octree, if the number of associated point cloud points at any level of the octree decreases or becomes zero, the octree may stop branching.

[0093] As a simple example, a three-dimensional scene may contain an 8x8x8 voxel space with points with the following (X,Y,Z) coordinates: (0,0,0), (0,1,1), (3,3,3), and (7,1,1).

[0094] These can be formed into an octree terminating in 2x2x2 sized voxels, where each branch node contains 8 bits indicating the sub-branch or sub-leaf node present in the XYZ sorted encoded octree, while each leaf node contains one or more 3D positions. The tree nodes are ordered breadth-first.

[0095] In this example, the ordering information is as follows, with the curly brackets representing the nodes of the octree and the parentheses representing the 3D position coordinates: {10001000},{1000001},{00001000},{2,(0,0,0),(0,1,1)},{1,(1,1,1)},{1,(1,1,1)}

[0096] Describing each node individually, the sequence is as follows:

[0097] 1.{10001000} The first node is the root node of level 0 of the octree, and the first and fifth subnodes, which correspond to the 4x4x4 voxels starting at (0,0,0) and (4,0,0), are at least point occupied. Thus, a decoder should understand that the encoded octree contains two nodes at level 1 of the octree.

[0098] 2.{10000001} The second node is at level 1 of the octree and corresponds to the first voxel indicated by the root node, i.e. the 4x4x4 voxel starting at (0,0,0). The second node indicates that its first subnode, and its eight subnodes, correspond to the 2x2x2 voxels starting at (0,0,0) and (2,2,2) and are occupied by at least one point. Thus, a decoder should understand that the encoded octree contains two nodes at level 2 of the octree that branch off from the second node.

[0099] 3.{00001000} The third node is at level 1 of the octree and corresponds to the second voxel pointed to by the root node, i.e., the 4x4x4 voxel starting at (4,0,0). The second node indicates that its fifth subnode corresponds to the 2x2x2 voxel starting at (6,0,0) and is occupied by at least one point. Thus, a decoder should understand that the encoded octree contains one node at level 2 of the octree that branches off from the third node.

[0100] 4.{2,(0,0,0),(0,1,1)} The fourth node is at level 2 of the octree and corresponds to the first voxel represented at level 1, i.e. the 2x2x2 voxel starting at (0,0,0) represented by the second node.

[0101] Because this octree ends at a voxel of size 2x2x2, the fourth node does not contain 8 bits to indicate a branch, but instead contains one or more positions of the individual points. Specifically, the fourth node indicates that the first point in the point order is at (0,0,0) and the second point in the point order is at (0,1,1).

[0102] In this embodiment, the leaf node also includes the number of points associated with it (in this case, two) so that a decoder reading the ordering information can identify the end of the bits associated with the fourth node. Alternatively, instead of including the number of points, a terminal symbol can be defined to indicate the end of the leaf node.

[0103] 5. {1,(1,1,1)} Similarly, the fifth node is at level 2 of the octree and corresponds to the second voxel represented at level 1, i.e., the 2x2x2 voxel starting at (2,2,2) represented by the second node.

[0104] The fifth node indicates that the third point in the point sequence is at an offset of (1,1,1) from the start point (2,2,2), i.e., the third point is at (3,3,3). Thus, in this example, we are using local coordinates within the volume represented by each leaf node of the octree. In other examples, the 3D position may be a global position within the entire volume of the 3D scene.

[0105] 6. {1,(1,1,1)} Again, the sixth node is at level 2 of the octree and corresponds to the third voxel represented at level 1, i.e., the 2x2x2 voxel starting at (6,0,0) represented by the third node.

[0106] The sixth node indicates that the fourth point in the point order is at an offset of (1,1,1) from the start point (6,0,0), i.e., the fourth point is at (7,1,1).

[0107] In the above example, representing the four points as an octree may not be efficient, but octrees become more useful the larger the number of points in a 3D scene, especially since the tree structure can be traversed in logarithmic time instead of linear time to match points between frames and also to identify where interframe predictive coding should be used.

[0108] Next, in step S340, the encoder 220 encodes each point using the encoding type identified in step S310, i.e., the encoder 220 encodes each point by performing inter-frame predictive coding if possible, and by performing intra-frame predictive coding if inter-frame predictive coding is not possible, where "possible" is defined as described above for step S310, and different requirements for when inter-frame predictive coding is "possible" may be set in different embodiments.

[0109] Step S340 may occur before or simultaneously with steps S320 and S330.

[0110] Referring again to FIG. 4, the various types of encoding that may be performed in step S340 can be seen.

[0111] Each of the encoded frames 10-1 and 10-2 includes two point attributes: color and light source direction.

[0112] The interframe predictive coded color data 420 of the first frame 10-1 indicates positive and negative hexadecimal values ​​that predict the difference between adjacent ordered point cloud points. For example, D9 in hexadecimal corresponds to 217 in traditional decimal notation. Although hexadecimal notation is commonly used to represent values ​​in groups of 4 bits (since 24=16), this application does not limit the manner in which the data may be represented. Referring to FIG. 1, points 20-B through 20-D will be darker in color and will have negative values ​​as shown in FIG. 4.

[0113] On the other hand, the intraframe coded color data 420 of the second frame 10-2 is composed mostly of zeros, indicating that this data can be stored efficiently using intraframe prediction.

[0114] In this simple embodiment, intraframe prediction is performed in particular based on the principle of differential coding, where each coded value is the difference between attribute values ​​of adjacent points in an ordered sequence of points defined by the ordering information 410.

[0115] In embodiments with more points, it may be useful to run more statistical methods.

[0116] In one embodiment, an average value of the attribute may be taken for multiple points and the encoded value may be defined as the difference between each point and the average, which may reduce the number of average bits required to represent the difference values ​​while incurring only the small overhead of a single average value.

[0117] This embodiment can be extended to local averages and local differences, as long as the overhead of the averages is not too large. For example, local points can be grouped together based on nodes at a certain level of the octree used to define the ordering of the points.

[0118] Finally, the color of non-moving objects remains constant despite changes in light source direction, so the interframe predictive coded color data 470 is also near zero.

[0119] Looking at the light source direction attributes 430, 480, frame 10-1 shows that the light source is in a single direction at all points. This direction changes from 0 to 90 at all points by the time of the second frame 10-2. This is a case where interframe predictive coding (as shown at 480) may be inefficient, since the same change is coded multiple times for different points. On the other hand, intraframe predictive coding (as shown at 430 for the second frame 10-2) for this example attribute is efficient, since the change in light source direction only needs to be recorded once for all points in the coded data.

[0120] As a generalization of the above, the encoded data may contain several different layers associated with different attributes. Each layer can be divided into inter-frame and intra-frame points that can be coded differently. However, it is not necessarily desirable to use as much inter-frame predictive coding as possible, since it does not necessarily improve compression or coding / decoding time.

[0121] After performing predictive encoding in step S340, the method may also optionally include more general data compression techniques. For example, in step S350, the encoder 220 may perform any of sparse array encoding, run-length compression, and / or entropy encoding.

[0122] Sparse array coding involves compressing a section of data that contains relatively little data. In sparse array coding, the bit sequence of the encoded data is divided into groups of N bits, where N is a pre-determined value. Each group that contains only zeros is removed. A list of the removals is then constructed (called a sparse array). The sparse array indicates which groups of N bits have been removed so that the groups of zeros can be reconstructed by the decoder 240.

[0123] In a specific example, the data undergoing sparse array encoding is x={4,0,3,0,0,0,0,0,0,0,1,3,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3}, and the value of N is 4. The data is divided into groups: x0={4,0,3,0}, x1={0,0,0,0}, x2={0,1,3,5}, x3={0,0,0,0}, x4={0,0,0,0}, x5={0,0,0,0}, x6={3}. After removing the groups of zeros, the encoded data is x'={4,0,3,0,0,1,3,5,3}. The sparse array is constructed as b={0,1,0,1,1,1,0}, where a value of 1 represents the group that contains only zeros and has been removed.

[0124] Sparse array encoding can also be applied at a higher level to multi-bit values ​​rather than individual bits. In such cases, the "zero" threshold can be raised to some small value greater than zero when encoding an attribute with acceptable compression loss. For example, if an attribute is known to contain a noise level, the threshold can be set such that there is no need to encode the noise component and small noise values ​​are treated equivalent to zero.

[0125] The omission of noise above is an example of quantization, where the entire range of raw data values ​​is mapped to one quantized value. Other types of quantization may be applied as part of the encoding. If quantization is used during encoding, then a decision must be made as to how to dequantize the data when it is decoded. In some instances, it may be appropriate to dequantize the quantized value so that it is the average of the range of raw data values ​​that may be mapped to the quantized value. In other cases, it may be appropriate to dequantize the quantized value so that it is the lower or upper end of the range of possible raw data values.

[0126] In addition to, or as an alternative to, sparse array encoding, the encoded data, or sparse array, may be run-length compressed, in which sequences of 1's or 0's are compressed and replaced with an indication of the original length of the compressed sequence, thereby further improving compression.

[0127] Additionally or alternatively, the encoded data may be entropy encoded. Entropy encoding may be applied to multiple parts of the compressed data, such as the encoding type information, the ordering information, the encoded attribute values, etc. Any type of entropy encoding may be used, including Huffman encoding, range encoding, or VC-6 type encoding.

[0128] FIG. 5 is a flow diagram that generally illustrates a method for encoding a frame of a sequence of frames of point cloud data.

[0129] In particular, FIG. 5 is effective in highlighting the principles of the present invention which distinguish it from the prior art identified at the time of filing.

[0130] More specifically, in step S510, obtain a current point to be encoded in the current frame. Referring back to the embodiment of Figure 1, the current point may be, for example, point 20-A in frame 10-2.

[0131] In step S520, a previous point encoded in a previous frame is identified, which corresponds to the current point to be encoded in the current frame. The previous point may be, for example, point 20-A in frame 10-1.

[0132] In step S530, at least one attribute of the current point is encoded by referring to the previous point using interframe predictive coding to generate encoded data of the current point, which can be, for example, the color difference between point 20-A of frame 10-2 and point 20-A of frame 10-1 (even if the difference is zero).

[0133] In step S540, a current encoded frame is generated, which includes the encoded data of the current point.

[0134] This technique uses interframe prediction for point cloud data to encode attributes other than position, a feature not found in either of the papers by GRAZIOSI and ROODAKI, both of which are mentioned in the Background of the Invention.

[0135] All of the techniques used by the encoder 220 are lossless (but not necessarily lossless), and thus, in essence, the decoder 240 is designed to perform the inverse operations of the techniques described above with respect to Figures 3 and 5. In particular, inter-frame predictive decoding and intra-frame predictive decoding essentially operate on the same principles as inter-frame predictive coding and intra-frame predictive coding.

[0136] 6 is a flow diagram illustrating generally significant features of a method for decoding a frame of a sequence of frames of point cloud data. The method is performed on a sequence of encoded frames of point cloud data for rendering a dynamic three-dimensional scene. Each frame includes a number of encoded points, each point including a three-dimensional position and one or more attributes. The method may be performed, for example, by the decoder 240 of a system such as that shown in FIG. 2.

[0137] Referring to FIG. 6, in step S610, a point of the frame to be decoded is obtained as the current point.

[0138] In step S620, a predecessor point decoded in the previous frame is identified, which predecessor point corresponds to the current point.

[0139] In step S630, at least one attribute of the current point is decoded by referring to the previous point using interframe predictive coding to generate decoded data of the current point.

[0140] This decoding method has in common the special technical feature mentioned above, namely, the use of inter-frame prediction to decode attributes of point cloud data other than their positions.

[0141] Figure 7 is a schematic block diagram of an encoder or decoder. As seen in Figure 7, the encoder and decoder may be implemented by common computer hardware including a processor 710 and a memory 720. The memory stores processor instructions, bytecode, uncompiled code, or any other form of computer instructions that can be converted into processor instructions, for instructing the processor to perform a method as described above with respect to Figure 5 or Figure 6. Alternatively, the encoder or decoder may be implemented entirely as hardware configured to perform the method of Figure 5 or Figure 6 without requiring a general purpose processor or memory.

[0142] The encoder or decoder preferably also comprises a communications interface 730 which, in the case of the encoder 220, allows for the reception of unencoded point cloud data and the transmission of encoded point cloud data, and, in the case of the decoder 240, allows for the reception of encoded point cloud data and the transmission of unencoded point cloud data.

Claims

1. 1. A method of encoding frames of a sequence of frames of point cloud data for rendering a dynamic three-dimensional scene, each frame including a plurality of points, each point including a three-dimensional position and one or more attributes, the method comprising: Obtaining a current point to be coded within a current frame; identifying a previous point encoded in a previous frame that corresponds to the current point to be encoded in the current frame; encoding at least one attribute of the current point by referencing the previous point using inter-frame predictive coding to generate encoded data for the current point; generating the current frame, the current frame including the encoded data for the current point.

2. 2. The method of claim 1, wherein inter-frame predictive coding of each of the at least one attribute of the current point comprises coding a difference between the attribute of the current point and the attribute of the predecessor point.

3. 2. The method of claim 1, wherein identifying the predecessor point that corresponds to the current point comprises identifying a predecessor point encoded in the previous frame that has the same three-dimensional position as the current point.

4. the one or more attributes of each point include non-predictive attributes; 4. The method of claim 3, wherein identifying the predecessor point that corresponds to the current point comprises identifying a predecessor point encoded in the previous frame that has the same three-dimensional position as the current point and the same value of the non-predictive attribute as the current point.

5. obtaining a first portion of the plurality of current points to be encoded within the current frame; for each current point of the first portion, identifying whether inter-frame predictive coding is enabled for the current point, wherein inter-frame predictive coding is enabled if there is a previous point coded in the previous frame that corresponds to the current point to be coded in the current frame; 2. The method of claim 1, further comprising: encoding the frame by performing inter-frame predictive coding for each current point for which inter-frame predictive coding is available, and performing intra-frame predictive coding for each current point for which inter-frame predictive coding is not available.

6. The method of claim 5 , wherein the current frame is generated including coding type information indicating a coding type for each point of the current frame, the type being inter-frame predictive coding or intra-frame predictive coding.

7. 2. The method of claim 1, further comprising generating ordering information defining an order of the second portions of the plurality of current points to be encoded within the current frame, wherein the current frame is generated including the ordering information and including encoded data ordered according to the ordering information.

8. The method of claim 7 , wherein the ordering information includes the three-dimensional position of each point in the current frame.

9. The method of claim 8 , wherein the ordering information comprises an octree based on the three-dimensional position of each point in the current frame.

10. The method of claim 9 , wherein each leaf node of the octree comprises at least one three-dimensional location of a point of the current frame.

11. 2. The method of claim 1, further comprising encoding a third portion of the plurality of current points of the current frame using intraframe predictive coding.

12. predictively encoding a third attribute of the one or more attributes for each of the fourth portions of the plurality of current points of the frame; performing sparse array compression on the encoded values ​​of the third attribute of the fourth portion of the plurality of current points, the sparse array compression comprising: dividing the encoded values ​​of the third attribute into groups of a predetermined size N; generating a sparse array that identifies whether each group contains only encoded values ​​below a predetermined threshold; generating the current frame including the sparse array and the group storing at least one encoded value above the predetermined threshold.

13. The method of claim 12 further comprising performing run-length compression on the sparse array.

14. The method of claim 1 , further comprising applying an entropy coding method to at least one of the current frame ordering information, coding type information, and coded value of the fourth attribute.

15. The method of claim 14, wherein the entropy coding method is Huffman coding, range coding, or a method conforming to the VC-6 standard.

16. 1. A method of decoding frames of a sequence of frames of point cloud data for rendering a dynamic three-dimensional scene, each frame including a plurality of points, each point including a three-dimensional position and one or more attributes, the method comprising: Obtaining a current point to be decoded within a current frame; identifying a previous point decoded in a previous frame that corresponds to the current point to be decoded in the current frame; and decoding at least one attribute of the current point by referencing the predecessor point using inter-frame prediction to generate decoded data for the current point.

17. 17. The method of claim 16, wherein inter-frame prediction of each of the at least one attribute of the current point comprises decoding a difference between the attribute of the current point and the attribute of the predecessor point.

18. 17. The method of claim 16, wherein identifying the predecessor point that corresponds to the current point comprises identifying a decoded predecessor point in the previous frame that has the same three-dimensional position as the current point.

19. 17. The method of claim 16, wherein the one or more attributes of each point include a non-predictive attribute, and wherein identifying the predecessor point that corresponds to the current point includes identifying a predecessor point decoded in the previous frame that has the same three-dimensional position as the current point and the same value of the non-predictive attribute as the current point.

20. obtaining a first portion of the plurality of current points to be decoded within the current frame; for each current point of the first portion, identifying whether inter-frame predictive coding was used for the current point, wherein inter-frame predictive coding can be used if there is a previous point coded in the previous frame that corresponds to the current point to be coded in the current frame; 17. The method of claim 16, comprising: decoding the frame by performing inter-frame predictive decoding for each current point for which inter-frame predictive decoding is available, and performing intra-frame predictive decoding for each current point for which inter-frame predictive decoding is not available.

21. The method of claim 16 , wherein the current frame includes coding type information indicating a coding type for each point of the current frame, the type being inter-frame predictive coding or intra-frame predictive coding.

22. 17. The method of claim 16, further comprising decoding ordering information defining an order in which second portions of the plurality of current points are encoded within the current frame.

23. 17. The method of claim 16, further comprising decoding a third portion of the plurality of current points of the current frame using intraframe predictive decoding.

24. 24. The method of claim 23, further comprising predictively decoding a first attribute of the one or more attributes, wherein intra-frame predictively decoding the first attribute comprises, for each of the plurality of current points, adding an average of the first attribute for the plurality of current points to a difference between the average and the value of the first attribute for the current point.

25. 17. The method of claim 16, wherein the plurality of current points is a local group of points corresponding to nodes at a given level of an octree.

26. 17. The method of claim 16, further comprising intra-frame predictive decoding a second attribute of the one or more attributes, wherein intra-frame predictive decoding the second attribute comprises obtaining a raw value of the second attribute for a first current point and obtaining a decoded value of the second attribute for a second current point, the decoded value being equal to the sum of the encoded value of the second attribute for the second current point and the raw value of the second attribute for the first current point.

27. 17. The method of claim 16, further comprising, for each fourth portion of the plurality of current points of the frame, inverting sparse array compression on the encoded values ​​of a third attribute, wherein inverting sparse array compression comprises reading the sparse array of encoded values ​​and identifying groups of encoded values ​​to which sparse array compression was applied during encoding, and for each group of encoded values ​​to which sparse array compression was applied, inserting a group of encoded zero values ​​of a predetermined size N at each location to which sparse array compression was applied.

28. The method of claim 16 further comprising inverting the run-length compression.

29. An encoder configured to perform the method of claim 1.

30. A decoder configured to perform the method of claim 16.

31. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method of claim 1.

32. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method of claim 16.

33. A non-transitory computer readable medium storing computer program instructions that, when executed by a computer, cause the computer to perform the method of claim 1.

34. 20. A non-transitory computer readable medium storing computer program instructions that, when executed by a computer, cause the computer to perform the method of claim 16.

35. A signal comprising computer program instructions that, when executed by a computer, cause the computer to perform the method of claim 1.

36. A signal comprising computer program instructions that, when executed by a computer, cause the computer to perform the method of claim 16.

37. 1. Frames of a sequence of frames of point cloud data for rendering a dynamic three-dimensional scene, each frame including a plurality of points, each point including a three-dimensional position and one or more attributes, the frames including encoded data for each point of the frames, the encoded data for at least one point being encoded using inter-frame predictive coding.

Images