Encoding and decoding point data identifying plurality of points in three-dimensional space

Abstract

There is provided a method (900) of encoding point data identifying a set of points in a three-dimensional (3D) space, the set being a set of 3D points. The 3D point set corresponds to a physical point set of the real-world environment. The method comprises: dividing (s902) a set of 3D points into a first subset of 3D points and a second subset of 3D points; encoding (s904) the first 3D point data identifying the first subset of 3D points using a first compression scheme, thereby generating first encoded point data; and encoding (s906) second 3D point data identifying the second subset of 3D points using a second compression scheme, thereby generating second encoded point data. The first compression scheme is different from the second compression scheme.

Description

Technical Field

[0001] Embodiments related to methods and apparatus for encoding and / or decoding point data identifying a plurality of points in a three-dimensional (3D) space (3D points) corresponding to a plurality of real-world points are disclosed. Background Art

[0002] Nowadays, 3D reconstruction of space is widely used in various fields. For example, for home decoration, one or more cameras capable of capturing a 360-degree view can be used to capture multiple shots of the kitchen to be decorated, and the captured multiple images can be used to reconstruct the kitchen in a 3D virtual space. The generated 3D reconstruction of the kitchen can be displayed on a screen and manipulated by the user to help the user visualize how to decorate the kitchen. In the 3D virtual space, there are multiple 3D points that identify objects or structures in the 3D virtual space. In the present disclosure, the multiple 3D points are also referred to as point clouds.

[0003] A point cloud is an unstructured collection of K points in 3D space. As mentioned above, these points are used to capture the scene geometry and scale of the real-world environment, that is, to represent the 3D structure. Point clouds can also store additional information about 3D points. This additional information is called attributes. Typical attributes are color information, reflectivity, normal vectors, etc. 3D points can be represented as .

[0004] Depending on the application and the type of scanning device used to capture the view of the real-world environment, the acquired 3D point cloud can have widely varying statistics. For example, dense 360° LiDAR point clouds can be obtained by using scanning devices such as the Leica BLK360, which are located on tripods at different locations on the floor. At each location, they rotate and perform a 360° scan of the physical environment. These point clouds are collected in order to create an accurate 3D map of the real-world environment (e.g., a "digital twin"), which can be used in a variety of industrial applications.

[0005] These point clouds are much denser than those generated by autonomous vehicles, and in near real-time, individual 360° scans do not have to be processed individually. A complete 3D map can be created offline by concatenating the individual 360° scans. Registered (stitched together) The point cloud can be represented as:

[0006]

[0007]

[0008] in, is the point cloud obtained at position n, Xkn is a set of X coordinates, Y coordinates of the 3D points included in the point cloud kn is a set of Y coordinates of the 3D points included in the point cloud, and Z kn K is a set of Z coordinates of the 3D points included in the point cloud. n is the total number of 3D points included in each point cloud, and K is the total number of 3D points included in a set of point clouds.

[0009] The point data of the point cloud is usually saved together in the E57 format. In addition, a set of scanning device poses corresponding to the point cloud Can be stored together with the point data of the point cloud: .

[0010] This allows easy access to each individual point cloud as well as the “fused” point cloud (the union of all individual scans) that defines a complete 3D map of the visual scene: . Summary of the invention

[0011] However, there are certain challenges. Typical sizes of dense LiDAR point clouds range from 1 GB to several GB. Therefore, storing such point clouds requires a lot of space in the storage medium, and transmitting such point clouds requires a lot of signal bandwidth. Therefore, there is a need to efficiently compress and decompress the point data identifying the multiple 3D points.

[0012] Therefore, in one aspect of some embodiments of the present disclosure, a method for encoding point data is provided, the point data identifying a three-dimensional (3D) point set corresponding to a physical point set of a real-world environment. The method includes: dividing the 3D point set into a first 3D point subset and a second 3D point subset; encoding first 3D point data identifying the first 3D point subset using a first compression scheme to generate first encoded point data; and encoding second 3D point data identifying the second 3D point subset using a second compression scheme to generate second encoded point data. The first compression scheme and the second compression scheme are different.

[0013] In a different aspect, a method is provided for decoding coded point data identifying a three-dimensional (3D) point set, the 3D point set corresponding to a physical point set of a real-world environment. The method includes obtaining coded point data including first coded point data and second coded point data, wherein the first coded point data identifies a first 3D point subset and the second coded point data identifies a second 3D point subset, and decoding the first coded point data using a first decompression scheme to generate first decoded point data. The method also includes decoding the second coded point data using a second decompression scheme to generate second decoded point data. The first compression scheme and the second compression scheme are different.

[0014] In a different aspect, there is provided a computer program comprising instructions (1144) which, when executed by a processing circuit, cause the processing circuit to perform the method of any of the above embodiments.

[0015] In various aspects, an apparatus for encoding point data is provided, the point data identifying a three-dimensional (3D) point set corresponding to a physical point set of a real-world environment. The apparatus is configured to: divide the 3D point set into a first 3D point subset and a second 3D point subset; encode first 3D point data identifying the first 3D point subset using a first compression scheme to generate first encoded point data; and encode second 3D point data identifying the second 3D point subset using a second compression scheme to generate second encoded point data. The first compression scheme and the second compression scheme are different.

[0016] In a different aspect, an apparatus is provided for decoding coded point data identifying a three-dimensional (3D) point set, the 3D point set corresponding to a physical point set of a real-world environment. The apparatus is configured to: obtain coded point data including first coded point data and second coded point data, wherein the first coded point data identifies a first 3D point subset and the second coded point data identifies a second 3D point subset; decode the first coded point data using a first decompression scheme to generate first decoded point data; and decode the second coded point data using a second decompression scheme to generate second decoded point data. The first compression scheme and the second compression scheme are different.

[0017] In different aspects, a device is provided. The device includes a memory; and a processing circuit coupled to the memory. The device is configured to perform the method of any of the above embodiments.

[0018] The embodiment of the present disclosure is structured as follows Given multiple scans of 3D point clouds in , the optimal compression scheme is selected to improve the compression efficiency.

[0019] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate various embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 An exemplary scenario for implementing an embodiment of the present disclosure is shown.

[0021] Figure 2A An exemplary apparatus according to some embodiments is shown.

[0022] Figure 2B Shows Figure 2A An exemplary view inside the device is shown.

[0023] Figure 3A An encoder according to some embodiments is shown.

[0024] Figure 3B A decoder according to some embodiments is shown.

[0025] Figure 4 A and Figure 4 B shows a compression scheme according to some embodiments.

[0026] Figure 5A and Figure 5B Compression schemes according to some embodiments are shown.

[0027] Figure 6 A process according to some embodiments is shown.

[0028] 7A to 7C Compression schemes according to some embodiments are shown.

[0029] FIG. 8A to FIG. 8C The overlap between adjacent point clouds is shown.

[0030] Fig. 9 A process according to some embodiments is shown.

[0031] Fig.10 A process according to some embodiments is shown.

[0032] Fig.11 An apparatus according to some embodiments is shown. DETAILED DESCRIPTION

[0033] Figure 1 An exemplary scene 100 for implementing an embodiment of the present disclosure is shown. In scene 100, a capture device 112 is used to capture a view of a kitchen 150 at each different location (e.g., 140, 142, and 144). In kitchen 150, an oven 152, a photo frame 154, and a refrigerator 156 are placed. Figure 1 As shown, the oven 152 is against the first wall 160 , the picture frame 154 is against the second wall 162 , and the refrigerator 156 is against the second wall 162 and the third wall 164 .

[0034] The capture device 112 includes a camera and a light detection and ranging (LiDAR) sensor. The camera is configured to capture a view of the kitchen 150. An example of a camera is a 360-degree camera, i.e., a camera capable of capturing a 360-degree view of a real-world environment.

[0035] The LiDAR sensor is configured to collect depth values ​​for various real-world points (e.g., points 171 to 178) of the kitchen 150. Here, the depth value of a particular real-world point indicates the distance between the viewpoint 158 ​​of the capture device 112 and the particular real-world point. For example, the depth value of the real-world point 173 indicates the distance 180 between the point 173 and the viewpoint 158. An example of the viewpoint 158 ​​is the center point of the camera.

[0036] Once the camera captures a view of the kitchen 150, and the LiDAR sensor measures depth values ​​of real-world points included in the view of the kitchen 150, the capture device 112 may transmit the captured / measured data to a computing device 190, which is connected to the capture device 112 (wirelessly or via a wired connection). Upon receiving the data, the computing device 190 may combine the data collected by the camera and the data collected by the LiDAR sensor to generate point data identifying a plurality of three-dimensional (3D) points.

[0037] In some embodiments, the point data identifying the 3D points may be used to reconstruct the real world environment captured by the capture device 112. For example, the point data identifying the 3D points may be used to reconstruct the real world environment captured by the capture device 112. Figure 2A An extended reality (XR) display 202 shown in FIG. 1 is used to generate an XR (including virtual reality, mixed reality, or augmented reality) scene. Figure 2B The illustrated view 200 is an example of a view seen by a user 204 via an XR display 202. The point data for each 3D point may include the 3D coordinates of the 3D point and / or a color / brightness value of the 3D point.

[0038] The point data identifying the plurality of 3D points generated by computing device 190 may be stored in a storage device (eg, included in computing device 190). However, as described above, the typical size of point data ranges from 1 GB to several GB, so storing the point data will require a large amount of storage space.

[0039] Furthermore, in some scenarios, point data of 3D points needs to be sent from one entity to another. For example, suppose a homeowner wants to renovate kitchen 150, but the desired kitchen designer is far away from the house. In this case, once the view of kitchen 150 is captured and computing device 190 generates point data identifying 3D points of kitchen 150, the point data needs to be sent from computing device 190 to XR display device 202 so that the kitchen designer can see the reconstructed 3D view of kitchen 150. However, due to the large size of the point data, sending the point data will consume a large amount of data bandwidth. Therefore, it is necessary to efficiently compress and decompress the point data identifying multiple 3D points.

[0040] Figure 3AAn encoder 302 is shown according to some embodiments, and Figure 3B A decoder 352 is shown according to some embodiments. The encoder 302 is configured to selectively apply a first compression scheme (also known as "compression scheme type A" or "CST A") and a second compression scheme (also known as "compression scheme type B" or "CST B") to point data corresponding to different groups of 3D points, thereby generating encoded point data.

[0041] More specifically, the encoder 302 is configured to encode the M point clouds in the set of N point clouds one by one by converting them into 2D range images (CST A), and to encode the remaining point clouds (NM point clouds) by first fusing them and then encoding them into a fused point cloud (CST B). In some embodiments, the M point clouds P1…P M The camera pose (i.e., the orientation of the camera used to capture the image) is needed by the decoder to reconstruct the range image Generate single scan point cloud .

[0042] The decoder 352 is configured to selectively apply the first decompression scheme and the second decompression scheme to the encoded point data corresponding to different groups of 3D points. Once the decoder 352 receives the bitstream from the encoder 302, the decoder 352 can be configured to directly reconstruct the point cloud compressed using CST B. For point data compressed using CSTA, the decoder 352 may be configured to first reconstruct the range image and pose from the bitstream, and then generate point clouds corresponding to the individual scans based on the reconstructed range image and pose. Finally, the decoder 352 may be configured to fuse all point clouds to generate a complete point cloud. In some embodiments, each bitstream received or obtained by decoder 352 may include a value indicating whether the code point data included in the bitstream was encoded using CST A or CST B. Thus, such a value indirectly indicates to decoder 352 whether to apply a decompression scheme according to CST A or according to CST B.

[0043] One of the reasons for selectively applying different compression schemes to different 3D point sets is as follows.

[0044] During the scanning process, there is usually significant overlap between the point clouds scanned at adjacent scanning positions. FIG. 8A to FIG. 8C As shown, due to occlusion, the overlap between adjacent point clouds can be significantly reduced. FIG. 8A to FIG. 8C A top view (floor plan style) of an area scanned from two locations 1 and 2 is shown. Fig. 8AThe scanner at position 1 and the location of the point cloud obtained from the scanner at position 1 are shown (dashed line). Figure 8B The scanner at position 2 and the location of the point cloud obtained from the scanner at position 2 are shown (dashed line). Figure 8C Shows Fig. 8A and Figure 8B The overlapping positions of the point clouds in .

[0045] As the overlap between adjacent point clouds decreases, the potential gain of encoding them as a joint 3D structure (relative to encoding them individually as 2D structures) decreases. In the extreme case, if the collection of entire point clouds If the point cloud consists of non-overlapping point clouds, encoding the point cloud as a 2D panoramic range image is the best option. On the other hand, if If the image consists of heavily overlapping point clouds, compressing them into a fused 3D structure is the best option. Selectively applying different compression schemes to different groups of 3D points allows searching for the best partitioning so that overlapping point clouds are encoded together, while distant (or behind corner scans) point clouds are encoded separately.

[0046] As described above, the capture device 112 can be configured to capture views of the kitchen 150 at N different locations (i.e., perform N scans). At each location, the capture device 112 and the computing device 190 can identify a plurality of 3D points and generate point data corresponding to the identified plurality of 3D points. In the present disclosure, the plurality of 3D points for each capture location (140, 142, or 144) is referred to as a "point cloud."

[0047] Thus, if views of the kitchen 150 are captured at N different locations, there will be N point clouds ( , … ). A point cloud ( ) corresponds to the capture position 140, while the other point cloud ( ) corresponds to capture position 142.

[0048] like Figure 3A As shown, the encoder 302 is configured to obtain N point clouds ( , … ) is divided into two groups: the first group ( , … ) and the second group ( , … The point clouds in the first group can be encoded one by one using CSTA, while the point clouds in the second group ( ) are fused and encoded using CST B.

[0049] like Figure 3B As shown, the decoder 352 is configured to receive the coded point data of the 3D points in the first group and the coded point data of the 3D points in the second group. After receiving the coded point data, the decoder 352 is configured to generate a decoded image in the reconstructed range image. and the reconstructed sensor pose Generate a collection of single scan point clouds with the help of These single scan point clouds are reconstructed Merge to generate a complete point cloud:

[0050]

[0051] 1. CST A

[0052] As described above, the LiDAR sensor included in the capture device 112 is configured to measure a depth value of a 3D point, which indicates a real-world distance between a real-world point corresponding to the 3D point and the position of the LiDAR sensor. Figure 1 and Figure 4 In A, the value of the distance 180 between the position of the LiDAR sensor (e.g., the viewpoint 158 ​​of the camera) and the real-world point 173 is the depth value of the 3D point corresponding to the real-world point 173. Figure 1 In the embodiment, the position of the LiDAR sensor is set to be the same as the viewpoint 158 ​​of the camera, but in other embodiments, the position of the LiDAR sensor can be at other positions.

[0053] Due to the nature of capture using a rotating LiDAR sensor, a single scan point cloud ( ) in spherical coordinates or cylindrical coordinates The 3D points in can be considered to be located on the surface. Therefore, in CST A, the depth values ​​of the 3D points are projected onto the 2D plane , thus generating Figure 4 B shows a panoramic range image (e.g., mapping a full 360° point cloud into a single panoramic image).

[0054] One way to generate the panoramic range image is to convert the image captured by the camera included in the capture device 112 into an equirectangular image in which the longitude and latitude of a 3D point are mapped to horizontal and vertical coordinates. By segmenting the resulting panoramic image with depth values ​​(also known as range values) into occupancy planes and range planes, they can be efficiently encoded.

[0055] 2.CST B

[0056] In CST B, instead of projecting the point cloud onto a 2D surface, octree encoding is used to compress the point data of the point cloud. More specifically, Figure 5A As shown, in CST B, the coordinates of each 3D point included in the point cloud are quantized into integer coordinates and placed in a The volume 502 (eg, a cube) can be divided into two blocks of size The 8 sub-cubes are 512.

[0057] If the sub-cube 512 contains at least one 3D point, the sub-cube 512 is split into cubes with a size of Then, if the smaller sub-cube 522 contains at least one 3D point, the smaller sub-cube 522 can be segmented into 8 micro-sub-cubes 532. The segmentation process can be repeated until a predetermined size (e.g., ). On the other hand, if the sub-cube does not contain any 3D points, the segmentation process for this sub-cube branch can be ended.

[0058] The above process generates a tree structure (octree) (such as Figure 5B ), where each node can be represented by 8 bits, and each bit indicates the occupancy status of a sub-cube. For example, 8 bits 00010000 can indicate that the fourth sub-cube 512 contains 3D point data, and 8 bits 00000011 can indicate that each of the seventh smaller sub-cube and the eighth smaller sub-cube 522 contains 3D points.

[0059] For lossy compression, the octree may be encoded into predetermined levels and the corresponding sequence of 8-bit words may be entropy encoded.

[0060] 3. Divide N point clouds into two groups

[0061] As described above, by using the capture device 112 and the computing device 190, a multi-scan point cloud may be obtained. Each pair of sensor poses and point cloud corresponds to the specific position at which the image used to generate the point cloud was captured. For example, the sensor pose and point cloud The pair can correspond to Figure 1 Position 140 shown.

[0062] According to some embodiments, these point clouds The point clouds of the first group contain 3D points whose point data will be encoded in independent mode using CSTA, while the point clouds of the second group contain 3D points whose point data will be encoded into one large fused point cloud using CSTB. Figure 6 The process 600 of dividing N point clouds into two groups is shown. The process 600 includes steps s602 to s610. Each point cloud included in In other words, you can (where n is an integer between 1 and N) Steps s602 to s610 are executed in a loop, where n=1:N. Process 600 may start from step s602.

[0063] Step s602 includes: A number (e.g., H) of random 3D points are selected from the . The number H (e.g., 1000) can be set depending on the complexity constraint. In some embodiments, Instead of setting the number H, the number of random 3D points to be selected is indicated as a percentage of the total number of 3D points included in .

[0064] Step s604 comprises: for each random 3D point (L 1， L2...L H ), calculate the ratio R of the number of 3D points obtained from the same scan as the selected random 3D point to the number of 3D points obtained from a scan different from the scan used to select the selected random 3D point. h .

[0065] For example, suppose the random 3D points selected in step s602 include Fig. 7A 3D point 702 is shown. In step s604, a space (eg, a cube) 740 is defined relative to the position of the 3D point 702. More specifically, Fig. 7A In FIG. 7 , a cube 740 centered at 3D point 702 is defined.

[0066] Cube 740 has dimensions of 3x3x3 voxels. Figure 7B An example of voxel 722 is shown in FIG. Fig. 7A The number of voxels (eg, 3) defining the size of the cube 740 is shown in FIG. 7 for illustration purposes only and does not limit the embodiments of the present disclosure in any way.

[0067] In step s604, the number of 3D points obtained from the same scan as the random 3D point 702 is determined. Fig. 7A , 3D points 702, 704 and 706 are from Figure 13D point 708 is obtained from a camera scan performed at position 140 as shown, while 3D point 708 is obtained from a camera scan performed at position 142 . Figure 7B A view 752 of cube 740 is shown, and Figure 7C A view 754 of cube 740 is shown.

[0068] Then calculate the ratio R h Here, the ratio R h is 1 (corresponding to 3D point 708) / 2 (corresponding to 3D points 704 and 706). Alternatively, the ratio R h is 2 (corresponding to 3D points 704 and 706) / 1 (corresponding to 3D point 708).

[0069] After calculating the ratio of each random 3D point selected in step s602, in step s606, the average of the calculated ratios is calculated. For example, the average ratio of the nth scan may be equal to , where Rn is the average ratio, R h is for the sample 3D point L h ratio.

[0070] Step s608 includes determining whether the average ratio is less than a threshold value (eg, 0.5). ), then in step s610, the point cloud A list of point clouds assigned to which CST A is to be performed.

[0071] Otherwise (i.e., if the average ratio is not less than the threshold), then in step s612, the point cloud A list of point clouds assigned to perform CST B on.

[0072] 4. Selectively The segmented point cloud performs CST A or CST B

[0073] After executing process 600, at encoder 302, point data of 3D points included in the first group are encoded using CST A, and point data of 3D points included in the second group are encoded using CST B. Similarly, at decoder 304, the encoded point data of 3D points included in the first group are decoded using a process opposite to that of CST A, and the encoded point data of 3D points included in the second group are decoded using a process opposite to that of CST B.

[0074] Fig. 9A process 900 for encoding point data identifying a three-dimensional (3D) point set is shown. The 3D point set corresponds to a physical point set of a real-world environment. Process 900 may begin at step s902. Step s902 includes dividing the 3D point set into a first 3D point subset and a second 3D point subset. Step s904 includes: encoding first 3D point data identifying the first 3D point subset using a first compression scheme, thereby generating first encoded point data. Step s906 includes: encoding second 3D point data identifying the second 3D point subset using a second compression scheme, thereby generating second encoded point data. The first compression scheme and the second compression scheme are different.

[0075] In some embodiments, encoding the first 3D point data using a first compression scheme includes: converting the first 3D point data into 2D point data corresponding to a range image containing a view of the real-world environment, wherein the range image includes a plurality of pixels and each pixel corresponds to a real-world point in the real-world environment; and encoding the 2D point data to generate first encoded point data.

[0076] In some embodiments, the 2D point data includes a plurality of depth values ​​mapped to the plurality of pixels, and the depth value mapped to a pixel indicates a distance between a reference point and a real-world point to which the pixel corresponds.

[0077] In some embodiments, the second 3D point subset includes 3D points, and encoding the second 3D point data using a second compression scheme includes: determining a position of the 3D point within a predefined volume; based on the determined position of the 3D point, generating a bit set indicating the position of the 3D point within the predefined volume; and encoding the bit set.

[0078] In some embodiments, multiple 3D blocks are defined within a predefined volume, multiple 3D sub-blocks are defined in each 3D block, and determining the position of the 3D point within the predefined volume includes: identifying the 3D block including the 3D point; and identifying the 3D sub-block containing the 3D point, the identified 3D sub-block being included in the identified 3D block.

[0079] In some embodiments, the set of bits includes a first subset of bits and a second subset of bits, the identified 3D block is mapped to the first subset of bits, and the identified 3D sub-block is mapped to the second subset of bits.

[0080] In some embodiments, the second compression scheme is an octree based encoding.

[0081] In some embodiments, the 3D point set includes a plurality of 3D point subsets, and the method further includes: evaluating a 3D point subset included in the 3D point set; and based on the evaluation, determining whether to encode the evaluated 3D point subset using a first compression technique or a second compression technique.

[0082] In some embodiments, evaluating a subset of 3D points includes: selecting one or more 3D points included in the subset of 3D points; identifying 3D points that are within a predefined distance of each of the one or more 3D points; and for each of the one or more 3D points, evaluating the subset of 3D points based on the identified 3D points.

[0083] In some embodiments, identifying 3D points that are within a predefined distance of each of the one or more 3D points includes: for each of the one or more 3D points, determining a first number of 3D points that are within a predefined distance of each of the one or more 3D points and obtained using one or more range images captured at a first position; and for each of the one or more 3D points, determining a second number of 3D points that are within a predefined distance of each of the one or more 3D points and obtained using one or more range images captured at one or more positions different from the first position.

[0084] In some embodiments, evaluating the subset of 3D points includes: determining, for each 3D point of the one or more 3D points, a ratio of the first number to the second number; and evaluating the subset of 3D points based on the ratio.

[0085] In some embodiments, evaluating the subset of 3D points based on the ratio includes evaluating the subset of 3D points based on an average of the ratio.

[0086] In some embodiments, evaluating the subset of 3D points includes comparing the average value to a threshold value, and whether to encode the evaluated subset of 3D points using the first compression technique or the second compression technique is determined based on the comparison.

[0087] Fig.10 A process 1000 is shown for decoding coded point data identifying a three-dimensional (3D) point set. The 3D point set corresponds to a physical point set of a real-world environment. Process 1000 may begin at step s1002. Step s1002 includes: obtaining coded point data including first coded point data and second coded point data, wherein the first coded point data identifies a first 3D point subset and the second coded point data identifies a second 3D point subset. Step s1004 includes: decoding the first coded point data using a first decompression scheme to generate first decoded point data. Step s1006 includes: decoding the second coded point data using a second decompression scheme to generate second decoded point data. The first compression scheme and the second compression scheme are different.

[0088] In some embodiments, the method includes: merging the first decoded point data and the second decoded point data to generate 3D point data, wherein the generated 3D point data indicates coordinates of the 3D point set in a 3D space.

[0089] In some embodiments, the method includes: receiving one or more bitstreams, the one or more bitstreams containing the coded point data including first coded point data and second coded point data, wherein the one or more bitstreams also include a first value associated with the first coded point data and a second value associated with the second coded point data, the first value indicating a first decompression scheme, and the second value indicating a second decompression scheme.

[0090] In some embodiments, decoding the first encoded point data using the first decompression scheme includes: converting 2D point data included in the first encoded point data into first 3D point data, wherein the 2D point data corresponds to a range image of a view including a real-world environment, and further, wherein the range image includes multiple pixels and each pixel corresponds to a real-world point in the real-world environment.

[0091] In some embodiments, the 2D point data includes a plurality of depth values ​​mapped to the plurality of pixels, and the depth value mapped to a pixel indicates a distance between a reference point and a real-world point to which the pixel corresponds.

[0092] In some embodiments, the second 3D point subset includes 3D points, and decoding the second encoded point data using a second decompression scheme includes: obtaining a set of bits indicating a position of the 3D point within a predefined volume; and determining coordinates of the 3D point based on the set of bits, wherein the coordinates of the 3D point are defined in a 3D space.

[0093] In some embodiments, multiple 3D blocks are defined within a predefined volume, multiple 3D sub-blocks are defined in each 3D block, the bit set identifies the 3D block including the 3D point and the 3D sub-block including the 3D point, and the identified 3D sub-block is included in the identified 3D block.

[0094] In some embodiments, the set of bits includes a first subset of bits and a second subset of bits, the identified 3D block is mapped to the first subset of bits, and the identified 3D sub-block is mapped to the second subset of bits.

[0095] In some embodiments, the second compression scheme is an octree based encoding.

[0096] Fig.11 A device (eg, a server, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) including an encoder 302 and / or a decoder 352 is shown. Fig.11As shown, the apparatus may include: a processing circuit (PC) 1102, which may include one or more processors (P) 1155 (e.g., one or more general-purpose microprocessors and / or one or more other processors, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc.); a communication circuit 1148, which is coupled to an antenna arrangement 1149, which includes one or more antennas and includes a transmitter (Tx) 1145 and a receiver (Rx) 1147 for enabling the apparatus to send data and receive data (e.g., wirelessly send / receive data); and a local storage unit (also known as a "data storage system") 1108, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In some embodiments, the apparatus may not include the antenna arrangement 1149, but may include a connection arrangement required to send and / or receive data using a wired connection. In embodiments where the PC 1102 includes a programmable processor, a computer program product (CPP) 1141 may be provided. The CPP 1141 includes a computer readable medium (CRM) 1142 storing a computer program (CP) 1143, which includes computer readable instructions (CRI) 1144. The CRM 1142 can be a non-transitory computer readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a storage device (e.g., a random access memory, a flash memory), etc. In some embodiments, the CRI 1144 of the computer program 1143 is configured so that: when executed by the PC 1102, the CRI causes the device to perform the steps described herein (e.g., the steps described herein with reference to the flowchart). In other embodiments, the device can be configured to perform the steps described herein without the need for code. That is, for example, the PC 1102 can be composed of only one or more ASICs. Therefore, the features of the embodiments described herein can be implemented in hardware and / or software.

Claims

1. A method (900) for encoding point data, the point data identifying a set of points in a three-dimensional 3D space, the set being a 3D point set, the 3D point set corresponding to a physical point set of a real-world environment, the method comprising: dividing (s902) the 3D point set into a first 3D point subset and a second 3D point subset; encoding first 3D point data identifying the first 3D point subset using a first compression scheme (s904), thereby generating first encoded point data; as well as The second 3D point data identifying the second 3D point subset is encoded using a second compression scheme (s906) to generate second encoded point data, wherein: The first compression scheme and the second compression scheme are different.

2. The method according to claim 1, wherein: Encoding the first 3D point data using the first compression scheme includes: converting the first 3D point data into 2D point data corresponding to a range image containing a view of the real-world environment, wherein the range image comprises a plurality of pixels and each pixel corresponds to a real-world point in the real-world environment; and The 2D point data is encoded to generate the first encoded point data.

3. The method according to claim 2, wherein: The 2D point data includes a plurality of depth values ​​mapped to the plurality of pixels, and The depth value mapped to a pixel indicates the distance between the reference point and the real-world point to which the pixel corresponds.

4. The method according to any one of claims 1 to 3, wherein: The second 3D point subset includes 3D points, and Encoding the second 3D point data using the second compression scheme includes: Determining the position of the 3D point within a predefined volume; Based on the determined position of the 3D point, generating a set of bits indicating the position of the 3D point within the predefined volume; and The set of bits is encoded.

5. The method according to claim 4, wherein: defining a plurality of 3D blocks within the predefined volume, Multiple 3D sub-blocks are defined in each 3D block. Determining the position of the 3D point within the predefined volume comprises: identifying a 3D block including the 3D point; and identifying a 3D sub-block including the 3D point, and The identified 3D sub-block is included in the identified 3D block.

6. The method according to claim 5, wherein: The bit set includes a first bit subset and a second bit subset, The identified 3D block is mapped to the first bit subset, and The identified 3D sub-block is mapped to the second subset of bits.

7. The method according to any one of claims 1 to 6, wherein: The second compression scheme is an octree-based encoding.

8. The method according to any one of claims 1 to 7, wherein: The 3D point set includes a plurality of 3D point subsets, and The method further comprises: evaluating a subset of 3D points included in the 3D point set; Based on the evaluation, it is determined whether to encode the evaluated subset of 3D points using the first compression technique or the second compression technique.

9. The method according to claim 8, wherein: Evaluating the 3D point subset includes: Select one or more 3D points included in the 3D point subset; identifying 3D points that are within a predefined distance of each of the one or more 3D points; and For each 3D point of the one or more 3D points, the subset of 3D points is evaluated based on the identified 3D point.

10. The method according to claim 9, wherein: Identifying a 3D point within the predefined distance of each of the one or more 3D points comprises: determining, for each of the one or more 3D points, a first number of 3D points located within a predefined distance of each of the one or more 3D points and obtained using one or more range images captured at a first location; and For each of the one or more 3D points, a second number of 3D points located within a predefined distance of each of the one or more 3D points and obtained using one or more range images captured at one or more locations different from the first location is determined.

11. The method according to claim 10, wherein: Evaluating the 3D point subset includes: For each 3D point of the one or more 3D points, determining a ratio of the first number to the second number; and The subset of 3D points is evaluated based on the ratio.

12. The method according to claim 11, wherein: Evaluating the subset of 3D points based on the ratios includes evaluating the subset of 3D points based on an average of the ratios.

13. The method according to claim 12, wherein: Evaluating the 3D point subset comprises: comparing the average value with a threshold, and Whether to encode the evaluated subset of 3D points using the first compression technique or the second compression technique is determined based on the comparison.

14. A method (1000) for decoding coded point data, the coded point data identifying a set of points in a three-dimensional 3D space, the set being a 3D point set, the 3D point set corresponding to a physical point set of a real-world environment, the method comprising: obtaining (s1002) said coded point data comprising first coded point data and second coded point data, wherein said first coded point data identifies a first subset of 3D points and said second coded point data identifies a second subset of 3D points; Decoding the first coded point data using a first decompression scheme (s1004) to generate first decoded point data; and The second coded point data is decoded using a second decompression scheme (s1006) to generate second decoded point data, wherein: The first compression scheme and the second compression scheme are different.

15. The method according to claim 14, comprising: The first decoded point data and the second decoded point data are combined to generate 3D point data, wherein: The generated 3D point data indicates the coordinates of the 3D point set in the 3D space.

16. The method according to claim 14 or 15, comprising: receiving one or more bitstreams, the one or more bitstreams containing the code point data including the first code point data and the second code point data, wherein: The one or more bitstreams further include a first value associated with the first coded point data and a second value associated with the second coded point data, The first value indicates the first decompression scheme, and The second value indicates the second decompression scheme.

17. The method according to any one of claims 14 to 16, wherein: Decoding the first code point data using the first decompression scheme includes: Converting 2D point data included in the first encoded point data into first 3D point data, wherein the 2D point data corresponds to a range image of a view including the real-world environment, and further, wherein the range image includes a plurality of pixels, and each pixel corresponds to a real-world point in the real-world environment.

18. The method according to claim 17, wherein: The 2D point data includes a plurality of depth values ​​mapped to the plurality of pixels, and The depth value mapped to a pixel indicates the distance between the reference point and the real-world point to which the pixel corresponds.

19. The method according to any one of claims 14 to 18, wherein: The second 3D point subset includes 3D points, and Decoding the second code point data using the second decompression scheme includes: obtaining a set of bits indicating a position of the 3D point within a predefined volume; and Coordinates of the 3D point are determined based on the bit set, wherein the coordinates of the 3D point are defined in a 3D space.

20. The method according to claim 19, wherein: defining a plurality of 3D blocks within the predefined volume, Multiple 3D sub-blocks are defined in each 3D block. The bit set identifies a 3D block including the 3D point and a 3D sub-block including the 3D point, and The identified 3D sub-block is included in the identified 3D block.

21. The method according to claim 20, wherein: The bit set includes a first bit subset and a second bit subset, The identified 3D block is mapped to the first subset of bits, The identified 3D sub-block is mapped to the second subset of bits.

22. The method according to any one of claims 14 to 21, wherein: The second compression scheme is an octree-based encoding.

23. A computer program (1143) comprising instructions (1144) which, when executed by a processing circuit (1102), cause the processing circuit to perform the method according to any one of claims 1 to 22.

24. A carrier comprising a computer program according to claim 23, wherein: The carrier is one of an electric signal, an optical signal, a radio signal and a computer-readable storage medium.

25. An apparatus (1100) for encoding point data, the point data identifying a set of points in a three-dimensional 3D space, the set being a 3D point set, the 3D point set corresponding to a physical point set of a real-world environment, the apparatus being configured to: dividing (s902) the 3D point set into a first 3D point subset and a second 3D point subset; encoding first 3D point data identifying said first subset of 3D points using a first compression scheme (s904), thereby generating first encoded point data; and The second 3D point data identifying the second 3D point subset is encoded using a second compression scheme (s906) to generate second encoded point data, wherein: The first compression scheme and the second compression scheme are different.

26. The device according to claim 25, wherein The apparatus is further configured to perform a method according to any one of claims 2 to 13.

27. An apparatus (1100) for decoding encoded point data, the point data identifying a set of points in a three-dimensional 3D space, the set being a 3D point set, the 3D point set corresponding to a physical point set of a real-world environment, the apparatus being configured to: Obtaining (s1002) the code point data including first code point data and second code point data, wherein: The first encoded point data identifies a first subset of 3D points, and the second encoded point data identifies a second subset of 3D points; Decoding the first coded point data using a first decompression scheme (s1004) to generate first decoded point data; as well as The second coded point data is decoded using a second decompression scheme (s1006) to generate second decoded point data, wherein: The first compression scheme and the second compression scheme are different.

28. The device according to claim 27, wherein The apparatus is further configured to perform a method according to any one of claims 15 to 22.

29. An apparatus (1100), comprising: memory (1141); as well as A processing circuit (1102) coupled to the memory, wherein the apparatus is configured to perform the method according to any one of claims 1 to 22.

Images