Method and device for point cloud coding and decoding, medium and method for processing 3D point cloud data

Abstract

A code stream including encoded information of a three-dimensional (3D) point cloud is received. The 3D point cloud includes at least two points in a 3D space. And analyzing the code stream into a first sub-code stream and a second sub-code stream. An occupancy graph is determined based on the first sub-code stream, and a sparse range image (SRI) is determined based on the second sub-code stream. The SRI is obtained through conversion from the 3D point cloud in the 3D space and comprises at least two samples. The occupancy map indicates whether at least one sample of the SRI has a corresponding point in the 3D point cloud. And reconstructing the 3D point cloud based on the SRI and the occupancy map.

Description

By citation and inclusion in this article

[0001] This application claims U.S. Provisional Application No. 63 / 680,552, filed August 7, 2024, entitled "LiDAR Point Cloud Compression Based on Sparse Range Images"; U.S. Provisional Application No. 63 / 695,749, filed September 17, 2024, entitled "LiDAR Point Cloud Compression Based on At Least Two Sparse Range Images"; and U.S. Provisional Application No. 63 / 708,123, filed October 16, 2024, entitled "LiDAR Point Cloud Compression Using Unprojected Point Lists". Priority is claimed in U.S. Provisional Application No. 63 / 708,665, filed October 17, 2024, entitled "LiDAR Point Cloud Compression Using Vertical Band Quantization"; U.S. Provisional Application No. 63 / 710,540, filed October 22, 2024, entitled "LiDAR Point Cloud Compression Using Projection Based on Auxiliary Information"; and U.S. Patent Application No. 19 / 268,899, filed July 14, 2025, entitled "LiDAR Point Cloud Compression Based on Sparse Range Images". The entire disclosure of the earlier applications is incorporated herein by reference. Technical Field

[0002] This application relates to point cloud compression technology, and more particularly to methods and apparatus for point cloud encoding and / or decoding, methods for processing 3D point cloud data, methods for storing or transmitting video streams, and non-volatile computer-readable storage media. Background Technology

[0003] The background description provided herein is intended to present the general context of this disclosure. The extent of the work of the currently attributed inventors described in the background section and in various aspects of this specification does not indicate that it was prior art at the time of this disclosure filing, nor is it expressly or implied that it was acknowledged as prior art to this disclosure.

[0004] MPEG has released point cloud compression standards, including the V-PCC standard, primarily applicable to dense point clouds, and the G-PCC standard (ISO-IEC 23090-9), designed for sparse point clouds. Since point clouds generated by LiDAR are often sparse, V-PCC is considered unsuitable for compressing LiDAR-generated point clouds. G-PCC was originally designed to compress sparse point clouds, including those generated by LiDAR. However, G-PCC (e.g., the octree-based method in its second version) may not provide sufficient compression efficiency for the computational demands of some LiDAR-based applications. Moreover, existing technologies cannot achieve a reasonable balance between compression efficiency and computational complexity. Therefore, improved point cloud coding techniques are needed, which can benefit from artificial intelligence-based approaches. Summary of the Invention

[0005] Embodiments of this disclosure include bitstreams, methods, and apparatus for point cloud encoding / decoding. In some embodiments, the point cloud encoding / decoding apparatus includes processing circuitry.

[0006] According to embodiments of this disclosure, a method for point cloud decoding is provided. In this method, a bitstream comprising encoded information of a three-dimensional (3D) point cloud is received. The 3D point cloud comprises at least two points in 3D space. The bitstream is parsed into a first sub-bitstream and a second sub-bitstream. An occupancy map is determined based on the first sub-bitstream, and a sparse distance image (SRI) is determined based on the second sub-bitstream. The SRI is derived from the 3D point cloud in the 3D space and comprises at least two samples, the occupancy map indicating whether at least one sample of the SRI has a corresponding point in the 3D point cloud. The 3D point cloud is reconstructed based on the SRI and the occupancy map.

[0007] According to another embodiment of this disclosure, a method for point cloud encoding is provided. In this method, a 3D point cloud comprising at least two points in 3D space is converted into a SRI in 2D space, the SRI comprising at least two samples. An occupancy map is determined based on the SRI. The occupancy map indicates whether at least one sample of the SRI has a corresponding point in the 3D point cloud. The SRI is packaged based on the occupancy information of the occupancy map to obtain a packaged SRI. The packaged SRI does not include at least one unoccupied sample of the SRI that does not have a corresponding point in the 3D point cloud. At least one unoccupied sample of the packaged SRI is filled based on adjacent occupied samples of the packaged SRI to obtain a filled SRI. The occupancy map is encoded into a first sub-bitstream of the bitstream, and the filled SRI is encoded into a second sub-bitstream of the bitstream.

[0008] According to another embodiment of this disclosure, a method for processing three-dimensional 3D point cloud data is provided. The method includes: processing a bitstream comprising encoded information of a 3D point cloud according to a format rule specifying that the 3D point cloud comprises at least two points in a 3D space; the format rule specifying that the bitstream is parsed into a first sub-bitstream and a second sub-bitstream; determining an occupancy map based on the first sub-bitstream and determining a sparse distance image (SRI) based on the second sub-bitstream, the SRI being derived from the 3D point cloud in the 3D space and comprising at least two samples, the occupancy map indicating whether at least one sample of the SRI has a corresponding point in the 3D point cloud; and reconstructing the 3D point cloud based on the SRI and the occupancy map.

[0009] According to another embodiment of this disclosure, a method for storing or transmitting a video stream is provided, the video stream being generated according to the point cloud encoding method or decoded according to the point cloud decoding method.

[0010] According to another embodiment of this disclosure, a non-volatile computer-readable storage medium is provided for storing a video stream, the video stream being generated according to the point cloud encoding method or decoded according to the point cloud decoding method.

[0011] According to another embodiment of this disclosure, a non-volatile computer-readable storage medium is provided for storing a bitstream encoded by an encoding method. In this encoding method, a 3D point cloud comprising at least two points in 3D space is converted into a Sample Injection Registry (SRI) in 2D space, the SRI comprising at least two samples. An occupancy map is determined based on the SRI. The occupancy map indicates whether at least one sample of the SRI has a corresponding point in the 3D point cloud. The SRI is packaged based on the occupancy information of the occupancy map to obtain a packaged SRI. The packaged SRI does not include at least one unoccupied sample of the SRI that does not have a corresponding point in the 3D point cloud. At least one unoccupied sample of the packaged SRI is padded based on adjacent occupied samples of the packaged SRI to obtain a padded SRI. The occupancy map is encoded into a first sub-bitstream of the bitstream, and the padded SRI is encoded into a second sub-bitstream of the bitstream.

[0012] Embodiments of this disclosure also provide an apparatus for point cloud decoding and / or encoding. The apparatus includes processing circuitry configured to implement any of the methods for point cloud decoding and / or encoding.

[0013] The technical solution disclosed herein includes a method and apparatus for compressing sparse point clouds. In some embodiments, the sparse point cloud is compressed based on a 2D representation of the generated point cloud and the application of at least one cascaded module. Based on the disclosed method, a reasonable balance is achieved between compression efficiency and computational complexity. In an example, a 3D point cloud including at least two points in 3D space is converted into a SRI in 2D space. An occupancy map is determined based on the SRI. This occupancy map indicates whether at least one sample in the SRI has a corresponding point in the 3D point cloud. The SRI is packaged based on the occupancy information of the occupancy map to obtain a packaged SRI. The packaged SRI does not include at least one unoccupied sample of the SRI that does not have a corresponding point in the 3D point cloud. At least one unoccupied sample of the packaged SRI is padded based on adjacent occupied samples of the packaged SRI to obtain a padded SRI. The occupancy map is encoded into a first sub-bitstream of the bitstream, and the padded SRI is encoded into a second sub-bitstream of the bitstream. Therefore, based on the 2D representation of the generated point cloud and the application of at least one cascaded module, the sparse point cloud is compressed, and a balance can be achieved between compression efficiency and computational complexity. Attached Figure Description

[0014] Other features, properties, and various advantages of the disclosed subject matter will become further apparent from the following detailed description and accompanying drawings, wherein:

[0015] Figure 1 This is an example of a sparse 3D point cloud rendered into a two-dimensional (2D) space.

[0016] Figure 2 This is a schematic diagram illustrating examples of encoders and decoders according to some embodiments of this disclosure.

[0017] Figure 3 This is a schematic diagram illustrating an example of converting a 3D point cloud into a 2D image.

[0018] Figure 4 This is a schematic diagram illustrating an example of point selection based on the rho(r) value in a 3D point cloud.

[0019] Figure 5 This is a schematic diagram illustrating an example of classifying 3D point clouds using the θ angle.

[0020] Figure 6 This is a schematic diagram of an example encoder according to some embodiments of the present disclosure.

[0021] Figure 7 This is a schematic diagram of an example decoder according to some embodiments of the present disclosure.

[0022] Figure 8 This is a schematic diagram of an example of a flowchart for generating a list of unprojected points associated with a 3D point cloud.

[0023] Figure 9 This is a schematic diagram of an example of a 3D point cloud rendered in a 2D polar coordinate system.

[0024] Figure 10 This is a schematic diagram illustrating an example of the division of a 3D point cloud rendered in a 2D polar coordinate system.

[0025] Figure 11 This is a schematic diagram illustrating an example of error correction in the 2D image representation of a 3D point cloud.

[0026] Figure 12 A flowchart outlining the decoding process according to some embodiments of this disclosure is shown.

[0027] Figure 13 A flowchart outlining the coding process according to some embodiments of this disclosure is shown.

[0028] Figure 14 This is a schematic diagram of a computer system according to an embodiment. Detailed Implementation

[0029] Embodiments of this disclosure include techniques for encoding and decoding three-dimensional (3D) point clouds (e.g., LiDAR point clouds). The encoding process may involve generating a two-dimensional (2D) representation of the point cloud, and the application of at least one cascaded module capable of generating a bitstream. The decoding process may reverse the steps of the encoding process to convert the decoded 2D representation of the point cloud back into a 3D point cloud.

[0030] Compressing sparse point clouds (such as those generated by LiDAR devices) presents challenges. For example, existing techniques may not be able to strike a reasonable balance between compression efficiency and computational complexity.

[0031] Figure 1 An example of a point cloud obtained using LiDAR is provided. Figure 1 As shown, the 3D point cloud generated by the LiDAR device is rendered into a 2D image (100).

[0032] Figure 2 An example of the process (200) of encoding and decoding a 3D point cloud (201) based on an encoder (200A) and a decoder (200B) is shown. Embodiments of this disclosure include compressing an uncompressed, point cloud-represented signal (201) obtained by LiDAR into a bitstream (211), and generating a decompressed point cloud (218) through an inverse decoding process. Figure 1 An example of a point cloud dataset obtained by LiDAR is shown.

[0033] like Figure 2As shown, the decoding process can be bit-exact. For example, regardless of the implementation details of the decoder, decoding a given bitstream (e.g., bitstream (211)) can produce the same point cloud (218). The encoding process can be specified in a non-bit-exact manner. The point cloud (201) obtained by LiDAR can be compressed into a bitstream (211) by the conversion component of the encoder (200A) by converting (203) the 3D point cloud (201) into a 2D representation (e.g., a sparsely populated 2D image). An occupancy map (204) can be generated by the generation component of the encoder (200A). The occupancy map can be encoded using an encoding mode such as lossless encoding (208). The occupancy map is available for use by the bitstream merger / generator (209). The occupancy map can be used to pack (205) sparsely populated 2D images into smaller 2D images with denser filling (e.g., densely populated 2D images). In the example, the sparsely padded 2D image is packaged by the packing component of the encoder (200A) to remove certain samples that do not have corresponding points in the 3D point cloud. The densely padded 2D image may still contain samples with values ​​not defined by the transformation / packing. To improve the efficiency of subsequent image encoding (207), the densely padded 2D image can be padded (206) with sufficient sample values. In the example, the filling component of the encoder (200A) can padded the densely padded 2D image with interpolated sample values. The image encoding (207) performed by the image encoder (or the encoding component of the encoder (200A)) can use conventional or novel (e.g., AI-based) encoding techniques. The output of the image encoder is available to a merger / generator (209), which creates a bitstream (211) that may include extracted metadata (202).

[0034] Decoding can employ the opposite technique. The bitstream (211) can be parsed and entropy-decoded (212) into symbols or sub-bitstreams by a decoding component, such as that of a decoder (200B). The symbols or sub-bitstreams can involve, for example, a sparse coded image, an occupancy map, and metadata. The sparse coded image can be decoded (214) by a decoding component, such as that of a decoder (200B), to obtain a padded image. The padded image can be depadded (215) by a depadded component, such as that of a decoder (200B), to obtain an unpadded image. The unpadded image can mark sample values ​​not indicated by the occupancy map as unused. The unpadded image can also be unpacked (216) by an unpacking component, such as that of a decoder (200B), to create a sparse coded image. The depadded (215) and unpacking (216) steps can utilize information from the occupancy map available after decoding (213). A sparsely filled 2D image can be converted into a 3D representation (217) by a conversion component of a decoder (200B), which can produce a reconstructed point cloud (218).

[0035] The basic process of converting a 3D point cloud (or 3D point cloud sample values) to a 2D image (referred to herein as a sparse distance image or SRI) (203) can employ spherical coordinate mapping. Points in the 3D point cloud can be converted from Cartesian coordinates (x, y, z) to spherical coordinates (r, θ, φ), where rho(r) is the distance from the LiDAR sensor to that point, theta(θ) is the azimuth angle, and phi(φ) is the elevation angle. The spherical coordinates can be mapped onto the 2D image plane. For example, the horizontal axis of the 2D image can represent the azimuth angle θ, and the vertical axis can represent the elevation angle φ. Each pixel (or sample) in the 2D image can be assigned a value corresponding to the distance r of the nearest point (to the sensor) mapped to that pixel. If at least two points project to the same pixel, the point with the smallest distance can be selected, thus capturing the nearest surface of the 3D point cloud. The resolution of the 2D image can be determined by the angular resolution of the LiDAR sensor. Scaling factors (e.g., horizontal and / or vertical scaling) can be used to increase or decrease resolution. The field of view of a LiDAR sensor defines the range of angles covered in a 2D image. Increasing the horizontal and / or vertical scaling factors can correspondingly increase the image size, the accuracy of the reconstructed angle values ​​(θ, φ) during decoding, and the accuracy of the decoded 3D points.

[0036] Figure 3 This is an example of the transformation of a point (300) in a 3D point cloud from Cartesian coordinates (x,y,z)(301) to spherical coordinates (r,θ,φ)(302). Figure 3 As shown, rho(r) is the distance from the origin (305) to point (300), theta(θ) is the azimuth angle, and phi(φ) is the elevation angle. Spherical coordinates can be mapped onto the 2D image plane (303). For example... Figure 3 As shown, the horizontal axis of the 2D image representing the azimuth angle θ can be quantized to a horizontal index i, and the vertical axis of the 2D image (303) representing the elevation angle φ can be quantized to a vertical index j. Each pixel (or sample) in the 2D image (303) is assigned a value corresponding to the distance r mapped to the nearest point to that pixel. Figure 3 In the example, the distance r of point (300) is mapped to the pixel (or pixel position) corresponding to the azimuth angle θ and elevation angle φ of point (300).

[0037] Still referencing Figure 2Creating an occupancy map (204) may be advantageous because the 2D images (or at least two 2D images) produced by the previously described transformation (203) may be inherently sparse. Higher resolution may be required due to scaling factors, which may exceed the resolution capabilities of the associated 2D image codec. The occupancy map may have the same size as the SRI and may use a binary representation to indicate whether a point (e.g., a point in a 3D point cloud) has been projected to a given location (e.g., a pixel location in the SRI). Each point in the occupancy map (e.g., a point in a 3D point cloud) may represent (or indicate) whether the corresponding point (or sample or pixel) is present (1) or absent (0) in the sparse distance image. If the SRI does not present any interference from subsequent processes (e.g., packing (205) and / or padding (206)), the occupancy map may be redundant because it may not add important information. In this case, the occupancy map can be omitted from the bitstream (211) and replaced with an indication that does not include the occupancy map. As indicated in the bitstream, a missing occupancy map can represent all points (or samples or pixels) in the SRI that are correlated and is equivalent to an encoded occupancy map with all values ​​set to 1.

[0038] Packing (205) the SRI using the created occupancy map (204) may involve reorganizing the data in the SRI to utilize storage more efficiently by focusing on occupancy points. The occupancy map can be used to determine which samples in a 2D image have corresponding points. Packed images can be created to include more values ​​from occupancy points (or occupancy samples or occupancy pixels) and fewer values ​​from unoccupancy points. Various packing algorithms can be applied. For example, occupied pixels can be moved at the beginning of each row or column, or moved in raster order or z-scan order, etc. In embodiments of this disclosure, packing may not be necessary or advantageous. For example, if the SRI is already dense, the packing operation can be as simple as using the unmodified SRI (e.g., the original SRI) as the packed SRI.

[0039] Packing (205) can reduce the size of the packed SRI relative to the original SRI by excluding empty, unoccupied points.

[0040] To ensure that the 2D image is unpacked at the decoder without compromising the functionality of the entire system, lossless encoding of the occupancy map can be advantageously performed (208). In the example, the occupancy map can be encoded using lossless profiles of HEVC or VVC. In the example, the occupancy map can be represented by a one-dimensional signal and encoded using zip encoding, run-length encoding, context-adaptive binary arithmetic encoding (CABAC), etc.

[0041] To improve the efficiency of the image encoder, unoccupied samples of the packed SRI can be padded (206) using sample data sufficient to design the image coding mechanism used. In the example, the sample data to be padded into the unoccupied samples can be determined based on the values ​​of neighboring samples. For example, the unoccupied samples can be padded by copying the values ​​of the nearest occupied samples in scan order. In the example, the unoccupied samples can be padded using more complex padded algorithms, such as seed padded, row and column padded, push / pull, and inverse intra-prediction based on intra-prediction techniques employed by the image codec. Padded (206) can be performed independently regardless of whether the SRI has been packed.

[0042] Compression of the packaged SRI (207) may involve known compression techniques, including zip, HEVC, VVC or neural coding methods.

[0043] The bitstream (211) format may include at least one of the following: (i) an encoded (packed and padded) SRI image bitstream; (ii) a bitstream representing an encoded occupancy map; and (iii) metadata encoded in an appropriate format (e.g., in the form of at least one Supplemental Enhancement Information (SEI) message, a possibly compressed XML document, etc.). The metadata may be extracted (202) from the source point cloud (201) and may be used to manipulate (or control) certain decoding steps, such as 2D-to-3D conversion (217). The metadata may be extracted as output by the decoder (200B) in an appropriate format to control rendering (not depicted). The metadata may be used for certain purposes. In the example, the 2D image resolution depends on the angle of the LiDAR coverage. In the example, an elevation angle is required to set the vertical resolution of the 2D image. In the example, the minimum and maximum elevation angles at which points can be found can be signaled in the metadata.

[0044] The decoder (200B) can perform steps characterized as the reverse of the encoder steps described above. The first step of the decoder (200B) is, for example, to parse (212) the bitstream (211) by the demultiplexer of the decoder (200B) to extract the packed / padded SRI and the sub-bitstream of the occupancy map, as well as metadata, respectively. The sub-bitstream (or second sub-bitstream) of the packed / padded SRI can be decoded (214) by the first decoding component of the decoder (200B) using an appropriate decoding technique that responds to the bitstream format created by the encoder (207). Similarly, the sub-bitstream (or first sub-bitstream) with the encoded occupancy map can be decoded (213) by the second decoding component of the decoder (200B) using an appropriate decoding technique. The outputs of decoding (214) and decoding (213) can be used by de-padded (215) and unpacking (216) mechanisms (or steps) that operate on the decoded SRI and the decoded / unpadded SRI, respectively. For example, the second sub-stream is decoded to obtain a processed image, which is generated by packing (205) and padding (206) the SRI. The processed image can be de-paddled by the de-padding component of the decoder (200B) to obtain an unpadded image based on the occupancy information of the occupancy map. The unpadded image can be unpacked by the unpacking component of the decoder (200B) to obtain the SRI based on the occupancy information of the occupancy map. The obtained SRI can be converted (217) back to the point cloud, which can utilize the metadata transmitted in the bitstream (211).

[0045] The embodiments of this disclosure include various implementations, depending on the capture system recording the 3D point cloud (201), the resolution of the SRI and occupancy map, the different processing that may interfere with the data, the capabilities of the encoding and decoding machines, and / or the user's expectations for quality and bandwidth used.

[0046] In the example, for low-power systems and low-quality, low-bitrate scenarios, a method that does not require large-scale image factors can be used. In this case, both the occupancy map and SRI can remain relatively small and can be directly encoded without a packing step.

[0047] In the example, for high-quality scenes, larger horizontal and vertical scaling factors can be used to ensure higher precision of the (θ,φ) coordinates. In this case, the occupancy map may be larger and sparser. To maintain a sparse distance image that is easy to encode, a packing process can be performed. Lossless configuration of the encoder (e.g., a video encoder) can be used to maintain high quality of the (r) value.

[0048] Embodiments of this disclosure include techniques for encoding and decoding 3D point clouds (e.g., LiDAR point clouds). The encoding process may involve generating a 2D representation of the 3D point cloud and the application of at least one cascaded module capable of generating a bitstream. The decoding process may reverse these steps to convert the decoded 2D representation back into a 3D point cloud. Among other data, the point cloud may be represented by an appropriately encoded occupancy map and at least two encoded 2D images representing packed and filled point cloud sample values, wherein the sparsely filled 3D images can be subdivided (or classified) into at least two 3D images based on categories (e.g., distances of point cloud samples to predefined points in 3D space).

[0049] Compressing sparse point clouds (such as those from LiDAR devices) presents challenges because existing techniques may fail to strike a reasonable balance between compression efficiency and computational complexity. The sparsity of the 2D image of point cloud sample values, created from the geometric projection of the 3D point cloud, can be a hurdle when compressing such images. Furthermore, there may be some degree of concern regarding the accuracy of point location, which depends on the distance of the point to a given point in space (e.g., a LiDAR sensor). Effective techniques are needed to manage the trade-off between coding efficiency and these aforementioned concerns.

[0050] Still referencing Figure 2 In this embodiment, the occupancy map can be used to pack (205) a sparsely filled 2D image into at least two smaller, more densely filled 2D images. The densely filled 2D images may still contain samples with values ​​not defined by the transformation / packaging. To improve the efficiency of subsequent image encoding (207), the densely filled 2D image can be padded (206) with interpolated sample values. The 2D image used to pack the sample values ​​can be selected based on criteria such as the distance of the sample to a given point in space (e.g., a LiDAR sensor).

[0051] Decoding can employ the opposite technique. In the example, the bitstream (211) can be parsed and entropy-decoded (212) into symbols or sub-bitstreams associated with at least two 2D images (representing sparsely coded images representing point cloud sample values), an occupancy map, and metadata. The at least two 2D images can be decoded (214), depadded (215) to mark sample values ​​not indicated by the occupancy map as unused, and unpacked (216) to create the sparsely coded image. The depadded (215) and unpacking (216) steps can utilize information from the occupancy map, which is available after decoding (213). The sparsely padded 2D image can be converted into a 3D representation (217), which can produce a reconstructed point cloud (218).

[0052] One challenge of the above mapping is encoding the distance *r* between point cloud samples and predetermined points (e.g., the spatial location of a LiDAR sensor) as sample values. Depending on the bit depth of the image codec used, the distance *r* is constrained to a 2-sample depth. In the example, 256 distances correspond to an 8-bit sample depth, 1024 distances correspond to a 10-bit sample depth, and so on. The resulting quantization error of the distance *r* can be high, which may hinder the effective use of the reconstructed point cloud.

[0053] In this embodiment, the r value is appropriately scaled such that the range of sample values ​​determined by the image encoder settings is utilized across the entire range of sample values. In this embodiment, the sample with the highest r value can be assigned the maximum value allowed by the bit depth constraint of the 2D image encoder. In the example, if the image codec uses 10-bit samples, the 3D sample with the maximum distance (or highest r value) can be assigned a 2... 10 -1 = 1023. Considering the maximum distance r (e.g., 1023), the r value of the remaining samples in the point cloud can be represented by smaller and linearly scaled sample values.

[0054] In this embodiment, a non-linear scaling of the r parameter is utilized. For example, quantization can involve the logarithm of r, thereby reducing quantization error and increasing the accuracy of the representation of r. The smaller R is, the closer the point cloud samples are to the sensor. Non-linear scaling is suitable for applications requiring high accuracy for points close to the sensor and lower accuracy for points far from the sensor.

[0055] In embodiments, other scales can be used, which can be optimized for certain application environments. In embodiments, LiDAR devices can be optimized to obtain results within a specific range category. An example is a LiDAR rangefinder optimized for ranges of 300 feet and 1000 feet. In this case, the r-value between 300 and 1000 feet can be emphasized, with a few outlier values ​​indicating larger or closer distances. This mapping can be achieved through linear mapping at different scales for r-values ​​between 0..300, 300..1000, and 1000..max, where max is the maximum phase delay provided by the LiDAR algorithm.

[0056] Figure 4 This is a schematic diagram illustrating point selection in a 3D point cloud based on the rho(r) value. For example... Figure 4As shown, the 3D point cloud generated by the LiDAR device is rendered into a 2D image (400). Based on range categories, the 2D image (400) can be classified into three regions (or scales) (402), (404), and (406). Region (402) is defined by the distance range between the origin (408) and d1, region (404) is defined by the distance range between d1 and d2, and region (406) is defined by the distance range greater than d2. Linear or non-linear scaling can be performed on the distances of samples within each region.

[0057] In an embodiment, the mapping from the 3D point cloud sample values ​​to the 2D representation may involve at least two 2D images.

[0058] In some cases, even after the mapping described above, the bit depth of a 2D image codec can be limited to accurately represent the distance (r) value of the input point cloud without significant quantization errors. The bit depth in an image codec may be limited by the codec design (e.g., 8, 10, or 12 bits). Practical implementation constraints may further limit the availability of bit depth. For example, while some image and video coding standards support up to 16-bit depths in their syntax, actual hardware implementations in chipset form may only be available in 8-bit or 10-bit variants. This situation can be partially addressed by classifying r sample values ​​into different categories and encoding samples from each category into a corresponding 2D image. Using the example above with a LiDAR rangefinder, a first 2D image can be used to represent points within 100 feet of the sensor, a second 2D image can be used to represent points between 300 and 1000 feet from the center (e.g., the location of the LiDAR sensor), and a third 2D image can be used to represent points beyond 1000 feet from the center. Embodiments of this disclosure include any number of categories and the resulting 2D images. The number of categories / images can be determined by factors including application requirements, the number of available decoders, etc.

[0059] The mapping of a single r value within each 2D image can follow a linear or non-linear scale, including the scales mentioned above.

[0060] In an embodiment, a value other than the distance (r) from the center can also be used to classify the sample into at least two 2D images. Figure 5 This illustrates the classification of 3D point clouds (500) generated by a LiDAR device based on the azimuth angle θ (theta) (or t value). For example... Figure 5As shown, the sensor's 360-degree view can be subdivided into four equally spaced regions, where the t-values ​​are between 0 and 90 degrees, 90 and 180 degrees, 180 and 270 degrees, and 270 and 360 degrees. This scaling can also be applied to the elevation angle φ. When classified using azimuth or elevation angles, the 2D image representation can cover certain directions observed from the sensor. For example, if the sensor is mounted on a car, one can imagine that the 2D image representing the range of angles in the car's direction of travel is likely to be most relevant, the range of angles in the rear of the car may be less relevant, and the range of angles in the sides of the car may only be relevant in certain situations or for certain purposes (such as collision avoidance or lane changing). Furthermore, the perceptible range of angles may also vary with the car's speed. So far, the category of angles can be advantageously selected by the application and can be adaptive based on parameters such as speed.

[0061] In embodiments of this disclosure, combinations of value ranges for the three dimensions of the spherical coordinate system (e.g., distance, azimuth θ, and elevation φ) are also possible. For example, eight 2D images can be created based on the following combination: two azimuth ranges (e.g., in front of / behind the sensor), two elevation ranges (e.g., upper / lower part of the view), and three different distances, resulting in 2×2×3=8 different categories and producing eight 2D images.

[0062] In this embodiment, the SRI classification described above can also be applied to the occupancy map.

[0063] Embodiments of this disclosure include techniques for encoding and decoding 3D point clouds (e.g., LiDAR point clouds). The encoding process may involve generating at least one 2D representation of the 3D point cloud. Transformation from the 3D point cloud to at least one 2D representation can lead to information loss when at least two points of the point cloud are projected onto the same sample in the 2D representation (e.g., a distance image). To reduce information loss, techniques are needed to create and encode (in the encoder) and decode (in the decoder) a list of unprojected points.

[0064] Compressing sparse point clouds (such as those from LiDAR devices) presents challenges because existing techniques may not offer a reasonable balance between compression efficiency and computational complexity. The sparsity of the 2D image of point cloud sample values, created from the geometric projection of the 3D point cloud, can be particularly problematic when compressing such images. When removing sparsity from 2D images through scaling, a large scaling factor increases the likelihood that at least two point cloud samples will be projected onto a single sample in the 2D representation. Therefore, techniques are needed to address this situation.

[0065] Figure 6An encoder (600) is shown for compressing (or encoding) an uncompressed, point cloud-represented signal (601) obtained by LiDAR (with associated encoding parameters (602) in some examples, such as sensor position / angle / tilt) into a bitstream (603). See again... Figure 6 Because regardless of the details of the decoder implementation, decoding a given bitstream (603) will produce the same point cloud, therefore, although the decoding process can be bit-precise, Figure 6 The encoding process can be specified in a non-bit-precise manner. The point cloud (601) obtained by LiDAR can be compressed into a bitstream (603) by transforming (604) the 3D point cloud (601) into a 2D representation called a sparse distance image (SRI). An occupancy map (OCM) can be generated (605) and encoded using, for example, lossless encoding (606) to make it usable by a bitstream merger / generator (607). The occupancy map can be used to pack (608) the sparsely filled 2D image into at least two smaller 2D images with denser filling. Those densely filled 2D images may still contain samples with values ​​not defined by the transformation / packaging and can be filled with interpolated sample values ​​to improve the efficiency of subsequent image encoding (609). The 2D image used to pack the sample values ​​can be selected based on criteria such as the distance of the sample to a given point in space (e.g., the LiDAR sensor). The image encoder (609) can use conventional or novel (e.g., AI-based) encoding techniques. The output of the image encoder is available to a merger / generator (607), which creates a bitstream (603) including extracted metadata (611). After appropriate extraction (611), input metadata (MTD) can be received from the input point cloud (601) and from the encoding parameter source (602), and after appropriate metadata encoding (610), it is used as input to the merger (607) to form the bitstream (603). Finally, the 3D to 2D conversion process (604) may result in points in the point cloud that are not represented in the sparse distance image. Information relating to those unprojected points (UNPs) can be forwarded to the unprojected point encoder (612). After encoding, the unprojected points can be sent to the merger (607) to form the bitstream (603).

[0066] refer to Figure 7Decoding can employ the opposite technique. For example, the bitstream (701) can be parsed and entropy decoded (702) into symbols or sub-bitstreams relating to, for example, a 2D image (representing a sparsely coded image (SRI) representing point cloud sample values), an occupancy map, unprojected point data, and metadata. The 2D image can be decoded (703), depadded to mark sample values ​​not indicated by the occupancy map as unused, and unpacked (707) to create a sparsely filled image and use it as input to a 2D-to-3D conversion (708). The depadded and unpacking steps (707) can utilize information from the occupancy map, which is available after the occupancy map has been encoded (704). The output of the depadded and unpacking steps (707) can also be input to the 2D-to-3D conversion (708). The parser (702) can also parse information relating to unprojected code points (UNPs), which can be decoded (705) and further input into the 2D-to-3D conversion (708). Furthermore, all the above data is converted (708) into a point cloud and output as a reconstructed point cloud (709). Finally, any encoding parameters and other metadata included in the bitstream (701) can be decoded (706) and output as parameters (710) in an appropriate format outside the point cloud.

[0067] When a 3D point cloud (601) is transformed (604) onto a 2D distance image (SRI), each point can be assigned to a specific pixel in the SRI based on its spatial coordinates in the 3D point cloud space. However, due to the projection from 3D to 2D, at least two points may correspond to the same pixel, for example, in dense or complex scenes. This overlap can present two challenges:

[0068] Data loss: In a distance-based image (SRI), a pixel can only be occupied by one point. Without mitigation measures, due to the mathematical principles of the transformation process, other points may populate (or fill) the same pixel. These other points may not be encoded, potentially leading to information loss.

[0069] Occlusion: Points representing details that are occluded or behind surfaces may be lost, thus affecting the accuracy of depth and structure representation.

[0070] In embodiments of this disclosure, unprojected points refer to points that are not directly assigned to the distance image without causing data loss or misrepresentation.

[0071] In an embodiment, when at least two conflicting points in the point cloud map to the same pixel in the SRI (e.g., corresponding to the same horizontal axis θ and the same vertical axis φ), the SRI can be filled with the point closest to a predefined point in space, such as the origin of the coordinate system (e.g., the location of a LiDAR sensor). Other data outside the SRI, i.e., a list of unprojected code points, can be used to gather information about these points not represented in the SRI to improve the completeness and accuracy of 3D reconstruction.

[0072] In this embodiment, for a point in the source point cloud, a depth value (e.g., distance to the sensor) is calculated for that point. The depth value can be used to classify whether a given point in the point cloud is represented in the SRI or in a list of unprojected points.

[0073] Figure 8 An example of the process flow (800) for generating a list of unprojected points associated with a 3D point cloud is shown. Figure 8 As shown, initially, all pixels (or samples) of the SRI are set to values ​​indicating "unoccupied" or "empty". In the example, this value is 0.

[0074] Let "current point" be a point in the point cloud currently being processed. At (S801), a 2D-to-3D projection technique (e.g., based on...) can be used. Figure 3 (Transform from Cartesian coordinates to spherical coordinates) to calculate the pixel position of the current point in the SRI.

[0075] Based on the calculated pixel position, if the pixel position in the SRI has a value indicating that it is unoccupied (according to (S802)), then in (S803) the current point is directly assigned to the pixel position. In other words, the depth value of the current point is stored as a pixel value in the SRI distance image.

[0076] If, according to (S802), the pixel position in the SRI is already occupied, the process proceeds to (S804) to determine whether the current pixel in the pixel position of the SRI can be replaced. The current pixel can be added to the list of unoccupied pixels, or the current pixel can be left unchanged while adding the current point of the 3D point cloud to the list of unoccupied pixels. The determination in (S804) can be driven by application requirements, content adaptation, etc.

[0077] In this embodiment, points closer to the LidAR sensor are encoded in the SRI, while points farther away can be added to the list of unprojected points. If the current pixel occupying a pixel location has a smaller depth (e.g., closer to the sensor), replacing the current pixel in the SRI with the current point (farther away) in the 3D point cloud might result in the loss of foreground information. Therefore, the SRI remains unchanged, and the current point is added to the list of unprojected points according to (S805) to preserve the data of the current point.

[0078] If the current pixel occupying the pixel location has a greater depth (farther from the sensor), the current point (closer to the sensor) can cover (or replace) the current pixel in the SRI at (S806). At (S807), the existing point (or current pixel) in the SRI can be added to the list of unprojected points.

[0079] The replacement decision at (S804) ensures that SRI prioritizes points closest to the sensor (e.g., a LiDAR sensor) without discarding valuable data. This is achieved by storing the furthest points in an unprojected list during a single pass through the point list.

[0080] In an embodiment, the replacement decision at (S804) may preferentially fill the SRI with pixel values ​​that are close to each other to facilitate subsequent SRI compression steps through reasonable frequency variation. In the example, the replacement decision may consider the pixel values ​​of the filling pixels in the geometric neighborhood within the SRI. Optimization variants can be designed, ranging from simple optimizations using already filled neighboring samples (circularly arranged around pixel addresses) to more complex optimizations rebalancing the list of unprojected samples after processing all points in the point cloud.

[0081] Once a list of unprojected points for the point cloud (in some cases, the entire point cloud) has been computed, it may be advantageous to remove certain entries based on the content of at least one of the distance image, the source point cloud, and the unprojected point cloud itself. This approach can effectively remove application-irrelevant outliers (e.g., due to excessive depth), thus preventing them from hindering the efficient encoding of the unprojected point list.

[0082] In the example, the mechanism (or algorithm) can calculate the distance to the source point cloud for each point in the unprojected point list using a KD-tree search. Based on the distance, points that do not contribute significantly to the total distance of the point cloud can be removed. Based on a fixed threshold or the total number of points to retain as needed, some points in the unprojected point list can be removed.

[0083] In the example, more advanced techniques, such as rate-distortion cost calculation, can be applied to assess the impact of each unprojected point on the metric (e.g., fidelity or total distance) and the cost required to encode the unprojected points.

[0084] refer to Figure 6 and Figure 7Once the unprojected point list is generated, it may be necessary to encode it (612). Similarly, once the relevant syntax elements are parsed (702) from the bitstream, and before the reverse process described above can be applied, the unprojected point list needs to be reconstructed (705). The unprojected point list includes entries with three components: the X (e.g., azimuth) and Y (e.g., elevation) coordinate components of the pixel location in the SRI, and the depth value Z (e.g., rho(r)). The numbering range of X, Y, and Z can vary depending on the application. For some applications, binary encoding with 32 bits for X and Y and 16 bits for Z has been found to be sufficient. The total number of entries in the unprojected point list can also vary depending on the application, typically ranging from a few to several hundred. Because the unprojected point list collects “outlier” points that the SRI may not be able to cover, the X, Y, and Z values ​​of a given entry in the unprojected point list typically do not show much correlation between themselves or with other entries in the list. Considering these parameters, designing encoding techniques for compressing the unprojected point list can be challenging. Furthermore, for many applications, the unprojected point list may be very small compared to a packed, padded, and compressed SRI. Therefore, even if strong compression is ideal, complex compression mechanisms may only offer limited benefits.

[0085] In an embodiment, the list of unprojected points is encoded in an uncompressed form, for example, using binary-encoded integer values ​​of the entries in the list of unprojected points.

[0086] In embodiments, known or newly designed mechanisms may be used based on variable-length codes, Huffman codes, exponential Golomb codes, etc. These codes can encode certain values ​​more efficiently than other codes, which may require application-specific knowledge of the probability distributions of the x, y, and z values ​​to use such codes effectively.

[0087] In this embodiment, more advanced adaptive entropy coding techniques can be used. For example, context-adaptive binary arithmetic coding (CABAC) can be trained and used to encode a list of unprojected points, which can lead to better coding efficiency than the mechanisms described above, but at the cost of additional complexity.

[0088] Still referencing Figure 6 and Figure 7Creating an occupancy map (605) may be advantageous because the 2D images (or at least two 2D images) produced by the previously described transformation (604) may be inherently sparse. Due to scaling factors, a resolution may be required, and this resolution may exceed the resolution needed by the relevant 2D image codec to process dense distance image scenes. An occupancy map with the same size as the SRI can use a binary representation to indicate whether a point (e.g., a point in a 3D point cloud) has been projected to a given location (e.g., a location in the SRI). Each point in the occupancy map (e.g., a point in a 3D point cloud) can represent (or indicate) whether the corresponding point (or sample or pixel) is present (1) or absent (0) in the sparse distance image. If the SRI does not present any interference from subsequent processes (e.g., packing and padding (608)), the occupancy map may be redundant, as it may not add significant information. In this case, the occupancy map can be omitted from the bitstream (603) and replaced with an indication that does not include the occupancy map. As indicated by this indication, a missing occupancy map can indicate that all points (or samples or pixels) in the SRI can be correlated and can be equivalent to an encoded occupancy map with all values ​​set to 1.

[0089] Packing (608) the SRI using the created occupancy map (605) may involve reorganizing the data in the SRI to utilize storage more efficiently by focusing on occupancy points. The occupancy map can be used to determine which samples in a 2D image have corresponding points. Packed images can be created to include more values ​​from occupancy points (or occupancy samples or occupancy pixels) and fewer values ​​from unoccupancy points. Various packing algorithms can be applied. For example, occupied pixels can be moved at the beginning of each row or column, or moved in raster order or z-scan order, etc. In embodiments of this disclosure, packing may not be necessary or advantageous. For example, if the SRI is already dense, the packing operation can be as simple as using the unmodified SRI (e.g., the original SRI) as the packed SRI.

[0090] Packaging (608) can reduce the size of the packaged SRI relative to the original SRI by excluding empty, unoccupied points.

[0091] To ensure that the 2D image can be unpacked at the decoder without compromising the functionality of the entire system, lossless encoding of the occupancy map can be advantageously performed (606). In the example, the occupancy map can be encoded using lossless profiles of HEVC or VVC. In the example, the occupancy map can be represented by a one-dimensional signal and encoded using zip encoding, run-length encoding, context-adaptive binary arithmetic encoding (CABAC), etc.

[0092] To improve the efficiency of the image encoder, unoccupied samples of the packed SRI can be padded (608) using sample data sufficient to design the image coding mechanism used. In the example, the sample data to be padded into the unoccupied samples can be determined based on the values ​​of neighboring samples. For example, the unoccupied samples can be padded by copying the values ​​of the nearest occupied samples in scan order. In the example, the unoccupied samples can be padded using more complex padded algorithms, such as seed padded, row and column padded, push / pull, and inverse intra-prediction based on intra-prediction techniques employed by the image codec. Padded (608) can be performed independently regardless of whether the SRI has been packed.

[0093] Compression (or encoding) of the packaged SRI (609) may involve known compression techniques, including zip, HEVC, VVC or neural coding methods.

[0094] The bitstream (603) format may include at least one of the following: (i) an encoded (packed and padded) SRI image bitstream; (ii) a bitstream representing an encoded occupancy map; (iii) metadata encoded in an appropriate format (e.g., in the form of at least one Supplemental Enhancement Information (SEI) message, a compressed XML document, etc.); and (iv) information relating to the unprojected points already described. The metadata can be extracted (611) from the source point cloud (601) and can be used to manipulate (or control) certain decoding steps, such as 2D-to-3D conversion (708). The metadata can be extracted by the decoder in an appropriate format as output to control rendering (not depicted). The metadata can be used for various purposes. In the example, the 2D image resolution depends on the angle of the LiDAR coverage. In the example, an elevation angle is required to set the vertical resolution of the 2D image. In the example, the minimum and maximum elevation angles at which points can be found can be signaled in the metadata.

[0095] The decoder (700) can perform steps characterized as the reverse of the encoder steps described above. The first step of the decoder (700) is, for example, to parse (702) the bitstream (701) by the demultiplexer of the decoder (700) to extract the packed / padded SRI and the sub-bitstream of the occupancy map, as well as metadata. The sub-bitstream (or second bitstream) of the packed / padded SRI can be decoded (703) by the first decoding component of the decoder (700) using a decoding technique that responds to the bitstream format created by the encoder (600). Similarly, the sub-bitstream (or first sub-bitstream) with the encoded occupancy map can be decoded (704) by the second decoding component of the decoder (700) using an appropriate decoding technique. The outputs of decoding (703) and decoding (704) can be used by de-padded (707) and unpacking (707) mechanisms (or steps) that operate on the decoded SRI and the decoded / unpadded SRI, respectively. For example, the second sub-stream is decoded to obtain a processed image, which is generated by packing and padding the SRI. The processed image can be de-padded by the de-padded component of the decoder (700) to obtain an unpadded image based on the occupancy information of the occupancy map. The unpadded image can be unpacked by the unpacking component of the decoder (700) to obtain the SRI based on the occupancy information of the occupancy map. The obtained SRI can be converted (708) back to a point cloud, which can utilize the decoded metadata (706) and decoded (705) unprojected points transmitted in the sub-stream of the bitstream (701) and the previously parsed (702).

[0096] Embodiments of this disclosure include techniques for encoding and decoding three-dimensional point clouds (e.g., LiDAR point clouds). The encoding process may involve generating at least one 2D representation of the point cloud. This 2D representation may be an image. The X and Y coordinates of the image may indicate the azimuth and elevation angles in polar coordinates of the point cloud, and sample values ​​in the image may indicate the depth of points in the point cloud. Depth may be the distance from the origin or the sensor to that point. To effectively represent depth, it may be necessary to quantize the calculated depth values.

[0097] Compression of sparse point clouds (such as those generated by LiDAR devices) presents challenges because related techniques may not offer a reasonable balance between compression efficiency and computational complexity. When a 3D point cloud represented in Cartesian coordinates is converted to polar coordinates, the 3D point cloud can be represented by a 2D image of sufficient resolution, where the X and Y coordinates of the 2D image can represent azimuth and elevation angles, and the sample values ​​in the 2D image can be the depth values ​​(e.g., rho(r)) of points in the 3D point cloud. In this scenario, the X and Y coordinates and the sample values ​​are all floating-point numbers and need to be properly rounded to integers before use. Furthermore, it is beneficial to appropriately quantize the depth to reflect application requirements. The nature of the quantization needs to be studied and specified.

[0098] One challenge of the above mapping is encoding the distance r between the point cloud sample and a predetermined point (e.g., the location of a LiDAR sensor in space) as a sample value. Figure 9 An example of a point cloud (900) rendered in 3D space in 2D space is shown. As shown in the 2D image (902), the point cloud (900) is converted to a polar coordinate representation, where the X-axis represents the azimuth, the Y-axis represents the elevation, and the depth (Z-axis) values ​​are represented as sample values ​​in grayscale. At the graph (904), the minimum and maximum Z values ​​corresponding to each azimuth value are provided. This distance can be limited to 2 based on the bit depth of the image codec used. 样本 Depth. For example, 256 distances correspond to 8-bit sample depth, 1024 distances correspond to 10-bit sample depth, and so on. The resulting quantization error can be high for effective use in reconstructing point clouds.

[0099] In this embodiment, vertical band quantization is used to quantize the r value.

[0100] Vertical band quantization can involve dividing a depth image into N vertical bands. These N vertical bands can be of equal or unequal size. Each vertical band represents a specific vertical portion of the depth image, corresponding to a portion of a 3D point cloud divided by azimuth measurements. By averaging or selecting representative depth values ​​within each band, the depth image can be efficiently quantized based on the average of the selected representative depth values ​​in each band, and thus compressed, resulting in faster processing and lower storage requirements. The quantized data can then be encoded using image or video encoders such as JPEG, HEVC, or VVC, employing bit depths such as 8, 10, or 12 bits, providing efficient storage and transmission.

[0101] Before quantizing each vertical band, the minimum and maximum depth values ​​can be used as auxiliary information and may therefore need to be encoded in the data stream to allow for proper inverse quantization on a band-by-band basis. Vertical band quantization can be more efficient if certain regions are closer to the sensor, as the precision within each band can be adjusted to better capture these variations. Therefore, full-range depth information for each band is preserved during compression and reconstruction.

[0102] Figure 10 Point clouds (e.g.) are shown Figure 9 An example of vertical band quantization on a point cloud (900). Figure 10As shown, the 2D image (1002) formed by projecting the 3D point cloud into 2D space is divided into four vertical bands (1006), (1008), (1010), and (1012). As shown in graph (1004), three vertical bands (1006), (1008), and (1010) are shown as continuous, while the fourth band (1012) covers the azimuth angle between 320 degrees and 40 degrees, and is therefore depicted as two intervals. For the bands (1006) and (1010) between 40 degrees and 130 degrees and between 230 degrees and 320 degrees, the numbering range of the minimum and maximum depth values ​​is significantly reduced compared to the other bands. Therefore, finer quantization is allowed, resulting in a more faithful representation of the depth data.

[0103] Embodiments of this disclosure include techniques for encoding and decoding three-dimensional point clouds (e.g., LiDAR point clouds). The encoding process may involve generating at least one 2D representation of the point cloud. While models for 3D Cartesian to 2D+ value polar coordinate transformations are known, these models can be improved by techniques that take into account auxiliary information of the source point cloud (such as the locations of at least two sensor sources). The auxiliary information, or information derived from the auxiliary information, can be used to transform 2D+ values ​​(e.g., 2D images) back to 3D.

[0104] Compression of sparse point clouds (such as those from LiDAR devices) presents challenges because existing techniques may not offer a reasonable balance between compression efficiency and computational complexity. The technique disclosed in this paper involves converting a 3D point cloud in Cartesian coordinates, auxiliary information about the sensor position, and other data into a polar coordinate representation arranged in a 2D format. The 2D format may include azimuth and elevation angles at the X and Y coordinates, as well as depth values ​​(e.g., distance from the polar origin) of the 3D point cloud as sample values ​​from a 2D image. When the source point cloud is acquired from at least two sensors, techniques for improving this mapping need to be considered by taking into account the positions and auxiliary information of at least two sensors. When the encoding system applies this mapping, the decoding system can advantageously invert it to recover the original point cloud.

[0105] The above mapping might be appropriate if the LiDAR sensor's location coincides with the origin of the polar coordinate system. However, point clouds can be acquired from at least two sensors (e.g., from at least two LiDAR sensors). Of these at least two sensors, at most one, may not necessarily coincide with the origin of the polar coordinate system. If the point cloud is correctly generated in Cartesian coordinates, it should accurately reflect the sensor returns (or return information) from all sensors. However, each sensor can measure different distances to a given point in the point cloud, resulting in varying return accuracy between sensors. Assuming azimuth and elevation angles are quantized (necessary for 3D-to-2D conversions to create integer pixel addresses), angle quantization errors may be smaller for sensors closer to the point. Finally, the sensor's capture angle may differ for a given point in space, which has an impact because LiDAR relies on laser light reflected back to the sensor. These are just three intuitive scenarios demonstrating the advantages of considering the sensor locations used to acquire point clouds.

[0106] In an embodiment, the LiDAR acquisition system can be characterized by three parameters: the total number of lasers, the angle of one of the lasers in the aine of points (lasers_theta), and the height of one of the lasers in the aine of points (lasers_z).

[0107] In this embodiment, an iterative correction mechanism (or process) can be used to improve the accuracy of projecting a 3D point cloud onto a 2D distance image based on auxiliary information including the three parameters described above. In this embodiment, calibration adjustments can also be applied through multiple iterations.

[0108] Initially, the vertical angle of a point in a 3D point cloud can be calculated using the arctangent of the z-coordinate on the horizontal distance (e.g., the Euclidean norm of the x and y coordinates). However, due to sensor defects and misalignment, the raw measurements of the x, y, and z coordinates may not be perfectly aligned with calibrated sensor readings. To address this misalignment, calibrated data can be employed, such as an array containing calibrated vertical angles (lasers_theta) and z-coordinate offsets (lasers_z).

[0109] In the first step of the iterative process, the difference between the measured vertical angle and the calibrated vertical angle for some or all points is calculated. The initial row index in the distance image (SRI) can be determined by finding a calibrated vertical angle that minimizes the difference for each point. The initial row index can correspond to the vertical position (row) of the point in the 2D image onto which it is projected.

[0110] After initialization, the loop iteratively corrects the vertical angle and row index. In each iteration, the z-coordinate of the point is adjusted by adding a calibration offset corresponding to the current row index of the calibration offset. This adjustment aims to correct any vertical displacement caused by sensor calibration errors. Based on the updated z-coordinate, the vertical angle can be recalculated. From the recalculated data, the difference between these updated vertical angles and the calibrated angle can be derived. By identifying the minimum difference, the row index can be updated to better align with the calibrated sensor measurements.

[0111] The aforementioned iterative adjustments can be performed for a predetermined number of iterations (e.g., 10 times), thereby gradually correcting the vertical alignment of points in the distance image. While adhering to computational complexity constraints, the predetermined number of iterations can be adjusted to the desired accuracy. Iteration can help converge to the most accurate representation of the 3D environment in 2D space by compensating for systematic errors in the sensor data.

[0112] As an alternative to a predetermined number of iterations, an adaptive criterion can be used, for example, if high accuracy is desired and computational complexity is not a concern. For instance, iterations can continue until, between two iterations, no further corrections are observed for all or a predetermined (e.g., a large number) of points. Another alternative could be that the cumulative correction error observed between two iterations becomes less than a predefined margin. Other similar adaptive techniques can also be designed. Furthermore, under the condition of terminating the iteration process described above, the system can consider the relative importance of each point when calculating the correction error. For example, in a LiDAR system optimized for parking, points closer to each sensor may be more relevant than points farther from the sensor. Conversely, for LiDAR in another application, points at a distance within another effective range may be more relevant. Therefore, the correction error for points within the effective range can be advantageously reduced before processing other points.

[0113] In the final step, the calibration offset can be applied to the z-coordinate one last time to ensure that the point is correctly vertically positioned.

[0114] By focusing on vertical alignment, the aforementioned iterative mechanism (or process) can significantly improve the fidelity of distance images. It ensures that every point in 3D space is accurately mapped to its corresponding location in the 2D image, which can be crucial for applications requiring precise depth information, such as autonomous navigation, obstacle detection, and 3D reconstruction. The iterative nature of this mechanism allows for gradual convergence to the most accurate solution, more effectively minimizing the impact of calibration discrepancies.

[0115] The above description can be illustrated with a simplified example. Consider a LiDAR system comprising five lasers. Each of the five lasers has a calibrated vertical angle. The z-coordinate offset zi = {0.1, 0.15, 0.2, 0.25, 0.3}.

[0116] Capture a 3D point at coordinates (x, y, z) = (10, 0, 1). The initial measured vertical angle can be calculated. Therefore, the 3D point... It equals arctan(1 / 10) ≈ 5.71°. Compare the measured vertical angle with the calibrated vertical angle. The values ​​are compared, and 5° is found to be the closest calibrated vertical angle. The row index of the second laser is assigned (or selected). During iteration, the z-coordinate of the 3D point is adjusted to znew by adding the corresponding offset zi indicated by the row index of the second laser: znew = z + zi = 1m + 0.15m = 1.15m. Therefore, the coordinates of the 3D point are updated to (10, 0, 1.15). The coordinates can be recalculated based on the new coordinates (10, 0, 1.15). For example In this embodiment, if the nearest laser does not change, the row index of each laser can remain constant. After adjusting the vertical angle of the 3D point, another... If the row becomes closer, update the row index accordingly to the one that is closer. Based on the zi corresponding to the new row index, further update znew. The iterative process can be repeated until convergence is achieved. For example, the iterative process in the current example can be performed on the vertical angle... The calibration process ends when the laser is calibrated, which ensures that the 3D points are correctly aligned with the calibration data.

[0117] The above algorithm (or correction algorithm) compensates for sensor errors by iteratively adjusting the z-coordinate and vertical angle, thereby improving the accuracy of projecting 3D point clouds onto 2D images. Figure 11 The iterative process based on the example above is shown, where the corrected points are ultimately projected into the distance image. For example... Figure 11 As shown, after iteration 1 (or the first iteration), the z-coordinate (1102) of the 3D point is 1, and the new (or updated) z-coordinate (1104) is 1.15.

[0118] Figure 12A flowchart of an overview process (1200) according to an embodiment of the present disclosure is shown. Process (1200) can be used in a decoder. In various embodiments, process (1200) is executed by processing circuitry, such as processing circuitry that performs the functions of decoder (200B), processing circuitry that performs the functions of decoder (700), etc. In some embodiments, process (1200) is implemented as software instructions, so that when processing circuitry executes the software instructions, processing circuitry executes process (1200). The process begins at (S1201) and proceeds to (S1210).

[0119] At (1210), a bitstream comprising encoded information of a 3D point cloud is received, the 3D point cloud comprising at least two points in 3D space.

[0120] In (1220), the bitstream is parsed into a first sub-bitstream and a second sub-bitstream.

[0121] In (1230), an occupancy map is determined based on the first sub-bitstream, and an SRI is determined based on the second sub-bitstream. The SRI is derived from the 3D point cloud in the 3D space and includes at least two samples. The occupancy map indicates whether at least one sample of the SRI has a corresponding point in the 3D point cloud.

[0122] At (1240), the 3D point cloud is reconstructed based on the SRI and the occupancy map.

[0123] In this embodiment, the second sub-stream is decoded to obtain a processed image. The processed image is generated by packing and padding the SRI. The processed image is de-padded based on the occupancy information of the occupancy map to obtain an unpadded image. The unpadded image is unpacked based on the occupancy information of the occupancy map to obtain the SRI.

[0124] In one embodiment, at least two points of the 3D point cloud are reconstructed based on the transformation from spherical coordinates to Cartesian coordinates of at least two samples of the SRI.

[0125] In an embodiment, the SRI is packaged based on the occupancy information of the occupancy map to obtain a packaged SRI by excluding at least one unoccupied sample of the SRI, the at least one unoccupied sample having no corresponding point in the 3D point cloud, and the packaged SRI is filled by filling at least one unoccupied sample of the packaged SRI with adjacent occupied samples to obtain a filled SRI.

[0126] In one embodiment, the bitstream is parsed into a third sub-bitstream associated with metadata from at least two decoding steps. This metadata is extracted from the 3D point cloud.

[0127] In an embodiment, the distance values ​​of at least two samples of the SRI are dequantized based on one of the following: (i) linear scaling, wherein the highest distance value among the distance values ​​of the at least two samples is determined based on a bit depth value, and (ii) nonlinear scaling, wherein the distance values ​​of at least two samples of the SRI are scaled based on a nonlinear algorithm, each of the distance values ​​indicating the distance between the corresponding sample of the SRI and a sensor configured to generate at least two points of the 3D point cloud.

[0128] In an embodiment, the distance values ​​of at least two samples of the SRI are divided into at least two scales, and the samples in each of the at least two scales are quantized based on the corresponding linear scaling.

[0129] In an embodiment, each of the at least two samples of the SRI includes a corresponding distance value, a corresponding azimuth value, and a corresponding elevation value, the corresponding distance value indicating the distance between the corresponding sample sensors configured to generate at least two points of the 3D point cloud. The SRI includes at least two sub-2D images, which are classified by one or a combination of the distance, azimuth, and elevation values ​​of the at least two samples of the SRI. Determining the SRI further includes dequantizing each of the at least two sub-2D images based on one of linear scaling and non-linear scaling.

[0130] In an embodiment, when at least two points in the 3D point cloud correspond to the same pixel location in the SRI with the same azimuth and elevation values, one of the at least two points in the 3D point cloud with the minimum distance to a predefined location is selected as the projection point to the SRI, and the other points in the at least two points in the 3D point cloud are included in the unprojected point list. The unprojected point list includes at least two unprojected points relative to the SRI.

[0131] In an embodiment, when the current point of the 3D point cloud corresponds to an unoccupied pixel position in the SRI, the current point is assigned to an unoccupied pixel position in the SRI. When (i) the current point of the 3D point cloud corresponds to an occupied pixel position in the SRI, and (ii) the distance value of the current point relative to a predefined position is less than the distance value of the pixel in the occupied pixel position relative to the predefined position, the pixel in the occupied pixel position is replaced by the current point. When (i) the current point of the 3D point cloud corresponds to an occupied pixel position in the SRI, and (ii) the distance value of the current point relative to the predefined position is greater than the distance value of the pixel in the occupied pixel position relative to the predefined position, the current point is assigned to an unprojected point list.

[0132] In an embodiment, whether a pixel at the occupied pixel position of the SRI is replaced by the current point of the 3D point cloud is determined based on which of the pixel at the occupied pixel position and the current point is closer to the neighboring pixel of the SRI.

[0133] In an embodiment, which of the at least two unprojected points in the list of unprojected points is not encoded is determined based on one of the following: (i) the contribution of one of the at least two unprojected points to the total distance, and (ii) the rate-distortion cost calculated based on the influence of one of the at least two unprojected points and the encoding cost of encoding the one of the at least two unprojected points.

[0134] In an embodiment, each of the at least two samples of the SRI includes a corresponding distance value, a corresponding azimuth value, and a corresponding elevation value indicating the distance between the corresponding sample and the sensor. The at least two sub-2D images are classified using the corresponding range of the azimuth values ​​of the at least two samples of the SRI. The minimum and maximum depth values ​​associated with each of the at least two sub-2D images are decoded. Each of the at least two sub-2D images is dequantized based on (i) a representative depth value of the corresponding one of the at least two sub-2D images, and (ii) the minimum and maximum depth values ​​associated with the corresponding one of the at least two sub-2D images.

[0135] In an embodiment, at least two points of the 3D point cloud are captured by at least two sensors, and each of the at least two sensors is associated with a corresponding calibrated vertical angle and a corresponding z-coordinate offset.

[0136] In this embodiment, the horizontal, vertical, and depth coordinates of at least two points in the 3D point cloud are determined based on the spherical coordinates of at least two samples from the SRI. The vertical angle of the current point is calculated based on the arctangent of the depth coordinate of the current point in the 3D point cloud at the horizontal distance from the current point, where the horizontal distance is the Euclidean norm of the horizontal and vertical coordinates of the current point. It is determined which of the calibrated vertical angles from the at least two sensors is closest to the vertical angle of the current point. According to preset conditions, the vertical coordinates of the current point are updated by adding a z-coordinate offset corresponding to one of the calibrated vertical angles that is closest to the vertical angle of the current point.

[0137] In this embodiment, the total number of updates is less than a predefined value, and the difference between one of the calibrated vertical angles and the vertical angle of the current point is greater than a predefined tolerance.

[0138] Then, the process proceeds to (S1299) and ends.

[0139] The process (1200) can be adjusted as appropriate. At least one step in the process (1200) can be modified and / or omitted. At least one additional step can be added. Any suitable implementation order can be used.

[0140] Figure 13 A flowchart of an overview process (1300) according to an embodiment of the present disclosure is shown. Process (1300) can be used in an encoder. In various embodiments, process (1300) is executed by processing circuitry, such as processing circuitry that performs the functions of encoder (200A), processing circuitry that performs the functions of encoder (600), etc. In some embodiments, process (1300) is implemented as software instructions, so that when processing circuitry executes the software instructions, processing circuitry executes process (1300). The process begins at (S1301) and proceeds to (S1310).

[0141] In (1310), a three-dimensional (3D) point cloud including at least two points in 3D space is converted into a sparse distance image (SRI) in two-dimensional (2D) space, the SRI including at least two samples.

[0142] At (1320), an occupancy map is determined based on the SRI. The occupancy map indicates whether at least one sample of the SRI has a corresponding point in the 3D point cloud.

[0143] At (1330), the SRI is packaged based on the occupancy information of the occupancy map to obtain a packaged SRI. The packaged SRI does not include at least one unoccupied sample of the SRI that does not have a corresponding point in the 3D point cloud.

[0144] In (1340), at least one unoccupied sample of the packaged SRI is filled based on the adjacent occupied samples of the packaged SRI to obtain a filled SRI.

[0145] In (1350), the occupancy map is encoded into the first sub-stream of the bitstream, and the filled SRI is encoded into the second sub-stream of the bitstream.

[0146] In an embodiment, the distance values ​​of the samples of the filled SRI are quantized based on one of the following: (i) linear scaling, wherein the highest distance value among the distance values ​​of the samples is determined according to the bit depth value, and (ii) nonlinear scaling, wherein the distance values ​​of the samples of the filled SRI are scaled based on a nonlinear algorithm, each of the distance values ​​indicating the distance between the corresponding sample of the filled SRI and a sensor configured to generate at least two points of the 3D point cloud.

[0147] In an embodiment, when at least two points in the 3D point cloud correspond to the same pixel location in the SRI with the same azimuth and elevation values, one of the at least two points in the 3D point cloud with the minimum distance to a predefined location is selected as the projection point to the SRI, and the other points in the at least two points in the 3D point cloud are included in the unprojected point list. The unprojected point list includes at least two unprojected points relative to the SRI.

[0148] Then, the process proceeds to (S1399) and ends.

[0149] The process (1300) can be adjusted as appropriate. At least one step in the process (1300) can be modified and / or omitted. At least one additional step can be added. Any suitable implementation order can be used.

[0150] According to another embodiment of this disclosure, a non-volatile computer-readable storage medium is provided for storing a bitstream encoded by an encoding method. In this encoding method, a 3D point cloud comprising at least two points in 3D space is converted into a Sample Injection Registry (SRI) in 2D space, the SRI comprising at least two samples. An occupancy map is determined based on the SRI. The occupancy map indicates whether at least one sample of the SRI has a corresponding point in the 3D point cloud. The SRI is packaged based on the occupancy information of the occupancy map to obtain a packaged SRI. The packaged SRI does not include at least one unoccupied sample of the SRI that does not have a corresponding point in the 3D point cloud. At least one unoccupied sample of the packaged SRI is padded based on adjacent occupied samples of the packaged SRI to obtain a padded SRI. The occupancy map is encoded into a first sub-bitstream of the bitstream, and the padded SRI is encoded into a second sub-bitstream of the bitstream.

[0151] According to another embodiment of this disclosure, a non-volatile computer-readable storage medium is provided for storing a bitstream that, when processed by a processor, causes the processor to perform point cloud decoding. For example, point cloud decoding is performed according to a format rule. The bitstream includes encoded information of a 3D point cloud. The format rule specifies that the 3D point cloud includes at least two points in 3D space. The format rule specifies that the bitstream is parsed into a first sub-bitstream and a second sub-bitstream. The format rule specifies that an occupancy map is determined based on the first sub-bitstream, and a Special Interest Rate (SRI) is determined based on the second sub-bitstream. The SRI is derived from the 3D point cloud in the 3D space and includes at least two samples. The occupancy map indicates whether at least one sample of the SRI has a corresponding point in the 3D point cloud. The format rule specifies that the 3D point cloud is reconstructed based on the SRI and the occupancy map.

[0152] According to another embodiment of this disclosure, a method for processing 3D point cloud data is provided. In this method, a bitstream comprising encoded information of a 3D point cloud is processed according to format rules. The format rules specify that the 3D point cloud comprises at least two points in 3D space. The bitstream is parsed into a first sub-bitstream and a second sub-bitstream. The format rules specify that an occupancy map is determined based on the first sub-bitstream, and a Special Interest Rate (SRI) is determined based on the second sub-bitstream. The SRI is derived from the 3D point cloud in the 3D space and comprises at least two samples, the occupancy map indicating whether at least one sample of the SRI has a corresponding point in the 3D point cloud. The format rules specify that the 3D point cloud is reconstructed based on the SRI and the occupancy map.

[0153] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored on at least one computer-readable medium. For example, Figure 14 A computer system (1400) is shown, which is adapted to implement certain embodiments of the disclosed subject matter.

[0154] The computer software can be encoded using any suitable machine code or computer language, and code including instructions can be created through mechanisms such as assembly, compilation, and linking. These instructions can be executed directly by at least one computer central processing unit (CPU), graphics processing unit (GPU), or through decoding, microcode, or other means.

[0155] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0156] Figure 14The components shown for the computer system (1400) are examples and are not intended to limit the scope or functionality of the computer software implementing embodiments of this disclosure. Nor should the configuration of the components be construed as having any dependency or requirement on any component or combination thereof shown in the exemplary embodiments of the computer system (1400).

[0157] The computer system (1400) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from at least one human user via tactile input (e.g., keyboard input, swiping, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-machine interface device may also be used to capture certain media, which need not be directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from still cameras), video (e.g., two-dimensional video, three-dimensional video including stereoscopic video), and point clouds (e.g., LiDAR point clouds).

[0158] The human-machine interface input device may include at least one of the following (only one is shown): keyboard (1401), mouse (1402), touchpad (1403), touch screen (1410), data glove (not shown), joystick (1405), microphone (1406), scanner (1407), camera (1408).

[0159] The computer system (1400) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of at least one human user through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (1410), data gloves (not shown), or joystick (1405), but may also include tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (1409), headphones (not shown)), visual output devices (e.g., screens (1410) including cathode ray tube screens, liquid crystal screens, plasma screens, organic light-emitting diode screens, each of which may or may not have touchscreen input functionality, each of which may or may not have tactile feedback functionality—some of which may output two-dimensional or more three-dimensional visual outputs by means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

[0160] The computer system (1400) may also include human-accessible storage devices and their associated media, such as optical media including high-density read-only / rewritable optical discs (CD / DVD ROM / RW) (1420) or similar media (1421), thumb drives (1422), removable hard disk drives or solid-state drives (1423), conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated devices based on ROM / ASIC / PLD such as security software protectors (not shown), and so on.

[0161] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0162] The computer system (1400) may also include an interface (1454) to at least one communication network (1455). For example, the network may be wireless, wired, or optical. The network may also be a local area network (LAN), wide area network (WAN), metropolitan area network (MAN), vehicular and industrial network, real-time network, latency-tolerant network, etc. Networks also include Ethernet, wireless LANs, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), cable or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicular and industrial networks (including CANbus), etc. Some networks typically require external network interface adapters for connection to certain general-purpose data ports or peripheral buses (1449) (e.g., a USB port on the computer system (1400); other systems are typically integrated into the core of the computer system (1400) via a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). By using any of these networks, the computer system (1400) can communicate with other entities. The communication can be unidirectional, used only for receiving (e.g., wireless television), unidirectional, used only for sending (e.g., CAN bus to certain CAN bus devices), or bidirectional, such as through a local area or wide area digital network to other computer systems. Each of the above networks and network interfaces can use certain protocols and protocol stacks.

[0163] The aforementioned human-computer interface device, human-accessible storage device, and network interface can be connected to the core (1440) of the computer system (1400).

[0164] The core (1440) may include at least one central processing unit (CPU) (1441), a graphics processing unit (GPU) (1442), a dedicated programmable processing unit in the form of a field-programmable gate array (FPGA) (1443), a task-specific hardware accelerator (1444), a graphics adapter (1450), etc. These devices, along with read-only memory (ROM) (1445), random access memory (1446), internal mass storage (e.g., internal non-user-accessible hard disk drive, solid-state drive, etc.) (1447), etc., may be connected via a system bus (1448). In some computer systems, the system bus (1448) may be accessed in the form of at least one physical connector to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus (1448) or connected via a peripheral bus (1449). In an embodiment, a screen (1410) may be connected to the graphics adapter (1450). Peripheral bus architectures include External Peripheral Component Interconnect (PCI), Universal Serial Bus (USB), etc.

[0165] The CPU (1441), GPU (1442), FPGA (1443), and accelerator (1444) can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM (1445) or RAM (1446). Transient data can also be stored in RAM (1446), while permanent data can be stored, for example, in internal mass storage (1447). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with at least one CPU (1441), GPU (1442), mass storage (1447), ROM (1445), RAM (1446), etc.

[0166] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of this disclosure, or they may be media and code well-known and usable by those skilled in the art of computer software.

[0167] By way of example and not limitation, a computer system having an architecture (1400), particularly a core (1440), can provide functionality as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to execute software contained in at least one tangible computer-readable medium. Such a computer-readable medium can be a medium associated with the aforementioned user-accessible mass storage, as well as specific memory of the non-volatile core (1440), such as internal mass storage (1447) or ROM (1445). Software implementing various embodiments of this disclosure can be stored in such a device and executed by the core (1440). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the core (1440), particularly the processor therein (including a CPU, GPU, FPGA, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (1446) and modifying such data structures according to software-defined processes. Alternatively or as an alternative, the computer system may provide logic hardwired or otherwise included in circuitry (e.g., an accelerator (1444)) that may replace or operate with the software to perform the specific process or a specific portion of the specific process described herein. References to software may include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuitry storing the execution of software (such as an integrated circuit (IC)), circuitry containing execution logic, or both. This disclosure includes any suitable combination of hardware and software.

[0168] The disclosed techniques can be used individually or in combination in any order. Furthermore, for each technique, the encoder and decoder can be implemented using processing circuitry (e.g., at least one processor or at least one integrated circuit). In the example, at least one processor executes a program stored in a non-volatile computer-readable storage medium.

[0169] The disclosed techniques described above can be implemented in image and / or video decoding or image and / or video encoding processes. The decoding / encoding process can be used in video decoder devices. Furthermore, the decoding / encoding process can also be used in video encoder devices. In some embodiments, the process is executed by processing circuitry, such as processing circuitry that performs the functions of a video decoder (200B), processing circuitry that performs the functions of a video decoder (700), etc. In other embodiments, the process is executed by processing circuitry, such as processing circuitry that performs the functions of a video encoder (200A), processing circuitry that performs the functions of a video encoder (600), etc. In some embodiments, the process is implemented as software instructions, so the processing circuitry executes the process when it executes the software instructions. In other embodiments, the process can be implemented as a hardware process on a chip, so the processing circuitry executes the process when it executes the hardware instructions. The process can be appropriately adjusted. Steps in the above process can be modified and / or omitted. Additional steps can be added. Any suitable implementation order can be used.

[0170] The above-described technologies can be implemented as computer software using computer-readable instructions and physically stored in at least one computer-readable medium. For example, a computer system may be adapted to implement certain embodiments of the disclosed subject matter. The computer software can be encoded using any suitable machine code or computer language, which can be used to create code including instructions through assembly, compilation, linking, or similar mechanisms. These instructions can be executed directly by at least one computer central processing unit (CPU), graphics processing unit (GPU), etc., or through interpretation, microcode execution, etc. These instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc. The components of the computer system are exemplary in nature and are not intended to impose any limitation on the scope or functionality of the computer software implementing the embodiments of this disclosure. The configuration of the components should not be construed as having any dependency or requirement on any or a combination of the components shown in the exemplary embodiments of the computer system. Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not cover transmission media, carrier waves, or other volatile signals.

[0171] As used in this disclosure, "at least one" or "one of" is intended to include any one or a combination of the listed elements. For example, references to at least one of A, B, or C, at least one of A, B, and C, at least one of A, B, and / or C, and at least one of A through C are intended to include only A, only B, only C, or any combination thereof. References to one of A or B and one of A and B are intended to include either A or B or (A and B). Where applicable, the use of "one of" does not exclude any combination of the listed elements, for example, when the elements are not mutually exclusive.

[0172] While at least two exemplary embodiments have been described in this disclosure, various modifications, arrangements, and equivalent substitutions of the embodiments are within the scope of this disclosure. Therefore, it should be understood that those skilled in the art can design various systems and methods that, while not explicitly shown or described herein, embody the principles of this disclosure and are thus within its spirit and scope.

[0173] The above disclosure also includes the following features. These features can be combined in various ways, and are not limited to the combinations mentioned below.

[0174] (1) A method for decoding a point cloud, the method comprising: receiving a bitstream comprising encoded information of a three-dimensional (3D) point cloud, the 3D point cloud comprising at least two points in a 3D space; parsing the bitstream into a first sub-bitstream and a second sub-bitstream; determining an occupancy map based on the first sub-bitstream and determining a Special Interest Rate (SRI) based on the second sub-bitstream, the SRI being derived from the 3D point cloud in the 3D space and comprising at least two samples, the occupancy map indicating whether at least one sample of the SRI has a corresponding point in the 3D point cloud; and reconstructing the 3D point cloud based on the SRI and the occupancy map.

[0175] (2) The method according to feature (1), wherein determining the SRI further comprises: decoding the second sub-stream to obtain a processed image, the processed image being generated by packing and padding the SRI; de-padding the processed image based on the occupancy information of the occupancy map to obtain an unfilled image; and unpacking the unfilled image based on the occupancy information of the occupancy map to obtain the SRI.

[0176] (3) The method according to claim (1) or (2), wherein the reconstruction of the 3D point cloud further comprises: reconstructing at least two points of the 3D point cloud based on the transformation from spherical coordinates to Cartesian coordinates of at least two samples of the SRI.

[0177] (4) The method according to any one of features (1) to (3), wherein the SRI is packaged based on occupancy information of the occupancy map to obtain a packaged SRI by excluding at least one unoccupied sample of the SRI, the at least one unoccupied sample having no corresponding point in the 3D point cloud, and the packaged SRI is filled by filling at least one unoccupied sample of the packaged SRI with adjacent occupied samples of the packaged SRI to obtain a filled SRI.

[0178] (5) The method according to any one of features (1) to (4), wherein the parsing further comprises: parsing the bitstream into a third sub-bitstream associated with metadata of at least two decoding steps, the metadata being extracted from the 3D point cloud.

[0179] (6) The method according to any one of features (1) to (5), wherein determining the SRI further comprises: dequantizing the distance values ​​of at least two samples of the SRI based on one of: (i) linear scaling, wherein the highest distance value among the distance values ​​of the at least two samples is determined based on a bit depth value, and (ii) nonlinear scaling, wherein the distance values ​​of at least two samples of the SRI are scaled based on a nonlinear algorithm, each of the distance values ​​indicating the distance between the corresponding sample of the SRI and a sensor configured to generate at least two points of the 3D point cloud.

[0180] (7) The method according to any one of features (1) to (6), wherein the distance values ​​of at least two samples of the SRI are divided into at least two scales, and the samples in each of the at least two scales are quantized based on the corresponding linear scaling.

[0181] (8) The method according to any one of features (1) to (7), wherein each of at least two samples of the SRI includes a corresponding distance value, a corresponding azimuth value and a corresponding elevation value, the corresponding distance value indicating the distance between corresponding sample sensors, the sensors being configured to generate at least two points of the 3D point cloud, the SRI including at least two sub-2D images, the at least two sub-2D images being classified by one or a combination of the distance value, azimuth value and elevation value of at least two samples of the SRI, and the determination of the SRI further includes: dequantizing each of the at least two sub-2D images based on one of linear scaling and nonlinear scaling.

[0182] (9) The method according to any one of features (1) to (8), wherein when at least two points in the 3D point cloud correspond to the same pixel position in the SRI having the same azimuth and elevation values, one of the at least two points in the 3D point cloud having the minimum distance value to a predefined position is selected as the projection point to the SRI, and other points in the at least two points in the 3D point cloud are included in an unprojected point list, the unprojected point list including at least two unprojected points relative to the SRI.

[0183] (10) The method according to any one of features (1) to (9), wherein when the current point of the 3D point cloud corresponds to an unoccupied pixel position in the SRI, the current point is assigned to an unoccupied pixel position in the SRI; when (i) the current point of the 3D point cloud corresponds to an occupied pixel position in the SRI, and (ii) the distance value of the current point relative to a predefined position is less than the distance value of the pixel in the occupied pixel position relative to the predefined position, the pixel in the occupied pixel position is replaced by the current point; and when (i) the current point of the 3D point cloud corresponds to an occupied pixel position in the SRI, and (ii) the distance value of the current point relative to the predefined position is greater than the distance value of the pixel in the occupied pixel position relative to the predefined position, the current point is assigned to an unprojected point list.

[0184] (11) The method according to any one of features (1) to (10), wherein whether a pixel in the occupied pixel position of the SRI is replaced by the current point of the 3D point cloud is determined based on which of the pixel in the occupied pixel position and the current point is closer to the neighboring pixel of the SRI.

[0185] (12) The method according to feature (9), further comprising: determining which of the at least two unprojected points in the list of unprojected points was not encoded based on one of the following: (i) the contribution of one of the at least two unprojected points to the total distance, and (ii) the rate-distortion cost calculated based on the influence of one of the at least two unprojected points and the encoding cost of encoding the one of the at least two unprojected points.

[0186] (13) The method according to feature (8), wherein each of the at least two samples of the SRI includes a corresponding distance value, a corresponding azimuth value, and a corresponding elevation value indicating the distance between the corresponding sample and the sensor, classifying the at least two sub-2D images by the corresponding range of the azimuth values ​​of the at least two samples of the SRI; and determining the SRI further includes: decoding the minimum depth value and the maximum depth value associated with each of the at least two sub-2D images; and dequantizing each of the at least two sub-2D images based on: (i) a representative depth value of the corresponding one of the at least two sub-2D images, and (ii) the minimum depth value and the maximum depth value associated with the corresponding one of the at least two sub-2D images.

[0187] (14) The method according to any one of features (1) to (13), wherein at least two points of the 3D point cloud are captured by at least two sensors, and each of the at least two sensors is associated with a corresponding calibrated vertical angle and a corresponding z-coordinate offset.

[0188] (15) The method according to feature (14), wherein the reconstruction of the 3D point cloud further includes: determining the horizontal coordinates, vertical coordinates and depth coordinates of at least two points of the 3D point cloud based on the spherical coordinates of at least two samples of the SRI; calculating the vertical angle of the current point based on the arctangent of the depth coordinate of the current point of the 3D point cloud on the horizontal distance of the current point, wherein the horizontal distance is the Euclidean norm of the horizontal and vertical coordinates of the current point; determining which of the calibrated vertical angles of the at least two sensors is closest to the vertical angle of the current point; and updating the vertical coordinates of the current point by adding a z-coordinate offset corresponding to one of the calibrated vertical angles, wherein one of the calibrated vertical angles is close to the vertical angle of the current point, according to preset conditions.

[0189] (16) The method according to feature (15), wherein the preset condition includes one of the following: the total number of updates is less than a predefined value, and the difference between one of the calibrated vertical angles and the vertical angle of the current point is greater than a predefined tolerance.

[0190] (17) A method for point cloud encoding, the method comprising: converting a three-dimensional (3D) point cloud including at least two points in a 3D space into a sparse distance image (SRI) in a two-dimensional (2D) space, the SRI including at least two samples; determining an occupancy map based on the SRI, the occupancy map indicating whether at least one sample of the SRI has a corresponding point in the 3D point cloud; packing the SRI based on occupancy information of the occupancy map to obtain a packed SRI, the packed SRI not including at least one unoccupied sample of the SRI that does not have a corresponding point in the 3D point cloud; filling at least one unoccupied sample of the packed SRI based on adjacent occupied samples of the packed SRI to obtain a filled SRI; and encoding the occupancy map into a first sub-bitstream of a bitstream and encoding the filled SRI into a second sub-bitstream of the bitstream.

[0191] (18) The method according to feature (17), wherein the encoding further comprises: quantizing the distance values ​​of the samples of the filled SRI based on one of: (i) linear scaling, wherein the highest distance value among the distance values ​​of the samples is determined according to a bit depth value, and (ii) nonlinear scaling, wherein the distance values ​​of the samples of the filled SRI are scaled based on a nonlinear algorithm, each of the distance values ​​indicating the distance between the corresponding sample of the filled SRI and a sensor configured to generate at least two points of the 3D point cloud.

[0192] (19) The method according to feature (17) or (18), wherein the conversion further comprises: when at least two points in the 3D point cloud correspond to the same pixel position in the SRI having the same azimuth value and the same elevation value, selecting one of the at least two points in the 3D point cloud having a minimum distance value to a predefined position as a projection point to the SRI, and including other points in the at least two points in the 3D point cloud in a list of unprojected points, the list of unprojected points including at least two unprojected points relative to the SRI.

[0193] (20) A method for processing three-dimensional (3D) point cloud data, the method comprising: processing a bitstream comprising encoded information of a 3D point cloud according to a format rule, the format rule specifying that the 3D point cloud comprises at least two points in a 3D space; the format rule specifying that the bitstream is parsed into a first sub-bitstream and a second sub-bitstream; determining an occupancy map based on the first sub-bitstream and determining a sparse distance image (SRI) based on the second sub-bitstream, the SRI being derived from the 3D point cloud in the 3D space and comprising at least two samples, the occupancy map indicating whether at least one sample of the SRI has a corresponding point in the 3D point cloud; and reconstructing the 3D point cloud based on the SRI and the occupancy map.

[0194] (21) A method for storing or transmitting a video stream, the video stream being generated according to the point cloud encoding method of any one of features (17) to (19), or decoded according to the point cloud decoding method of any one of features (1) to (16).

[0195] (22) An apparatus for point cloud decoding, comprising processing circuitry configured to perform the method described in any one of features (1) to (16).

[0196] (23) An apparatus for point cloud encoding, comprising processing circuitry configured to perform the method described in any one of features (17) to (19).

[0197] (24) A non-volatile computer-readable storage medium storing a video stream, the video stream being generated according to the point cloud encoding method of any one of features (17) to (19), or decoded according to the point cloud decoding method of any one of features (1) to (16). The encoding method includes: converting a three-dimensional (3D) point cloud comprising at least two points in 3D space into a sparse distance image (SRI) in two-dimensional (2D) space, the SRI comprising at least two samples; determining an occupancy map based on the SRI, the occupancy map indicating whether at least one sample of the SRI has a corresponding point in the 3D point cloud; packing the SRI based on the occupancy information of the occupancy map to obtain a packed SRI, the packed SRI not including at least one unoccupied sample of the SRI that does not have a corresponding point in the 3D point cloud; filling at least one unoccupied sample of the packed SRI based on adjacent occupied samples of the packed SRI to obtain a filled SRI; and encoding the occupancy map into a first sub-bitstream of the bitstream and encoding the filled SRI into a second sub-bitstream of the bitstream.

[0198] (25) A non-volatile computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform the method of any one of features (1) to (21).

Claims

1. A method for point cloud decoding, characterized in that, The method includes: Receive a bitstream comprising encoded information of a three-dimensional 3D point cloud, wherein the 3D point cloud comprises at least two points in 3D space; The bitstream is parsed into a first sub-bitstream and a second sub-bitstream; An occupancy map is determined based on the first sub-bitstream, and a sparse distance image (SRI) is determined based on the second sub-bitstream. The SRI is derived from the 3D point cloud in the 3D space and includes at least two samples. The occupancy map indicates whether at least one sample of the SRI has a corresponding point in the 3D point cloud. The 3D point cloud is reconstructed based on the SRI and the occupancy map.

2. The method according to claim 1, characterized in that, Determining the SRI further includes: The second sub-stream is decoded to obtain a processed image, which is generated by packing and padding the SRI; Based on the occupancy information of the occupancy map, the processed image is de-padded to obtain an unfilled image; and The unfilled image is unpacked based on the occupancy information of the occupancy map to obtain the SRI.

3. The method according to claim 1, characterized in that, The reconstruction of the 3D point cloud further includes: Based on the transformation from spherical coordinates to Cartesian coordinates of at least two samples from the SRI, reconstruct at least two points of the 3D point cloud.

4. The method according to claim 1, characterized in that, The SRI is packaged based on the occupancy information of the occupancy map, and the packaged SRI is obtained by excluding at least one unoccupied sample of the SRI, wherein the at least one unoccupied sample does not have a corresponding point in the 3D point cloud. The packaged SRI is filled by filling at least one unoccupied sample of the packaged SRI with adjacent occupied samples to obtain a filled SRI.

5. The method according to claim 1, characterized in that, The analysis further includes: The bitstream is parsed into a third sub-bitstream associated with metadata extracted from the 3D point cloud, which is associated with metadata from at least two decoding steps.

6. The method according to claim 1, characterized in that, Determining the SRI further includes: Dequantization of the distance values ​​of at least two samples of the SRI is performed based on one of the following: (i) linear scaling, wherein the highest distance value among the distance values ​​of the at least two samples is determined according to a bit depth value, and (ii) nonlinear scaling, wherein the distance values ​​of at least two samples of the SRI are scaled based on a nonlinear algorithm, each of the distance values ​​indicating the distance between the corresponding sample of the SRI and a sensor configured to generate at least two points of the 3D point cloud.

7. The method according to claim 1, characterized in that, Divide the distance values ​​of at least two samples of the SRI into at least two scales, and The samples in each of the at least two scales are quantized based on the corresponding linear scaling.

8. The method according to claim 1, characterized in that, Each of the at least two samples in the SRI includes a corresponding distance value, a corresponding azimuth value, and a corresponding elevation value, wherein the corresponding distance value indicates the distance between the corresponding sample sensors, which are configured to generate at least two points in the 3D point cloud. The SRI comprises at least two sub-2D images, which are classified using one or a combination of distance, azimuth, and elevation values ​​from at least two samples of the SRI. The determination of the SRI further includes: dequantizing each of the at least two sub-2D images based on one of linear scaling and nonlinear scaling.

9. The method according to claim 1, characterized in that, When at least two points in the 3D point cloud correspond to the same pixel location in the SRI with the same azimuth and elevation values... Select one of the at least two points in the 3D point cloud that has the minimum distance value to a predefined location as the projection point to the SRI, and Other points among the at least two points of the 3D point cloud are included in a list of unprojected points, which includes at least two unprojected points relative to the SRI.

10. The method according to claim 1, characterized in that, When the current point of the 3D point cloud corresponds to an unoccupied pixel position in the SRI, the current point is assigned to an unoccupied pixel position in the SRI. When (i) the current point of the 3D point cloud corresponds to an occupied pixel position in the SRI, and (ii) the distance value of the current point relative to a predefined position is less than the distance value of the pixel in the occupied pixel position relative to the predefined position, the pixel in the occupied pixel position is replaced by the current point. When (i) the current point of the 3D point cloud corresponds to the occupied pixel position in the SRI, and (ii) the distance value of the current point relative to the predefined position is greater than the distance value of the pixel in the occupied pixel position relative to the predefined position, the current point is assigned to the unprojected point list.

11. The method according to claim 1, characterized in that, Whether a pixel in the occupied pixel position of the SRI is replaced by the current point of the 3D point cloud is determined based on which of the pixel in the occupied pixel position and the current point is closer to the neighboring pixel of the SRI.

12. The method according to claim 9, characterized in that, Further includes: The un-coded one of the at least two un-projected points in the list is determined based on one of the following: (i) the contribution of one of the at least two un-projected points to the total distance, and (ii) the rate-distortion cost calculated based on the influence of one of the at least two un-projected points and the encoding cost of encoding the one of the at least two un-projected points.

13. The method according to claim 8, characterized in that, Each of the at least two samples in the SRI includes a corresponding distance value, a corresponding azimuth value, and a corresponding elevation value indicating the distance between the corresponding sample and the sensor. The at least two sub-2D images are classified using the corresponding range of azimuth angle values ​​of at least two samples of the SRI. as well as Determining the SRI further includes: Decode the minimum and maximum depth values ​​associated with each of the at least two sub-2D images; as well as Each of the at least two sub-2D images is dequantized based on: (i) a representative depth value of the corresponding one of the at least two sub-2D images, and (ii) the minimum depth value and the maximum depth value associated with the corresponding one of the at least two sub-2D images.

14. The method according to any one of claims 1-13, characterized in that, At least two points of the 3D point cloud were captured by at least two sensors, and Each of the at least two sensors is associated with a corresponding calibrated vertical angle and a corresponding z-coordinate offset.

15. The method according to claim 14, characterized in that, The reconstruction of the 3D point cloud further includes: Based on the spherical coordinates of at least two samples of the SRI, determine the horizontal, vertical, and depth coordinates of at least two points of the 3D point cloud; Based on the arctangent of the depth coordinates of the current point in the 3D point cloud on the horizontal distance of the current point, the vertical angle of the current point is calculated, where the horizontal distance is the Euclidean norm of the horizontal and vertical coordinates of the current point. Determine which of the calibrated vertical angles of the at least two sensors is closest to the vertical angle of the current point; and According to preset conditions, the vertical coordinate of the current point is updated by adding a z-coordinate offset corresponding to one of the calibrated vertical angles, where one of the calibrated vertical angles is close to the vertical angle of the current point.

16. The method according to claim 15, characterized in that, The preset conditions include one of the following: The total number of updates is less than the predefined value, and The difference between one of the calibrated vertical angles and the vertical angle of the current point is greater than a predefined tolerance.

17. A method for point cloud encoding, characterized in that, The method includes: A 3D point cloud comprising at least two points in 3D space is converted into a sparse distance image SRI in 2D space, wherein the SRI comprises at least two samples. An occupancy map is determined based on the SRI, the occupancy map indicating whether at least one sample of the SRI has a corresponding point in the 3D point cloud; The SRI is packaged based on the occupancy information of the occupancy map to obtain a packaged SRI. The packaged SRI does not include at least one unoccupied sample of the SRI that does not have a corresponding point in the 3D point cloud. At least one unoccupied sample of the packaged SRI is filled based on adjacent occupied samples of the packaged SRI to obtain a filled SRI; and The occupancy map is encoded into the first sub-stream of the bitstream, and the filled SRI is encoded into the second sub-stream of the bitstream.

18. The method according to claim 17, characterized in that, The encoding further includes: The distance values ​​of the samples of the filled SRI are quantized based on one of the following: (i) linear scaling, wherein the highest distance value among the distance values ​​of the samples is determined according to the bit depth value, and (ii) nonlinear scaling, wherein the distance values ​​of the samples of the filled SRI are scaled based on a nonlinear algorithm, each of the distance values ​​indicating the distance between the corresponding sample of the filled SRI and a sensor configured to generate at least two points of the 3D point cloud.

19. The method according to claim 17, characterized in that, The conversion further includes: When at least two points in the 3D point cloud correspond to the same pixel location in the SRI with the same azimuth and elevation values... Select one of the at least two points in the 3D point cloud that has the minimum distance value to a predefined location as the projection point to the SRI, and Other points among the at least two points of the 3D point cloud are included in a list of unprojected points, which includes at least two unprojected points relative to the SRI.

20. A method for processing three-dimensional point cloud data, characterized in that, The method includes: The bitstream containing encoded information of a 3D point cloud is processed according to format rules, wherein the format rules specify that the 3D point cloud includes at least two points in 3D space. The format rules specify that the bitstream is parsed into a first sub-bitstream and a second sub-bitstream; an occupancy map is determined based on the first sub-bitstream, and a sparse distance image (SRI) is determined based on the second sub-bitstream. The SRI is obtained from the 3D point cloud in the 3D space and includes at least two samples. The occupancy map indicates whether at least one sample of the SRI has a corresponding point in the 3D point cloud; and the 3D point cloud is reconstructed based on the SRI and the occupancy map.

21. A method for storing or transmitting a video stream, characterized in that, The video stream is generated by the point cloud encoding method according to any one of claims 17-19, or decoded by the point cloud decoding method according to any one of claims 1-16.

22. A device for point cloud decoding, characterized in that, The point cloud decoding apparatus includes processing circuitry and is configured to perform the point cloud decoding method according to any one of claims 1-16.

23. A point cloud encoding device, characterized in that, The point cloud encoding apparatus includes processing circuitry configured to perform the point cloud encoding method according to any one of claims 17-19.

24. A non-volatile computer-readable storage medium, characterized in that, The video stream is stored and generated by the point cloud encoding method according to any one of claims 17-19, or decoded by the point cloud decoding method according to any one of claims 1-16.

25. A non-volatile computer-readable storage medium, characterized in that, A storage instruction, when executed by at least one processor, causes the at least one processor to perform the method according to any one of claims 1-21.

Images