Method and device for point cloud processing and medium

Abstract

The embodiment of the invention provides a solution for point cloud processing. A method for point cloud processing is presented. The method comprises: for conversion between a current point cloud (PC) sample of a point cloud sequence and a bit stream of the point cloud sequence, obtaining a first group of points in the current PC sample, the first group of points representing low frequency information of the current PC sample; and performing a conversion based on the first set of points.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to point cloud processing techniques, and more specifically, to cooperative adaptive downsampling and upsampling of point clouds. Background Technology

[0002] A point cloud is a collection of data points in a three-dimensional (3D) plane, where each point has defined coordinates on the X, Y, and Z axes. Therefore, point clouds can be used to represent the physical content of three-dimensional space. For a wide range of immersive applications, from augmented reality to autonomous vehicles, point clouds have proven to be a promising way to represent 3D visual data.

[0003] Point cloud encoding and decoding standards have primarily evolved through the development of the well-known MPEG organization. MPEG, short for Moving Picture Experts Group, is one of the main standardization organizations for multimedia processing. In 2017, the MPEG 3D Graphics Codec Group (3DG) released a Call for Proposals (CFP) document, beginning the development of point cloud encoding and decoding standards. The final standard will include two categories of solutions. Video-based point cloud compression (V-PCC or VPCC) is suitable for point sets with relatively uniform point distributions. Geometry-based point cloud compression (G-PCC or GPCC) is suitable for sparser distributions. However, overall, there is a desire to further improve the encoding and decoding efficiency of conventional point cloud encoding and decoding techniques. Summary of the Invention

[0004] The embodiments of this disclosure provide a solution for point cloud processing.

[0005] In a first aspect, a method for point cloud processing is proposed. The method includes: for the conversion between a current point cloud (PC) sample of a point cloud sequence and a bitstream of the point cloud sequence, obtaining a first set of points in the current PC sample, the first set of points representing low-frequency information of the current PC sample; and performing the conversion based on the first set of points.

[0006] Based on the method according to the first aspect of this disclosure, point clouds are encoded and decoded based on a set of points representing low-frequency information of the point cloud. Compared with conventional solutions, the proposed method can advantageously utilize cooperative adaptive downsampling and upsampling of the point cloud to assist in encoding and decoding. In this way, encoding and decoding efficiency can be improved.

[0007] In a second aspect, an apparatus for point cloud processing is proposed. The apparatus includes a processor and a non-transitory memory having instructions thereon. When executed by the processor, the instructions cause the processor to perform the method according to the first aspect of this disclosure.

[0008] In a third aspect, a non-transitory computer-readable storage medium is proposed. This non-transitory computer-readable storage medium stores instructions that cause a processor to perform the method according to the first aspect of this disclosure.

[0009] In a fourth aspect, another non-transitory computer-readable recording medium is proposed. This non-transitory computer-readable recording medium stores a bitstream of a point cloud sequence generated by a method performed by an apparatus for point cloud processing. The method includes: acquiring a first set of points in a current PC sample of the point cloud sequence, the first set of points representing low-frequency information of the current PC sample; and generating a bitstream based on the first set of points.

[0010] In the fifth aspect, a method for storing a bitstream of a point cloud sequence is proposed. The method includes: acquiring a first set of points from the current PC sample of the point cloud sequence, the first set of points representing low-frequency information of the current PC sample; generating a bitstream based on the first set of points; and storing the bitstream in a non-transitory computer-readable recording medium.

[0011] The present invention is provided to present, in a simplified form, the concept choices further described below in the detailed description. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0012] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0013] Figure 1 This is a block diagram illustrating an example point cloud encoding / decoding system that can utilize the techniques disclosed herein; Figure 2 A block diagram illustrating an example point cloud encoder according to some embodiments of the present disclosure is shown; Figure 3 A block diagram illustrating an example point cloud decoder according to some embodiments of the present disclosure is shown; Figure 4 An example flow of a cooperative adaptive downsampling and upsampling method for point clouds according to embodiments of the present disclosure is shown; Figure 5 A flowchart of a method for point cloud processing according to embodiments of the present disclosure is shown; and Figure 6 A block diagram of a computing device in which various embodiments of the present disclosure may be implemented is shown.

[0014] Throughout all the accompanying drawings, the same or similar reference numerals generally refer to the same or similar elements. Detailed Implementation

[0015] The principles of this disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described for illustrative purposes only and to help those skilled in the art understand and implement this disclosure, and do not imply any limitation on the scope of this disclosure. In addition to the methods described below, the disclosure described herein can be implemented in various other ways.

[0016] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0017] The terms "an embodiment," "embodiment," "example embodiment," etc., used in this disclosure refer to embodiments that may include specific features, structures, or characteristics, but not every embodiment is required to include that specific feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Moreover, when a specific feature, structure, or characteristic is described in conjunction with an example embodiment, it is claimed that, whether explicitly described or not, such a feature, structure, or characteristic affecting its relation to other embodiments is within the knowledge of those skilled in the art.

[0018] It should be understood that although the terms “first” and “second”, etc., may be used herein to describe various elements, these elements should not be limited to these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0019] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” “having,” “having,” “containing,” and / or “comprising” as used herein indicate the presence of the said features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0020] Example Environment Figure 1This is a block diagram illustrating an example point cloud encoding / decoding system 100 from which the techniques of this disclosure can be utilized. As shown, the point cloud encoding / decoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a point cloud encoding device, and the destination device 120 may also be referred to as a point cloud decoding device. In operation, the source device 110 may be configured to generate encoded point cloud data, and the destination device 120 may be configured to decode the encoded point cloud data generated by the source device 110. The techniques of this disclosure are generally directed to encoding and / or decoding point cloud data, i.e., to support point cloud compression. Encoding and decoding can be effective in compressing and / or decompressing point cloud data.

[0021] Source device 100 and destination device 120 may include any of a variety of devices, including desktop computers, laptops, tablets, set-top boxes, handsets (such as smartphones and mobile phones), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, vehicles (e.g., land or sea vehicles, spacecraft, aircraft, etc.), robots, LiDAR devices, satellites, extended reality devices, etc. In some cases, source device 100 and destination device 120 may be equipped for wireless communication.

[0022] Source device 100 may include a data source 112, a memory 114, a GPCC encoder 116, and an input / output (I / O) interface 118. Destination device 120 may include an input / output (I / O) interface 128, a GPCC decoder 126, a memory 124, and a data consumer 122. According to this disclosure, the GPCC encoder 116 of source device 100 and the GPCC decoder 126 of destination device 120 may be configured to apply the point cloud encoding / decoding techniques of this disclosure. Therefore, source device 100 represents an example of an encoding device, and destination device 120 represents an example of a decoding device. In other examples, source device 100 and destination device 120 may include other components or arrangements. For example, source device 100 may receive data (e.g., point cloud data) from an internal or external source. Similarly, destination device 120 may interface with an external data consumer rather than including the data consumer in the same device.

[0023] Generally, data source 112 represents a source of point cloud data (i.e., raw, unencoded point cloud data) and can provide a sequential series of "frames" of point cloud data to GPCC encoder 116, which encodes the point cloud data for each frame. In some examples, data source 112 generates point cloud data. Data source 112 of source device 100 may include point cloud acquisition devices, such as any of various cameras or sensors, such as one or more cameras, archives containing previously acquired point cloud data, 3D scanners or light detection and ranging (LIDAR) devices, and / or data feed interfaces that receive point cloud data from data content providers. Thus, in some examples, data source 112 may generate point cloud data based on signals from a LIDAR device. Alternatively or additionally, point cloud data may be generated by a computer from scanners, cameras, sensors, or other data. For example, data source 112 may generate point cloud data, or a combination of real-time point cloud data, archived point cloud data, and computer-generated point cloud data. In each case, the GPCC encoder 116 encodes the acquired, pre-acquired, or computer-generated point cloud data. The GPCC encoder 116 can rearrange the frames of the point cloud data from the receiving order (sometimes referred to as the "display order") to an encoding / decoding order for encoding and decoding. The GPCC encoder 116 can generate one or more bitstreams comprising the encoded point cloud data. The source device 100 can then output the encoded point cloud data via I / O interface 118 for reception and / or retrieval by, for example, the I / O interface 128 of the destination device 120. The encoded point cloud data can be directly transmitted to the destination device 120 via I / O interface 118 through network 130A. The encoded point cloud data can also be stored on storage medium / server 130B for access by the destination device 120.

[0024] The memory 114 of the source device 100 and the memory 124 of the destination device 120 may represent general-purpose memory. In some examples, memory 114 and memory 124 may store raw point cloud data, such as raw point cloud data from data source 112 and raw, decoded point cloud data from GPCC decoder 126. Additionally or alternatively, memory 114 and memory 124 may store software instructions executable by, for example, GPCC encoder 116 and GPCC decoder 126. Although memory 114 and memory 124 are shown separately from GPCC encoder 116 and GPCC decoder 126 in this example, it should be understood that GPCC encoder 116 and GPCC decoder 126 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memory 114 and memory 124 may store encoded point cloud data, such as encoded point cloud data output from GPCC encoder 116 and input to GPCC decoder 126. In some examples, portions of memory 114 and memory 124 may be allocated as one or more caches, for example, to store raw, decoded, and / or encoded point cloud data. For example, memory 114 and memory 124 may store point cloud data.

[0025] I / O interfaces 118 and 128 may represent a wireless transmitter / receiver, a modem, a wired network component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where I / O interfaces 118 and 128 include wireless components, they may be configured to transmit data, such as encoded point cloud data, according to cellular communication standards such as 4G, 4G-LTE (Long Term Evolution), Advanced LTE, 5G, etc. In some examples where I / O interface 118 includes a wireless transmitter, they may be configured to transmit data, such as encoded point cloud data, according to other wireless standards such as the IEEE 802.11 specification. In some examples, source device 100 and / or destination device 120 may include corresponding system-on-chip (SoC) devices. For example, source device 100 may include a SoC device for performing functions belonging to GPCC encoder 116 and / or I / O interface 118, and destination device 120 may include a SoC device for performing functions belonging to GPCC decoder 126 and / or I / O interface 128.

[0026] The techniques disclosed herein can be applied to encoding and decoding to support any application in a variety of applications, such as communication between autonomous vehicles, communication between scanners, cameras, sensors and processing devices (such as local or remote servers), geomapping or other applications.

[0027] The I / O interface 128 of the destination device 120 receives an encoded bitstream from the source device 110. The encoded bitstream may include signaling information defined by the GPCC encoder 116, which is also used by the GPCC decoder 126, such as syntax elements having values ​​representing the point cloud. The data consumer 122 uses the decoded data. For example, the data consumer 122 may use the decoded point cloud data to determine the location of physical objects. In some examples, the data consumer 122 may include a display for presenting images based on the point cloud data.

[0028] The GPCC encoder 116 and GPCC decoder 126 can each be implemented as any of a variety of suitable encoder circuitry and / or decoder circuitry, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is implemented in part in software, the device may store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the technology of this disclosure. Each of the GPCC encoder 116 and GPCC decoder 126 may be included in one or more encoders or decoders, and either one may be integrated as part of a combined encoder / decoder (CODEC) in the respective device. Devices including the GPCC encoder 116 and / or GPCC decoder 126 may include one or more integrated circuits, microprocessors, and / or other types of devices.

[0029] The GPCC encoder 116 and GPCC decoder 126 can operate according to encoding / decoding standards, such as the Video Point Cloud Compression (VPCC) standard or the Geometric Point Cloud Compression (GPCC) standard. Generally, this disclosure can refer to the encoding and decoding (e.g., encoding and decoding) of frames to include encoded or decoded data. Encoded bitstreams typically include a series of values ​​for syntax elements representing encoding / decoding decisions (e.g., encoding / decoding modes).

[0030] A point cloud can contain a set of points in 3D space and can have attributes associated with those points. Attributes can be color information, such as R, G, B or Y, Cb, Cr, or reflectivity information, or other attributes. Point clouds can be acquired by various cameras or sensors, such as LiDAR sensors and 3D scanners, and can also be computer-generated. Point cloud data is used in a variety of applications, including but not limited to architecture (modeling), graphics (3D models for visualization and animation), and the automotive industry (LiDAR sensors for navigation aids).

[0031] Figure 2 This is a block diagram illustrating an example of a GPCC encoder 200 according to some embodiments of the present disclosure. The GPCC encoder 200 may be... Figure 1 An example of a GPCC encoder 116 in system 100 is shown. Figure 3 This is a block diagram illustrating an example of a GPCC decoder 300 according to some embodiments of the present disclosure. The GPCC decoder 300 may be... Figure 1 An example of the GPCC decoder 126 in the system 100 shown.

[0032] In both the GPCC encoder 200 and GPCC decoder 300, point cloud locations are encoded and decoded first. Attribute encoding and decoding depend on the decoded geometry. Figure 2 and Figure 3 In this configuration, Region Adaptive Hierarchical Transformation (RAHT) unit 218, Surface Approximation Analysis unit 212, RAHT unit 314, and Surface Approximation Synthesis unit 310 are options typically used for Category 1 data. Level of Detail (LOD) Generation unit 220, Lifting unit 222, LOD Generation unit 316, and Inverse Lifting unit 318 are options typically used for Category 3 data. All other units are common between Category 1 and Category 3.

[0033] For Category 3 data, the compressed geometry is typically represented as an octree from the root down to the leaf level of each voxel. For Category 1 data, the compressed geometry is typically represented by a pruned octree (i.e., an octree from the root down to the leaf level of blocks larger than voxels) plus a model approximating the surface within each leaf node of the pruned octree. In this way, both Category 1 and Category 3 data share the octree encoding / decoding mechanism, while Category 1 data can additionally utilize a surface model to approximate the voxels within each leaf node. The surface model used is a triangulation of each block comprising 1 to 10 triangles, producing a triangle soup. Therefore, the Category 1 geometry codec is called a triangle soup geometry codec, while the Category 3 geometry codec is called an octree geometry codec.

[0034] exist Figure 2 In the example, the GPCC encoder 200 may include a coordinate transformation unit 202, a color transformation unit 204, a voxelization unit 206, an attribute transfer unit 208, an octree analysis unit 210, a surface approximation analysis unit 212, an arithmetic coding unit 214, a geometric reconstruction unit 216, a RAHT unit 218, a LOD generation unit 220, a lifting unit 222, a coefficient quantization unit 224, and an arithmetic coding unit 226.

[0035] like Figure 2 As shown in the example, the GPCC encoder 200 can receive a set of locations and a set of attributes. Locations can include the coordinates of points in the point cloud. Attributes can include information about the points in the point cloud, such as the colors associated with those points.

[0036] The coordinate transformation unit 202 can apply transformations to the coordinates of a point to transform the coordinates from the initial domain to the transformation domain. The transformed coordinates can be referred to as transformed coordinates. The color transformation unit 204 can apply transformations to convert the color information of an attribute to different domains. For example, the color transformation unit 204 can convert color information from the RGB color space to the YCbCr color space.

[0037] In addition, Figure 2 In the example, voxelization unit 206 can voxelize the transformed coordinates. Voxelization of the transformed coordinates can include quantization and removal of some points in the point cloud. In other words, multiple points in the point cloud can be grouped into a single "voxel," which can then be considered a point in some respects. Furthermore, octree analysis unit 210 can generate an octree based on the voxelized transformed coordinates. Additionally, in Figure 2 In the example, the surface approximation analysis unit 212 can analyze points to potentially determine a surface representation of the set of points. The arithmetic coding unit 214 can perform arithmetic coding on the syntax elements representing the octree and / or surface determined by the surface approximation analysis unit 212. The GPCC encoder 200 can output these syntax elements in a geometric bitstream.

[0038] The geometric reconstruction unit 216 can reconstruct the transformed coordinates of points in the point cloud based on an octree, data indicating the surface determined by the surface approximation analysis unit 212, and / or other information. Due to voxelization and surface approximation, the number of transformed coordinates reconstructed by the geometric reconstruction unit 216 may differ from the number of original points in the point cloud. The resulting points may be referred to as reconstructed points. The attribute transfer unit 208 can transfer attributes of the original points in the point cloud to the reconstructed points in the point cloud data.

[0039] Furthermore, RAHT unit 218 can apply RAHT encoding to the attributes of the reconstructed points. Alternatively or additionally, LOD generation unit 220 and lifting unit 222 can apply LOD processing and lifting to the attributes of the reconstructed points, respectively. RAHT unit 218 and lifting unit 222 can generate coefficients based on the attributes. Coefficient quantization unit 224 can quantize the coefficients generated by RAHT unit 218 or lifting unit 222. Arithmetic encoding unit 226 can apply arithmetic encoding and decoding to the syntax elements representing the quantized coefficients. GPCC encoder 200 can output these syntax elements in the attribute bitstream.

[0040] exist Figure 3 In the example, the GPCC decoder 300 may include a geometric arithmetic decoding unit 302, an attribute arithmetic decoding unit 304, an octree synthesis unit 306, an inverse quantization unit 308, a surface approximation synthesis unit 310, a geometric reconstruction unit 312, a RAHT unit 314, an LOD generation unit 316, an inverse lifting unit 318, a coordinate inverse transformation unit 320, and a color inverse transformation unit 322.

[0041] The GPCC decoder 300 can acquire geometric bitstreams and attribute bitstreams. The geometric arithmetic decoding unit 302 of the decoder 300 can apply arithmetic decoding (e.g., CABAC or other types of arithmetic decoding) to the syntax elements in the geometric bitstream. Similarly, the attribute arithmetic decoding unit 304 can apply arithmetic decoding to the syntax elements in the attribute bitstream.

[0042] Octree synthesis unit 306 can synthesize octrees based on syntax elements parsed from the geometric bitstream. In the case of using surface approximation in the geometric bitstream, surface approximation synthesis unit 310 can determine the surface model based on syntax elements parsed from the geometric bitstream and based on the octree.

[0043] Furthermore, the geometric reconstruction unit 312 can perform reconstruction to determine the coordinates of points in the point cloud. The inverse coordinate transformation unit 320 can apply an inverse transformation to the reconstructed coordinates to transform the reconstructed coordinates (positions) of points in the point cloud from the transformation domain back to the initial domain.

[0044] Additionally, in Figure 3 In the example, dequantization unit 308 can dequantize attribute values. Attribute values ​​can be based on syntax elements obtained from the attribute bitstream (e.g., including syntax elements decoded by attribute arithmetic decoding unit 304).

[0045] Depending on how the attribute values ​​are encoded, RAHT unit 314 can perform RAHT encoding / decoding to determine the color value for a point in the point cloud based on the inversely quantized attribute values. Alternatively, LOD generation unit 316 and inverse boosting unit 318 can use level-of-detail (LOD) based techniques to determine the color value for a point in the point cloud.

[0046] In addition, Figure 3 In the example, the color inverse transformation unit 322 can apply an inverse color transformation to color values. The inverse color transformation can be the inverse of the color transformation applied by the color transformation unit 204 of the encoder 200. For example, the color transformation unit 204 can transform color information from the RGB color space to the YCbCr color space. Correspondingly, the color inverse transformation unit 322 can transform color information from the YCbCr color space to the RGB color space.

[0047] Figure 2 and Figure 3 Various units are shown to aid in understanding the operations performed by encoder 200 and decoder 300. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. A fixed-function circuit is a circuit that provides a specific function and is preset with respect to the operations that can be performed. A programmable circuit is a circuit that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by instructions in the software or firmware. A fixed-function circuit can execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more of these units can be different circuit blocks (fixed-function or programmable), and in some examples, one or more of these units can be integrated circuits.

[0048] The following describes in detail some exemplary embodiments of this disclosure. It should be understood that section headings are used in this document for ease of understanding and not to limit the embodiments disclosed in a section to that section only. Furthermore, while some embodiments are described with reference to GPCC or other specific point cloud codecs, the disclosed techniques are also applicable to other point cloud encoding and decoding techniques. Additionally, although some embodiments describe point cloud encoding and decoding steps in detail, it should be understood that the corresponding decoding steps for decoding will be implemented by the decoder.

[0049] 1. Brief Overview This disclosure relates to point cloud encoding / decoding preprocessing and post-processing techniques. Specifically, it relates to learning-based point cloud geometric downsampling and upsampling. This idea can be combined with point cloud encoding / decoding standards, such as the geometry-based point cloud compression (G-PCC) currently under development.

[0050] 2. Abbreviation G-PCC is a geometry-based point cloud compression technology. MPEG Motion Picture Experts Group 3. Introduction In point cloud compression, traditional octree encoding / decoding, mesh encoding / decoding, mapping encoding / decoding, and attribute encoding / decoding provide the basic ideas and frameworks for compression. Following their encoding principles and modular structures, various signal processing methods are used to design new modules or optimize and enhance existing ones. The same applies to learning-based point cloud compression; it can use neural network models to replace traditional modules and also optimize model parameters based on data-driven optimization.

[0051] Currently, both traditional and learning-based point cloud compression methods attempt to retain as much information as possible from the original point cloud while keeping the bitrate within acceptable limits. However, not all points in the original point cloud are important and worth preserving. Downsampling preprocessing can effectively reduce information redundancy in the original point cloud. Through downsampling preprocessing, the bitrate of the point cloud can be significantly reduced, while through upsampling postprocessing, the original reconstruction quality can be preserved as much as possible.

[0052] Furthest point sampling (FPS) has been widely used as a pooling operation in point cloud neural processing systems. However, FPS does not consider further processing of the sampled points and may lead to suboptimal performance. Recently, alternative subsampling methods have been proposed. Existing designs introduce a keypoint layer, which passes the points with the most active features to the next network layer. Existing designs use Gumbel subset sampling instead of FPS during the training of classification networks to improve their accuracy. For upsampling methods, most are based on the basic operations of PointNet. Existing designs introduce PU-Net, which learns multi-scale features for each point and expands the point set through a multi-branch MLP. However, PU-Net requires downsampling the input first to learn multi-scale features, which leads to unnecessary resolution loss. Existing designs propose PU-GAN, a generative adversarial network (GAN) designed to learn an upsampled point distribution. While the main contribution and performance gain come from the discriminator part, the generator architecture has received less attention in their work. All current upsampling and downsampling methods are executed separately; there is no joint upsampling and downsampling method. Furthermore, this approach is better suited for compression tasks, as it can preserve as much of the original reconstruction quality as possible while reducing the bit rate.

[0053] 3.1 Sparse Convolution To leverage the sparsity of point clouds, researchers have conducted numerous explorations, such as octree-based CNNs and sparse CNNs. In sparse CNNs, the data tensor is represented by a set of coordinates C and associated features F. Convolution only aggregates the features occupying the coordinates. It is defined as:

[0054] in and These are the input coordinates and the output coordinates. and Coordinates The input feature vector and the output feature vector at the location. Define 3D convolution kernels, covering... A set of positions centered, where the offset exist middle. Indicates offset The kernel value at the specified location. This sparse convolution leverages the sparsity of the point cloud to reduce complexity and computes only on the voxels that are currently occupied.

[0055] 4. Problem Existing learning-based point cloud downsampling and upsampling methods have the following problems: 1. Current downsampling and upsampling methods are separate and not well integrated for reconstruction tasks, post-processing, and pre-processing of compression.

[0056] 2. Most current downsampling methods are point-based, resulting in low-resolution point clouds that are not subsets of the original point cloud with errors. Furthermore, these methods are relatively expensive and can only handle small-scale point clouds.

[0057] 3. Most upsampling methods use features to directly regress the coordinates of the point cloud, which makes upsampling reconstruction difficult and prone to errors.

[0058] 4. Current upsampling and downsampling training mechanisms are simplistic, and this training method cannot obtain more suitable upsampling and downsampling networks.

[0059] 5. Detailed Solution To address the above-mentioned problems and some other issues not mentioned, the methods outlined below are disclosed. This invention should be considered as an example of explaining general concepts and not interpreted in a narrow sense. Furthermore, these inventions can be applied individually or in any combination.

[0060] In the following discussion, the term "encoder" refers to a model that encodes and decodes information to be transmitted via a signal. The term "decoder" refers to a model that decodes compressed bits to obtain the information transmitted via a signal.

[0061] 1) A method for downsampling point clouds to obtain low-frequency information was proposed.

[0062] a. In one example, low-frequency information can be evaluated based on the geometric properties of the point cloud, such as its geometry or geometric normals.

[0063] i. In addition, geometric structures can be used to evaluate low-frequency information in point clouds.

[0064] 1. In one example, a point set (such as a uniform point set, a farthest sampled point set, etc.) can be used to obtain the backbone structure of a point cloud.

[0065] a. In one example, the farthest sampling method can be used to obtain the farthest sample point set.

[0066] 2. In one example, a voxelized point cloud can be used to characterize the geometry of the point cloud.

[0067] a. In one example, sparse point clouds with large-grained voxels are represented as low-frequency information.

[0068] b. In one example, low-frequency information can be evaluated based on the properties of the point cloud, such as color information.

[0069] c. In one example, low-frequency information can be evaluated based on both the geometric and attribute properties of the point cloud.

[0070] d. In one example, low-frequency information can be obtained using traditional methods or learning-based schemes.

[0071] i. In one example, a neural network-based learning scheme can be used to acquire low-frequency information.

[0072] 1. In one example, sparse convolution can be used as a basic operation in convolutional networks.

[0073] a. In one example, the stride size of a sparse convolution can be N. For example, N equals 2.

[0074] b. In one example, N can be predefined.

[0075] c. In one example, N can be transmitted via a signal.

[0076] 2) A method for downsampling point clouds to obtain high-frequency features of point clouds was proposed.

[0077] a. In one example, high-frequency information in a point cloud can be evaluated using geometric structures.

[0078] i. In one example, the points lost through downsampling can be represented as high-frequency information.

[0079] b. In one example, high-frequency features can be extracted from high-frequency information using hand-designed operators or learning-based schemes.

[0080] i. In one example, a neural learning scheme can be used to acquire high-frequency features.

[0081] c. In one example, high-frequency features may include local high-frequency features and global high-frequency features.

[0082] i. In one example, a neural learning scheme can be used to acquire local high-frequency features and global high-frequency features.

[0083] ii. In one example, local high-frequency information can be obtained through graph convolution.

[0084] 1. In one example, graph convolution can form a graph based on the k nearest neighbors.

[0085] a. In one example, the graph in graph convolution can be obtained by finding the k nearest neighbors in Euclidean space.

[0086] b. In one example, the graph in graph convolution can be obtained by finding the k nearest neighbors in the feature space.

[0087] 2. In one example, features within a local graph can be aggregated using the summation of all points in the graph.

[0088] 3. In one example, features within a local graph can be aggregated by weighting the features of all points within the graph.

[0089] a. In one example, feature-weighted aggregation can be performed based on geometric distance.

[0090] b. In one example, feature-weighted aggregation can be performed based on feature similarity.

[0091] 4. In one example, sparse convolution can be used as a basic operation in graph aggregation.

[0092] a. In one example, features obtained through convolutional downsampling can be considered high-frequency features.

[0093] b. In one example, the stride size of a sparse convolution can be N. For example, N equals 2.

[0094] c. In one example, N can be predefined.

[0095] d. In one example, N can be transmitted via a signal.

[0096] iii. In one example, global high-frequency information can be obtained through graph aggregation.

[0097] 1. In one example, the global graph of global graph aggregation can be obtained over the entire point cloud.

[0098] a. In one example, the global graph can be obtained through a non-learning approach.

[0099] i. In one example, the global graph can be obtained by sampling from the farthest point.

[0100] b. In one example, the global graph can be obtained through a learning-based approach.

[0101] i. In one example, the global graph is formed from the top k high points based on the response value of each point obtained by the network.

[0102] 1. For example, k equals 8.

[0103] 2. In one example, features within a global graph can be aggregated using the summation of all points in the graph.

[0104] 3. In one example, features within a graph can be aggregated by weighting the features of all points within the graph.

[0105] a. In one example, feature-weighted aggregation can be performed based on geometric distance.

[0106] b. In one example, feature-weighted aggregation can be performed based on feature similarity.

[0107] 4. In one example, sparse convolution can be used as a basic operation in graph aggregation.

[0108] a. In one example, features obtained through convolutional downsampling can be considered high-frequency features.

[0109] b. In one example, the stride size of a sparse convolution can be N. For example, N equals 2.

[0110] c. In one example, N can be predefined.

[0111] d. In one example, N can be transmitted via a signal.

[0112] 3) A method for encoding and decoding the final sampled point cloud and transmitting it to the decoder via signal is proposed.

[0113] a. In one example, the final sampled point cloud can be encoded and decoded using a point cloud codec.

[0114] i. In one example, the point cloud codec could be G-PCC, V-PCC, Draco, etc.

[0115] 4) A method for decoding feature encoding and transmitting it to the decoder via signal is proposed.

[0116] a. In one example, features can be encoded and decoded using fixed-length codes, unary codes, rounded unary codes, etc.

[0117] b. In one example, features can be encoded and decoded in a predictive manner.

[0118] 5) It proposes using initial features to predict high-frequency features.

[0119] a. In one example, the prediction of high-frequency features can be achieved by simply using convolution operations to expand the feature dimensions.

[0120] b. In one example, the prediction of high-frequency features can be achieved by a general feature extraction network.

[0121] c. In one example, the prediction of high-frequency features can be performed by a generative adversarial network that includes a generator and a discriminator.

[0122] i. In one example, the generator can be executed using convolutional blocks with feature expansion operations.

[0123] 1. In one example, convolutional blocks can be implemented using sparse convolution.

[0124] a. In one example, multiple consecutive sparse convolution operations can be used. For example, three consecutive sparse convolution operations can be used.

[0125] ii. In one example, the discriminator can use convolutional blocks as the basic operation to perform.

[0126] 1. In one example, convolutional blocks can be implemented using sparse convolution.

[0127] a. In one example, multiple consecutive sparse convolution operations can be used. For example, three consecutive sparse convolution operations can be used.

[0128] 2. In one example, max pooling can be used to preserve details.

[0129] 3. In one example, average pooling can be used to preserve overall structural information.

[0130] d. In one example, the predicted high-frequency features can be constrained by an effective loss function.

[0131] i. In one example, the L2 distance loss function can be used to constrain the difference between the predicted high-frequency features and the true high-frequency features.

[0132] ii. In one example, cosine similarity loss can be used to constrain the difference between the predicted high-frequency features and the true high-frequency features.

[0133] iii. In one example, adversarial loss is used to constrain the distribution of the predicted high-frequency features.

[0134] 6) A method based on upsampling results of decoded point clouds to obtain reconstructed point clouds is proposed.

[0135] a. In one example, the reconstructed point cloud can be obtained directly through a single upsampling.

[0136] b. In one example, the reconstructed point cloud can be directly obtained through multiple progressive upsampling steps.

[0137] i. In one example, a point cloud can be reconstructed using N upsampling operations, for example, N=3.

[0138] 1. In one example, N can be predefined.

[0139] 2. In one example, N can be transmitted to the decoder via a signal.

[0140] a. In one example, N can be encoded or decoded using fixed-length codes, unary codes, rounded unary codes, etc.

[0141] b. In one example, N can be encoded and decoded in a predictive manner.

[0142] c. In one example, generative convolution based on sparse convolution can be used to implement point cloud upsampling.

[0143] d. In one example, during the upsampling training process, multi-stage loss functions with different granularities can be used to constrain the neural network.

[0144] i. In one example, the binary cross-entropy value can be used as the loss function in the first stage.

[0145] ii. In one example, the number of points used in the loss function may differ at different stages.

[0146] iii. In one example, the number of points used in the loss function for each stage can be indicated by at least one indication.

[0147] 1. In one example, there are 3 stages and 3 indications, M, N, and K.

[0148] a. In one example, M% of the points of the point cloud obtained by voxel sampling from the real point cloud can be used to constrain the reconstructed point cloud in the first stage.

[0149] b. In one example, N% of the points of the point cloud obtained by voxel sampling from the real point cloud can be used to constrain the reconstructed point cloud in the first stage.

[0150] c. In one example, K% of the points of the point cloud obtained by voxel sampling from the real point cloud can be used to constrain the reconstructed point cloud in the last stage.

[0151] d. In one example, M < N < K, such as M = 12.5, N = 50, K = 100.

[0152] iv. In one example, the indication can be predefined.

[0153] v. In one example, the indication can be transmitted by signal.

[0154] 7) Different training modes for training the downsampling and upsampling methods are proposed.

[0155] a. In one example, the downsampling and upsampling can be trained simultaneously.

[0156] b. In one example, the downsampling and upsampling can be trained separately.

[0157] c. In one example, the downsampling and upsampling can be trained simultaneously, and then the upsampling and downsampling can be trained separately.

[0158] 8) Whether and / or how to apply the methods disclosed above can be signaled from the encoder to the decoder in a bitstream / frame / slice / segment / octree / etc.

[0159] 9) Whether and / or how to apply the methods disclosed above can depend on the decoded information, such as dimension, color format, color component, slice / picture type.

[0160] 6. Embodiments Figure 4An example of the encoding / decoding flow for a cooperative adaptive downsampling and upsampling method for point clouds is shown. The flow of the cooperative adaptive downsampling and upsampling method for point clouds is illustrated below. First, the high-resolution original point cloud is passed through a learnable adaptive downsampling network to obtain a low-resolution point cloud best suited for the upsampling network. This significantly reduces the amount of data in the original point cloud and thus lowers the bit rate. Second, after the low-resolution point cloud with basic quality is solved from the decoder, initial features are assigned to the low-resolution point cloud. Third, the feature prediction module can generate high-frequency information features lost during downsampling, which can better guide the point cloud upsampling recovery. Finally, progressive upsampling is used to reconstruct the point cloud to reduce the reconstruction difficulty, and a multi-stage loss function is used to constrain the reconstruction results. Therefore, the multi-stage loss constraint can make the reconstruction better for the overall structure.

[0161] Further details of embodiments of this disclosure, relating to cooperative adaptive downsampling and upsampling of point clouds, will be described below. The embodiments of this disclosure should be considered as examples for explaining general concepts and should not be interpreted in a narrow manner. Furthermore, these embodiments may be applied individually or in any combination.

[0162] As used herein, the term "point cloud sequence" can refer to a sequence of one or more point clouds. The term "point cloud frame" or "frame" can refer to a point cloud within a point cloud sequence. The term "point cloud (PC) sample" can refer to a frame, a sub-region within a frame, an image, a slice, a subframe, a sub-image, a piece, a segment, or any other suitable processing unit.

[0163] Figure 5 A flowchart of a method 500 for point cloud processing according to some embodiments of the present disclosure is shown. Method 500 can be implemented during the conversion between a current PC sample of a point cloud sequence and a bitstream of the point cloud sequence. At 502, a first set of points in the current PC sample is acquired. The first set of points represents low-frequency information of the current PC sample. For example, the first set of points can be considered as a sparse point cloud, and the low-frequency information of the current PC sample can be determined based on the sparse point cloud.

[0164] In some embodiments, during encoding, the first set of points can be obtained by downsampling the current PC sample and transmitted as a bitstream via signal transmission. Correspondingly, during decoding, the first set of points can be obtained from the bitstream, for example, by parsing the bitstream. This will be described in detail below.

[0165] At point 504, the conversion is performed based on the first set of points. In some embodiments, the conversion may include encoding the current PC sample into a bitstream. Alternatively or additionally, the conversion may include decoding the current PC sample from the bitstream.

[0166] By way of example, and not limitation, at position 504, a first feature associated with the current PC sample is generated based on a first set of points. The first feature may be associated with high-frequency information of the current PC sample and may also be referred to as an "initial feature." For example, the first feature can be generated based on the first set of points using an ML-based model. In some example embodiments, the first feature may be transmitted via signaling in the bitstream. Alternatively, the first feature may be omitted from the bitstream.

[0167] Furthermore, predictions for high-frequency features can be generated based on the first feature. The high-frequency feature can represent high-frequency information of the current PC sample. Transformation can be performed based on predictions for the high-frequency features. For example, the current PC sample can be reconstructed based on predictions for the high-frequency features by applying an upsampling process to the first set of points. It should be understood that the above description is for illustrative purposes only. The scope of this disclosure is not limited in this respect.

[0168] In light of the above, point clouds are encoded and decoded based on a set of points representing low-frequency information. Compared to traditional solutions, the proposed method can advantageously utilize cooperative adaptive downsampling and upsampling of point clouds to assist in encoding and decoding. In this way, encoding and decoding efficiency can be improved.

[0169] In some embodiments, predictions for high-frequency features can be generated using machine learning (ML)-based models. As used herein, the term "model" refers to the association between inputs and outputs learned from training data, thus generating a corresponding output for a given input after training. Model generation can be based on machine learning techniques. Generally, machine learning models can be built that receive input information and make predictions based on that input. For example, a classification model can predict the class of input information in a predetermined set of classes. As used herein, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably herein.

[0170] For example, an ML-based model may include convolutional operations. Additionally or alternatively, an ML-based model may include a general feature extraction network. In some other example embodiments, an ML-based model may include generative adversarial networks, etc.

[0171] In some embodiments, the generator in a generative adversarial network may include convolutional blocks with feature expansion operations. For example, the convolutional block may be implemented using at least one sparse convolution. By way of example and not limitation, at least one sparse convolution may include multiple consecutive sparse convolutions, such as three or five consecutive sparse convolutions. It should be understood that the specific values ​​cited herein are intended to be exemplary and not to limit the scope of this disclosure.

[0172] In some additional or alternative embodiments, the discriminator in the generative adversarial network may include convolutional blocks. For example, the convolutional blocks may be implemented using at least one sparse convolution. By way of example, and not limitation, at least one sparse convolution may include multiple consecutive sparse convolutions, such as three or four consecutive sparse convolutions. Additionally, the discriminator may also include a max-pooling operation to preserve detail information. Additionally or alternatively, the discriminator may also include an average pooling operation to preserve overall structural information.

[0173] In some embodiments, at least one parameter of the ML-based model may be updated based on an adversarial loss function during the training process of the ML-based model. It should be noted that any other suitable loss function (such as L2 distance loss, cosine similarity loss, etc.) may be used to train the ML-based model.

[0174] In some embodiments, a second feature may be generated that is associated with local high-frequency information for the current point of the current PC sample. Furthermore, a third feature may be generated that is associated with global high-frequency information for the current PC sample. As used herein, the second feature may also be referred to as a “local high-frequency feature” and the third feature may also be referred to as a “global high-frequency feature”.

[0175] For example, the second and / or third features can be generated based on a second set of points obtained by downsampling the current PC samples. In some example embodiments, the second set of points can be the same as the first set of points. Alternatively, the second set of points can be different from the first set of points.

[0176] In some embodiments, the second feature may be generated using a neural network. For example, the second feature may be generated using a first graph convolution. As used herein, graph convolution may also be referred to as graph aggregation.

[0177] In some embodiments, the first graph used for convolution of the first graph (which may also be referred to as a “local graph”) may be generated based on a set of nearest neighbors of the current point, for example, based on the K nearest neighbors (KNN). In some embodiments, a set of nearest neighbors may be determined based on distances in Euclidean space or distances in feature space. As used herein, the features of a point may indicate occupancy information, specific distances, or values ​​derived based on geometric information.

[0178] In some embodiments, in the first graph convolution, features within the first graph may be aggregated by determining the sum of features of all points in the first graph. Alternatively, in the first graph convolution, features within the first graph may be aggregated by determining a weighted sum of features of all points in the first graph. In this case, the weights used to weight the features of a first point in the first graph may depend on the geometric distance between the first point and the current point. For example, the greater the distance between the first point and the current point, the higher the weight assigned to the features of the first point. Alternatively, the smaller the distance between the first point and the current point, the higher the weight assigned to the features of the first point. In some other example embodiments, the weights used to weight the features of a second point in the first graph may depend on the feature similarity between the second point and the current point. In this case, the higher the similarity between the features of the second point and the features of the current point, the higher the weight assigned to the features of the second point. Alternatively, the lower the similarity between the features of the second point and the features of the current point, the higher the weight assigned to the features of the second point.

[0179] In some embodiments, the first graph convolution can be implemented based on sparse convolution. For example, the stride size of the sparse convolution can be equal to a first number, such as 2, 3, etc. In one example embodiment, the first number can be predetermined. In another example embodiment, the first number can be indicated in the bitstream. In some embodiments, the second feature can be determined as a feature obtained through convolutional downsampling.

[0180] In some embodiments, the third feature may be generated using a neural network. For example, the third feature may be generated using a second graph convolution. In some embodiments, the second graph used for the second graph convolution (which may also be referred to as a "global graph") may be generated based on all points of the current PC sample.

[0181] In one example embodiment, the second graph may be generated using a non-learning-based scheme. By way of example, and not limitation, a non-learning-based scheme may include farthest point sampling, etc. In another example embodiment, the second graph may be generated using a learning-based scheme. For example, in a learning-based scheme, the response values ​​may be generated for each point in the current PC sample using an ML-based model. Then, one or more points used to generate the second graph may be selected from all points in the current PC sample based on the corresponding response values. By way of example, and not limitation, multiple points may include points with the top N response values, and N may be an integer, such as 8, 12, etc.

[0182] In some embodiments, in the second graph convolution, features within the second graph may be aggregated by determining the sum of features of all points in the second graph. In some alternative embodiments, in the second graph convolution, features within the second graph may be aggregated by determining a weighted sum of features of all points in the second graph. In this case, the weights used to weight the features of a third point in the second graph may depend on the geometric distance between the third point and the current point. For example, the greater the distance between the third point and the current point, the higher the weight assigned to the features of the third point. Alternatively, the smaller the distance between the third point and the current point, the higher the weight assigned to the features of the third point. In some further embodiments, the weights used to weight the features of a fourth point in the second graph may depend on the feature similarity between the fourth point and the current point. In this case, the higher the similarity between the features of the fourth point and the features of the current point, the higher the weight assigned to the features of the fourth point. Alternatively, the lower the similarity between the features of the fourth point and the features of the current point, the higher the weight assigned to the features of the fourth point.

[0183] In some embodiments, the second graph convolution can be implemented based on sparse convolution. For example, the stride size of the sparse convolution can be equal to a second number, such as 2, 4, etc. In one example embodiment, the second number can be predetermined. Alternatively, the second number can be indicated in the bitstream. In some embodiments, the third feature can be determined as a feature obtained through convolutional downsampling.

[0184] In some embodiments, the ML-based model used to generate predictions for high-frequency features may be trained based on second and / or third features. For example, a supervised training process can be applied to the ML-based model using second and third features. In this way, the trained ML-based model can better predict high-frequency features, thereby improving encoding and decoding quality.

[0185] In view of the above, the solutions according to some embodiments of this disclosure can advantageously improve encoding and decoding performance.

[0186] According to another embodiment of this disclosure, a non-transitory computer-readable recording medium is provided. This non-transitory computer-readable recording medium stores a bit stream of a point cloud sequence generated by a method performed by an apparatus for point cloud processing. In this method, a first set of points in a current PC sample of the point cloud sequence is acquired. The first set of points represents low-frequency information of the current PC sample. Furthermore, the bit stream is generated based on the first set of points.

[0187] According to further embodiments of this disclosure, a method for storing a bitstream of a point cloud sequence is provided. In this method, a first set of points in a current PC sample of the point cloud sequence is acquired. The first set of points represents low-frequency information of the current PC sample. Furthermore, a bitstream is generated based on the first set of points and stored in a non-transitory computer-readable recording medium.

[0188] The embodiments of this disclosure can be described according to the following entries, and their features can be combined in any reasonable manner.

[0189] Item 1. A method for point cloud processing, comprising: a conversion between a current point cloud (PC) sample of a point cloud sequence and a bitstream of the point cloud sequence; obtaining a first set of points in the current PC sample, the first set of points representing low-frequency information of the current PC sample; and performing the conversion based on the first set of points.

[0190] Item 2. The method according to Item 1, wherein performing the transformation comprises: generating a first feature associated with the current PC sample based on the first set of points; generating a prediction for a high-frequency feature based on the first feature, the high-frequency feature representing high-frequency information of the current PC sample; and performing the transformation based on the prediction for the high-frequency feature.

[0191] Item 3. The method according to Item 2, wherein performing the transformation based on the prediction for the high-frequency features comprises: reconstructing the current PC sample by applying an upsampling process to the first set of points based on the prediction for the high-frequency features.

[0192] Item 4. The method according to any one of Items 2 to 3, wherein the prediction for the high-frequency feature is generated using a machine learning (ML-based) model comprising at least one of the following: convolutional operation, general feature extraction network, or generative adversarial network.

[0193] Item 5. The method according to Item 4, wherein the generator in the generative adversarial network comprises a convolutional block having a feature expansion operation.

[0194] Item 6. The method according to Item 5, wherein the convolutional block is implemented using at least one sparse convolution.

[0195] Item 7. The method according to Item 6, wherein the at least one sparse convolution comprises a plurality of consecutive sparse convolutions.

[0196] Item 8. The method according to any one of Items 4 to 7, wherein the discriminator in the generative adversarial network comprises at least one of the following: a convolutional block, a max pooling operation, or an average pooling operation.

[0197] Item 9. The method according to Item 8, wherein the convolutional block is implemented using at least one sparse convolution.

[0198] Item 10. The method according to Item 9, wherein the at least one sparse convolution comprises a plurality of consecutive sparse convolutions.

[0199] Item 11. The method according to any one of Items 4 to 10, wherein during the training process of the ML-based model, at least one parameter of the ML-based model is updated based on an adversarial loss function.

[0200] Item 12. The method according to any one of items 2 to 11, wherein the first feature is generated based on the first set of points using an ML-based model.

[0201] Item 13. The method according to any one of items 1 to 12 further includes: generating a second feature associated with local high-frequency information for a current point of the current PC sample; and generating a third feature associated with global high-frequency information for the current PC sample.

[0202] Item 14. The method according to Item 13, wherein at least one of the second feature or the third feature is generated based on a second set of points obtained by downsampling the current PC sample.

[0203] Item 15. The method according to any one of items 13 to 14, wherein the second feature is generated using a neural network.

[0204] Item 16. The method according to any one of items 13 to 15, wherein the second feature is generated using convolution of the first graph.

[0205] Item 17. The method according to Item 16, wherein the first graph used for convolution of the first graph is generated based on a set of nearest neighbors of the current point.

[0206] Item 18. The method according to Item 17, wherein the set of nearest neighbors is determined based on distances in Euclidean space or distances in a feature space.

[0207] Item 19. The method according to any one of items 17 to 18, wherein in the first graph convolution, the features within the first graph are aggregated by determining the sum of features of all points in the first graph.

[0208] Item 20. The method according to any one of items 17 to 18, wherein in the first graph convolution, the features within the first graph are aggregated by determining a weighted sum of the features of all points in the first graph.

[0209] Item 21. The method according to Item 20, wherein the weights used to weight the features of the first point in the first graph depend on the geometric distance between the first point and the current point.

[0210] Item 22. The method according to Item 20, wherein the weights used to weight the features of the second point in the first graph depend on the feature similarity between the second point and the current point.

[0211] Item 23. The method according to any one of items 16 to 22, wherein the first graph convolution is implemented based on sparse convolution.

[0212] Item 24. The method according to Item 23, wherein the stride size of the sparse convolution is equal to the first number.

[0213] Item 25. The method according to Item 24, wherein the first number is predetermined or indicated in the bit stream.

[0214] Item 26. The method according to any one of items 13 to 25, wherein the second feature is determined as a feature obtained by convolutional downsampling.

[0215] Item 27. The method according to any one of items 13 to 26, wherein the third feature is generated using a neural network.

[0216] Item 28. The method according to any one of items 13 to 27, wherein the third feature is generated using second graph convolution.

[0217] Item 29. The method according to Item 28, wherein the second graph used for the second graph convolution is generated based on all points of the current PC sample.

[0218] Item 30. The method according to Item 29, wherein the second graph is generated using a non-learning-based scheme.

[0219] Item 31. The method according to Item 30, wherein the non-learning-based scheme includes farthest point sampling.

[0220] Item 32. The method according to Item 29, wherein the second graph is generated using a learning-based scheme.

[0221] Item 33. The method according to Item 32, wherein in the learning-based scheme, the response value is generated for each point in the current PC sample using an ML-based model, and multiple points for generating the second graph are selected from all points in the current PC sample based on the corresponding response value.

[0222] Item 34. The method according to Item 33, wherein the plurality of points includes points having the first N response values, and N is an integer.

[0223] Item 35. The method described in Item 34, wherein N equals 8.

[0224] Item 36. The method according to any one of items 29 to 35, wherein in the second graph convolution, the features within the second graph are aggregated by determining the sum of features of all points in the second graph.

[0225] Item 37. The method according to any one of items 29 to 35, wherein in the second graph convolution, the features within the second graph are aggregated by determining a weighted sum of the features of all points in the second graph.

[0226] Item 38. The method according to Item 37, wherein the weights used to weight the features of the third point in the second figure depend on the geometric distance between the third point and the current point.

[0227] Item 39. The method according to Item 37, wherein the weights used to weight the features of the fourth point in the second figure depend on the feature similarity between the fourth point and the current point.

[0228] Item 40. The method according to any one of items 28 to 39, wherein the second graph convolution is implemented based on sparse convolution.

[0229] Item 41. The method according to Item 40, wherein the stride size of the sparse convolution is equal to the second number.

[0230] Item 42. The method according to Item 41, wherein the second number is predetermined or indicated in the bit stream.

[0231] Item 43. The method according to any one of items 13 to 42, wherein the third feature is determined as a feature obtained by convolutional downsampling.

[0232] Item 44. The method according to any one of items 13 to 43, wherein the ML-based model for generating the prediction for the high-frequency feature is trained based on at least one of the second feature or the third feature.

[0233] Item 45. The method according to any one of items 1 to 44, wherein the first set of points is obtained from the bit stream.

[0234] Item 46. The method according to any one of items 1 to 44, wherein the first set of points is obtained by downsampling the current PC sample.

[0235] Item 47. The method according to any one of items 1 to 46, wherein the PC sample is one of the following: frame, picture, slice, subframe, subpicture, piece, or segment.

[0236] Item 48. The method according to any one of items 1 to 47, wherein the conversion includes encoding the current PC sample into the bitstream.

[0237] Item 49. The method according to any one of items 1 to 47, wherein the conversion includes decoding the current PC sample from the bitstream.

[0238] Item 50. An apparatus for point cloud processing, comprising a processor and a nontransitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of items 1 to 49.

[0239] Item 51. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method according to any one of items 1 to 49.

[0240] Item 52. A non-transitory computer-readable recording medium storing a bit stream of a point cloud sequence generated by a method performed by means of an apparatus for point cloud processing, wherein the method includes: acquiring a first set of points in a current PC sample of the point cloud sequence, the first set of points representing low-frequency information of the current PC sample; and generating the bit stream based on the first set of points.

[0241] Item 53. A method for storing a bitstream of a point cloud sequence, comprising: acquiring a first set of points in a current PC sample of the point cloud sequence, the first set of points representing low-frequency information of the current PC sample; generating the bitstream based on the first set of points; and storing the bitstream in a non-transitory computer-readable recording medium.

[0242] Example device Figure 6A block diagram of a computing device 600 in which various embodiments of the present disclosure may be implemented is shown. The computing device 600 may be implemented as a source device 110 (or GPCC encoder 116 or 200) or a destination device 120 (or GPCC decoder 126 or 300), or may be included in a source device 110 (or GPCC encoder 116 or 200) or a destination device 120 (or GPCC decoder 126 or 300).

[0243] It should be understood that, Figure 6 The computing device 600 shown is for illustrative purposes only and is not intended to imply any limitation on the functionality and scope of the embodiments of this disclosure.

[0244] like Figure 6 As shown, computing device 600 includes general-purpose computing device 600. Computing device 600 may include at least one or more processors or processing units 610, memory 620, storage unit 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660.

[0245] In some embodiments, computing device 600 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server, large computing device, etc., provided by a service provider. The user terminal can be, for example, any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, stations, units, devices, multimedia computers, multimedia tablet computers, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is conceivable that computing device 600 can support any type of interface to the user (such as "wearable" circuitry systems, etc.).

[0246] Processing unit 610 can be a physical processor or a virtual processor, and can perform various processes based on programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 600. Processing unit 610 may also be referred to as a central processing unit (CPU), microprocessor, controller, or microcontroller.

[0247] Computing device 600 typically includes various computer storage media. Such media can be any media accessible by computing device 600, including but not limited to volatile and non-volatile media, or removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or flash memory) or any combination thereof. Storage cell 630 can be any removable or non-removable media and may include machine-readable media, such as memory, flash drives, disks, or other media that can be used to store information and / or data and can be accessed within computing device 600.

[0248] The computing device 600 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Although in Figure 6 Not shown, but may provide disk drives for reading from and / or writing to removable non-volatile disks, and optical disc drives for reading from and / or writing to removable non-volatile optical discs. In this case, each drive may be connected to a bus (not shown) via one or more data media interfaces.

[0249] Communication unit 640 communicates with another computing device via a communication medium. Furthermore, the functionality of the components in computing device 600 can be implemented by a single computing cluster or by multiple computing machines communicating via communication connections. Therefore, computing device 600 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0250] Input device 650 can be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 660 can be one or more of various output devices, such as a monitor, speaker, printer, etc. With the aid of communication unit 640, computing device 600 can also communicate with one or more external devices (not shown), such as storage devices and display devices. Computing device 600 can also communicate with one or more devices that enable a user to interact with computing device 600, or any device that enables computing device 600 to communicate with one or more other computing devices (e.g., network card, modem, etc.), if needed. Such communication can be performed via an input / output (I / O) interface (not shown).

[0251] In some embodiments, some or all components of computing device 600 may be arranged in a cloud computing architecture, rather than integrated into a single device. In a cloud computing architecture, components may be remotely provided and work together to achieve the functionality described herein. In some embodiments, cloud computing provides computing, software, data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing uses a wide area network (WAN) such as the Internet with appropriate protocols to provide services. For example, a cloud computing provider provides applications via a WAN that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture, along with the corresponding data, may be stored on servers at remote locations. Computing resources in a cloud computing environment may be consolidated or distributed across locations in remote data centers. Cloud computing infrastructure may provide services through shared data centers, although they appear as a single access point to users. Therefore, a cloud computing architecture can be used to provide the components and functionality described herein from a service provider at a remote location. Alternatively, the components and functionality described herein may be provided from conventional servers or installed directly or otherwise on client devices.

[0252] In embodiments of this disclosure, computing device 600 may be used to implement point cloud encoding / decoding. Memory 620 may include one or more point cloud processing modules 625 having one or more program instructions. These modules are accessible and executable by processing unit 610 to perform the functions of the various embodiments described herein.

[0253] In an example embodiment of point cloud encoding, input device 650 may receive point cloud data as input 670 to be encoded. The point cloud data may be processed, for example, by point cloud processing module 625 to generate an encoded bitstream. The encoded bitstream may be provided as output 680 via output device 660.

[0254] In an example embodiment of point cloud decoding, input device 650 may receive an encoded bitstream as input 670. The encoded bitstream may be processed, for example, by point cloud processing module 625 to generate decoded point cloud data. The decoded point cloud data may be provided as output 680 via output device 660.

[0255] While this disclosure has been specifically shown and described with reference to preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this application as defined by the appended claims. These changes are intended to be covered by the scope of this application. Therefore, the foregoing description of embodiments of this application is not intended to be limiting.

Claims

1. A method for point cloud processing, comprising: For the conversion between the current point cloud (PC) sample of the point cloud sequence and the bit stream of the point cloud sequence, the first group of points in the current PC sample is obtained, and the first group of points represents the low-frequency information of the current PC sample; as well as The transformation is performed based on the first set of points.

2. The method of claim 1, wherein performing the conversion comprises: Generate a first feature associated with the current PC sample based on the first set of points; Based on the first feature, a prediction is generated for high-frequency features, where the high-frequency features represent high-frequency information of the current PC sample; and The transformation is performed based on the predictions made for the high-frequency features.

3. The method of claim 2, wherein performing the transformation based on the prediction for the high-frequency features comprises: Based on the prediction for the high-frequency features, the current PC sample is reconstructed by applying an upsampling process to the first set of points.

4. The method according to any one of claims 2 to 3, wherein the prediction for the high-frequency feature is generated using a machine learning (ML) based model comprising at least one of the following: Convolution operation, General feature extraction network, or Generate adversarial networks.

5. The method of claim 4, wherein the generator in the generative adversarial network comprises a convolutional block having a feature expansion operation.

6. The method of claim 5, wherein the convolutional block is implemented using at least one sparse convolution.

7. The method of claim 6, wherein the at least one sparse convolution comprises a plurality of consecutive sparse convolutions.

8. The method according to any one of claims 4 to 7, wherein the discriminator in the generative adversarial network comprises at least one of the following: Convolutional blocks, Max pooling operation, or Average pooling operation.

9. The method of claim 8, wherein the convolutional block is implemented using at least one sparse convolution.

10. The method of claim 9, wherein the at least one sparse convolution comprises a plurality of consecutive sparse convolutions.

11. The method according to any one of claims 4 to 10, wherein during the training process of the ML-based model, at least one parameter of the ML-based model is updated based on an adversarial loss function.

12. The method according to any one of claims 2 to 11, wherein the first feature is generated based on the first set of points using an ML-based model.

13. The method according to any one of claims 1 to 12, further comprising: Generate a second feature associated with the local high-frequency information of the current point for the current PC sample; as well as A third feature is generated that is associated with global high-frequency information for the current PC sample.

14. The method of claim 13, wherein at least one of the second feature or the third feature is generated based on a second set of points obtained by downsampling the current PC sample.

15. The method according to any one of claims 13 to 14, wherein the second feature is generated using a neural network.

16. The method according to any one of claims 13 to 15, wherein the second feature is generated using a first graph convolution.

17. The method of claim 16, wherein the first graph used for convolution of the first graph is generated based on a set of nearest neighbors of the current point.

18. The method of claim 17, wherein the set of nearest neighbors is determined based on distances in Euclidean space or distances in a feature space.

19. The method according to any one of claims 17 to 18, wherein in the first graph convolution, the features within the first graph are aggregated by determining the sum of features of all points in the first graph.

20. The method according to any one of claims 17 to 18, wherein in the first graph convolution, the features within the first graph are aggregated by determining a weighted sum of the features of all points in the first graph.

21. The method of claim 20, wherein the weights used to weight the features of the first point in the first graph depend on the geometric distance between the first point and the current point.

22. The method of claim 20, wherein the weights used to weight the features of the second point in the first graph depend on the feature similarity between the second point and the current point.

23. The method according to any one of claims 16 to 22, wherein the first graph convolution is implemented based on sparse convolution.

24. The method of claim 23, wherein the stride of the sparse convolution is equal to the first number.

25. The method of claim 24, wherein the first number is predetermined or indicated in the bit stream.

26. The method according to any one of claims 13 to 25, wherein the second feature is determined as a feature obtained by convolutional downsampling.

27. The method according to any one of claims 13 to 26, wherein the third feature is generated using a neural network.

28. The method according to any one of claims 13 to 27, wherein the third feature is generated using a second graph convolution.

29. The method of claim 28, wherein the second graph used for the convolution of the second graph is generated based on all points of the current PC sample.

30. The method of claim 29, wherein the second graph is generated using a non-learning-based scheme.

31. The method of claim 30, wherein the non-learning-based scheme includes farthest point sampling.

32. The method of claim 29, wherein the second graph is generated using a learning-based scheme.

33. The method of claim 32, wherein in the learning-based scheme, the response value is generated for each point in the current PC sample using an ML-based model, and a plurality of points for generating the second graph are selected from all points in the current PC sample based on the corresponding response value.

34. The method of claim 33, wherein the plurality of points includes points having the first N response values, and N is an integer.

35. The method of claim 34, wherein N equals 8.

36. The method according to any one of claims 29 to 35, wherein in the second graph convolution, the features within the second graph are aggregated by determining the sum of features of all points in the second graph.

37. The method according to any one of claims 29 to 35, wherein in the second graph convolution, the features within the second graph are aggregated by determining a weighted sum of the features of all points in the second graph.

38. The method of claim 37, wherein the weights used to weight the features of the third point in the second figure depend on the geometric distance between the third point and the current point.

39. The method of claim 37, wherein the weights used to weight the features of the fourth point in the second figure depend on the feature similarity between the fourth point and the current point.

40. The method according to any one of claims 28 to 39, wherein the second graph convolution is implemented based on sparse convolution.

41. The method of claim 40, wherein the stride size of the sparse convolution is equal to the second number.

42. The method of claim 41, wherein the second number is predetermined or indicated in the bit stream.

43. The method according to any one of claims 13 to 42, wherein the third feature is determined as a feature obtained by convolutional downsampling.

44. The method according to any one of claims 13 to 43, wherein the ML-based model for generating the prediction for the high-frequency feature is trained based on at least one of the second feature or the third feature.

45. The method according to any one of claims 1 to 44, wherein the first set of points is obtained from the bit stream.

46. ​​The method according to any one of claims 1 to 44, wherein the first set of points is obtained by downsampling the current PC sample.

47. The method according to any one of claims 1 to 46, wherein the PC sample is one of the following: frame, picture, slice, Subframe, Sub-images, film, or part.

48. The method according to any one of claims 1 to 47, wherein the conversion comprises encoding the current PC sample into the bitstream.

49. The method of any one of claims 1 to 47, wherein the conversion comprises decoding the current PC sample from the bitstream.

50. An apparatus for point cloud processing, comprising a processor and a nontransitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 49.

51. A non-transitory computer-readable storage medium storing instructions that cause a processor to perform the method according to any one of claims 1 to 49.

52. A non-transitory computer-readable recording medium storing a bitstream of a point cloud sequence generated by a method performed by means of a point cloud processing apparatus, wherein the method comprises: Obtain the first set of points in the current PC sample of the point cloud sequence, where the first set of points represents the low-frequency information of the current PC sample; as well as The bit stream is generated based on the first set of points.

53. A method for storing a bitstream of a point cloud sequence, comprising: Obtain the first set of points in the current PC sample of the point cloud sequence, where the first set of points represents the low-frequency information of the current PC sample; The bit stream is generated based on the first set of points; as well as The bitstream is stored in a non-transitory computer-readable recording medium.

Images