Unsupervised 3D point cloud refinement and segmentation

Abstract

In one implementation, we propose an unsupervised point cloud primitive learning method based on the principle of analysis by composition. In one example, the method uses a partitioning network and a point cloud autoencoder. The partitioning network partitions the input point cloud into a list of chunks. For each chunk, the encoder network of the autoencoder performs analysis to output a codeword in the feature space, and the decoder network performs synthesis to reconstruct the point cloud chunk. The reconstructed chunks are merged to output a fully reconstructed point cloud frame. The autoencoder discovers primitive shapes in the point cloud data by end-to-end training to minimize the misalignment between the original and reconstructed point clouds. During network training, the parameters of the partitioning network and the autoencoder are updated. The trained modules can be applied to different applications, including segmentation, detection, and compression.

Description

[Technical field]

[0001] The present embodiments relate generally to methods and apparatus for compression, analysis, interpolation, representation, and understanding of point cloud signals. Summary of the Invention [Problem to be solved by the invention]

[0002] The Point Cloud (PC) data format is a universal data format across several business domains, for example, from autonomous driving, to robotics, Augmented Reality / Virtual Reality (AR / VR), civil engineering, computer graphics, and animation / film industries. Three-dimensional Light Detection and Ranging (LiDAR) sensors are being deployed in autonomous vehicles, and affordable LiDAR sensors have been released from Velodyne Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR Camera L515. Advances in sensing technology make three-dimensional point cloud data more practical than ever and are expected to be a critical enabler in the applications discussed herein.

[0003] According to one embodiment, a method is provided, the method including: accessing a chunk of points from an input point cloud; generating, by a first neural network based module, a code word describing at least one shape in the point cloud chunk; reconstructing, by a second neural network based module, the point cloud chunk based on the code word; obtaining a mismatch metric based on the reconstructed point cloud chunk and the point cloud chunk; and adjusting parameters of the first neural network based module and the second neural network based module based on the mismatch metric.

[0004] According to another embodiment, an apparatus is provided, comprising: one or more processors; and at least one memory coupled to the one or more processors; the one or more processors are configured to access a chunk of points from an input point cloud; generate, by a first neural network-based module, a code word describing at least one shape in the point cloud chunk; reconstruct, by a second neural network-based module, the point cloud chunk based on the code word; obtain a mismatch metric based on the reconstructed point cloud chunk and the point cloud chunk; and adjust parameters of the first neural network-based module and the second neural network-based module based on the mismatch metric.

[0005] Additionally, one or more embodiments provide computer program instructions including instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of the embodiments described herein. Additionally, one or more of the present embodiments provide a computer readable storage medium having stored therein instructions for processing point cloud data according to the methods described herein.

[0006] One or more embodiments also provide a computer-readable storage medium that stores video data generated by the methods described herein. One or more embodiments also provide methods and apparatus for transmitting or receiving video data generated by the methods described herein. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates a block diagram of a system in which aspects of the present embodiments may be implemented. [Diagram 2] FIG. 2 illustrates a method for primitive learning according to one embodiment. [Diagram 3] Figure 3 illustrates PointNet as an encoder network. [Figure 4]Figure 4 illustrates FoldingNet as a decoder network. [Diagram 5] FIG. 5 illustrates a method for partition-based primitive learning according to one embodiment. [Figure 6] Figure 6 illustrates PointNet++ as a partitioned network. [Figure 7] FIG. 7 illustrates a method for generating primitives according to one embodiment. [Figure 8] FIG. 8 illustrates a method for learning primitives for segmentation according to one embodiment. [Figure 9] FIG. 9 illustrates a method of primitive learning for object detection according to one embodiment. [Figure 10] FIG. 10 illustrates a method of primitive learning for PCC (encoder) according to one embodiment. [Figure 11] FIG. 11 illustrates a method of primitive learning for PCC (decoder) according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] FIG. 1 illustrates a block diagram of an example of a system in which various aspects and embodiments may be implemented. System 100 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected appliances, and servers. The elements of system 100, alone or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 100 are distributed across multiple ICs and / or separate components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices, for example, through a communication bus or dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more of the aspects described herein.

[0009] The system 100 includes at least one processor 110 configured to execute instructions loaded therein, for example, to implement various aspects described herein. The processor 110 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 100 includes at least one memory 120 (e.g., volatile and / or non-volatile memory devices). The system 100 includes a storage device 140, which may include non-volatile and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. The storage device 140 may include, by way of non-limiting example, an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0010] The system 100 includes, for example, an encoder / decoder module 130 configured to process data to provide encoded or decoded video, which may include its own processor and memory. The encoder / decoder module 130 represents a module that may be included within a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. It is noted that the encoder / decoder module 130 may be implemented as a separate element of the system 100 or may be incorporated within the processor 110 as a combination of hardware and software known to those skilled in the art.

[0011] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described herein may be stored in the storage device 140 and then loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing of equations, formulas, operations, and computational logic.

[0012] In some embodiments, memory internal to the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as MPEG-2, JPEG Pleno, MPEG-1, HEVC, or VVC.

[0013] As indicated in block 105, inputs to the elements of system 100 may be provided through various input devices, including, but not limited to, (i) an RF section that receives, for example, an RF signal transmitted over the air by a broadcast station, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0014] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs these various functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and refiltering RF signals transmitted over a wired (e.g., cable) medium to a desired frequency band. Various embodiments rearrange the order of the above-mentioned (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0015] It should be noted that the USB and / or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 110, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, as desired, in a separate interface IC or in the processor 110. For example, various processing elements, including the processor 110 and the encoder / decoder 130 operating in combination with memory and storage elements, provide demodulated, error corrected, and demultiplexed streams to process the data streams for presentation on an output device, as desired.

[0016] The various elements of the system 100 may be provided within an integrated housing in which the various elements may be interconnected and transmit data between each other using suitable connection arrangements 115, such as internal buses known in the art, including an I2C bus, wiring, and printed circuit boards.

[0017] System 100 includes a communication interface 150 that enables communication with other devices over a communication channel 190. Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 190. Communication interface 150 may include, but is not limited to, a modem or a network card, and communication channel 190 may be implemented in a wired and / or wireless medium, for example.

[0018] In various embodiments, data is streamed to the system 100 using a Wi-Fi network, such as IEEE 802.11. The Wi-Fi signal in these embodiments is received by a communication channel 190 and communication interface 150 adapted for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, a set-top box delivering data through an HDMI connection of the input block 105 is used to provide the streamed data to the system 100. In yet other embodiments, an RF connection of the input block 105 is used to provide the streamed data to the system 100.

[0019] System 100 may provide output signals to various output devices, including display 165, speakers 175, and other peripheral devices 185. Other peripheral devices 185, in various examples of embodiments, include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 100. In various embodiments, control signals are communicated between system 100 and display 165, speakers 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communication protocols that allow inter-device control with or without user intervention. Output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, output devices may be connected to system 100 via communication interface 150 and using communication channel 190. Display 165 and speakers 175 may be integrated with other components of system 100 into a single unit, for example in an electronic device such as a television. In various embodiments, display interface 160 includes a display driver, for example a Timing Controller (T Con) chip.

[0020] Alternatively, display 165 and speakers 175 may be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which display 165 and speakers 175 are external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0021] It is contemplated that point cloud data may consume a large portion of network traffic, for example between cars connected via 5G networks and in immersive communications (VR / AR). Efficient representation formats are necessary for understanding and communicating point clouds. In particular, raw point cloud data needs to be properly organized and processed for world modeling and sensing purposes. Compression of raw point clouds is essential if storage and transmission of data is required in relevant scenarios.

[0022] Furthermore, point clouds may represent successive scans of the same scene containing multiple moving objects. They are called dynamic point clouds, in comparison to static point clouds captured from a static scene or static objects. Dynamic point clouds are typically organized into frames, with different frames captured at different times. Dynamic point clouds may require real-time or low latency processing and compression.

[0023] The automotive industry and autonomous vehicles are areas where point clouds can be used. Autonomous vehicles should be able to "probe" their environment and make good driving decisions based on the reality of their immediate surroundings. Common sensors like LiDAR generate (dynamic) point clouds that are used by perception engines. These point clouds are not intended to be seen by the human eye, they are usually sparse, not necessarily colored, and are highly populated and dynamic. They may have other attributes like reflectivity provided by LiDAR, since this attribute indicates the material of the object being sensed and can help to make decisions.

[0024] Virtual reality (VR) and immersive worlds are predicted by many as the future of two-dimensional flat video. With VR and immersive worlds, the viewer is immersed in the environment all around them, as opposed to standard TV, where the viewer can only see the virtual world in front of them. There are several degrees of immersion, depending on the viewer's degrees of freedom in the environment. Point clouds are a good format candidate for delivering virtual reality worlds. Point clouds used in VR can be static or dynamic, and are typically of average size, e.g., at most a few million points at a time.

[0025] Point clouds can also be used for various purposes such as cultural heritage / architecture, where objects such as statues or buildings are scanned in 3D to share the spatial configuration of the object without sending or visiting the object. Point clouds can also be used to ensure preservation of knowledge of objects when they may be destroyed, for example a temple may be destroyed by an earthquake. Such point clouds are usually static, color-coded and large.

[0026] Another use case is in topography and cartography, where using three-dimensional representations, maps are not limited to flat surfaces and can include relief. Google Maps is a good example of a three-dimensional map, but it uses meshes instead of point clouds. Nevertheless, point clouds can be the preferred data format for three-dimensional maps, and such point clouds are usually static, color-coded, and large.

[0027] For the applications considered herein, world modeling and sensing via point clouds can be a useful technique, enabling machines to gain knowledge about the three-dimensional world around them.

[0028] Point Cloud Analysis and Understanding 3D point cloud data are essentially discrete samples on the surface of an object or scene. To understand / analyze a point cloud scene composed of multiple objects, different tasks can be performed, such as segmentation, detection, and abstraction.

[0029] Point cloud semantic segmentation refers to the problem of label assignment for each individual point of a 3D point cloud with a class label. For example, in the case of an indoor point cloud, a segmentation method needs to label the points as chair, desk, sofa, etc. An advanced segmentation setting is called instance segmentation, which tries to distinguish points between object instances in a point cloud. For example, if there are two chairs, A and B, in the point cloud, an instance segmentation method needs to segment the points between chair A and chair B.

[0030] Another important point cloud understanding task is object detection. In object detection, the task is to find if an object of a given type is present in the scene. If detected, it further places a rectangular bounding box densely over the detected object in the point cloud frame. A related task is called point cloud abstraction, where besides the simple rectangular box used in detection, the algorithm uses various geometric primitives to fit objects / parts in the input point cloud. For example, the basic building blocks of the 3D world used include planes, spheres, ellipses, cones, or cylinders.

[0031] Unfortunately, to train such learning-based 3D point cloud understanding modules, a large amount of training data with ground truth labels is usually required. This requirement leads to two major drawbacks: First, collecting enough segmentation labels for practical use is costly. Second, such segmentation algorithms that operate with known labels fail when faced with unseen data in the real world. On the other hand, humans are able to perform understanding tasks in a more self-supervised manner. When observing new scenes, humans gradually learn to discover new object classes and segment them even if they have not been seen before.

[0032] This application is directed to an unsupervised method for 3D point cloud data analysis and understanding. As mentioned above, typical learning-based point cloud understanding systems require a large amount of labeled data to train the algorithm. However, obtaining sufficient labeled data is costly. In addition, such methods cannot generalize to unseen data without new training labels. To address these limitations, we propose an unsupervised 3D point cloud abstraction method that discovers similar / repeated primitive shapes from an input 3D point cloud in a self-supervised manner. Through primitive discovery, our proposal can be applied to point cloud understanding problems such as segmentation and detection.

[0033] Semantic Segmentation The PointNet work is the first semantic segmentation algorithm that operates on native point clouds. PointNet uses a combination of point-wise MLP layers and a global max pooling operator to obtain point-wise classification (segmentation) results. Subsequent works, such as PointNet++, DGCNN, and KP-Conv, extend PointNet to more complex point-based operations to count neighbors and process the input point cloud hierarchically. They provide reasonably good semantic segmentation performance not only for object-level point clouds, but also for scene point clouds that store multiple objects.

[0034] However, all these works require huge amounts of labeled data for training. If a segmentation algorithm needs to be deployed in an unseen environment, for example, we have a segmentation network trained on indoor scenes, but we need to perform segmentation on outdoor scenes, a simple solution is to directly apply the network trained on the original domain (indoor) to the new domain (outdoor). However, the mismatch between the two domains may cause a large performance degradation. Another solution is to collect new labels for the new domain, which can be very expensive, especially for large point clouds that store millions of points and a large number of objects.

[0035] To alleviate the above problems, the PointContrast work borrows the concept of transfer learning, which first pre-trains on the source dataset and then fine-tunes on a smaller set of data from the target domain (using segmentation labels). Meanwhile, some works perform direct training on a small set of labeled data in the target domain (also known as weakly supervised learning). This is possible by adding a smoothing constraint to regularize the network's behavior. However, all these methods still require ground truth labels for training. In contrast, our approach does not require any labels and automatically discovers repetitive primitive shapes from a given point cloud dataset.

[0036] Instance segmentation and object detection Point cloud instance segmentation has also been widely investigated, which requires outputting class labels for points and distinguishing between different objects. To solve this task, the SGPN (Similarity Group Proposal Network) work deploys a single network to predict both point grouping proposals and the semantic class of each proposal. Differently, PointGroup uses a bottom-up strategy, focusing on leveraging the empty space between objects. In the DyCo3D work, the authors propose a dynamic method to generate convolution kernels for voxelized point clouds.

[0037] To perform object detection on point clouds, the representative PointRCNN work presents an object detection approach on raw point clouds. That is, the method does not require the point cloud to be voxelized or projected to a bird's-eye view. Instead, it operates directly on the raw point cloud. The method consists of two steps: In the first step, a bottom-up strategy is applied to create 3D proposals. In the second step, for each point, local features are combined with global semantic features, and then box refinement is performed to predict the confidence.

[0038] It should be noted that all these existing works on instance segmentation and object detection require proper training labels to ensure their success.

[0039] point cloud abstraction Deep learning-based methods have already been explored to address the challenges in point cloud abstraction. Existing point cloud abstraction methods use a predefined set of primitives (e.g., planes, spheres, cones) to represent objects.

[0040] The two main strategies in this regard concern supervised point cloud abstraction (PCA) and unsupervised point cloud abstraction (PCA). Supervised PCA refers to a problem formulation in which the training process assumes access to ground truth information about primitives and point attributions to primitives. In contrast, unsupervised PCA assumes access only to the point cloud. Since obtaining ground truth information in addition to the raw point cloud data is very costly, unsupervised point cloud processing approaches are preferred even at the expense of some tolerable performance loss.

[0041] Our proposal also works in an unsupervised manner, but differs from the unsupervised PCA methods mentioned above: instead of using a predefined set of primitive shapes as in existing works, we directly abstract / discover / refine representative primitive shapes from a given set of point cloud data.

[0042] In particular, we propose a point cloud abstraction method, also called point cloud refinement. Given an input point cloud, our system outputs a set of primitives from the input point cloud during the inference phase using a neural network framework NNF. To achieve this goal, we propose a novel training strategy to obtain a network NNF based on the analysis-by-synthesis principle. The proposed NNF can be utilized for point cloud analysis / understanding, such as instance segmentation, object detection, etc.

[0043] For existing point cloud abstraction methods, their motivation was inspired by a statement made by Paul Cezanne in 1904: "...we treat nature with cylinders, spheres, and cones, and put everything in proper perspective..." Cezanne's insight is that objects can be thought of as constructed from a set of primitives, enumerated based on human observations of the world, and point cloud abstraction methods essentially fit point cloud objects to the known primitives.

[0044] We begin this research with a more fundamental question: "What can we do if we don't know the primitives?" Finding primitives is considered the starting point in understanding the 3D world. Primitives can refer to repeating geometric patterns. Simple primitives can be cylinders, spheres, cones, etc. Advanced primitives can be objects in our world, such as pedestrians, cars, trees, buildings, etc.

[0045] For example, if we imagine that we live in a world where everything is made up of atomic elements that have a cube-like shape, then after observing the world for a long enough time, we will have the concept of a "cubic" shape. If the world were instead made up of spherical atomic shapes, we would have the concept of a "spherical" shape.

[0046] Therefore, the goal of this work is to have the ability to discover a list of primitives by observing a sufficient number of point cloud frames. The discovered primitives can be useful for different downstream tasks (e.g., point cloud segmentation, detection, and compression) without additional training. For example, in an autonomous driving application, once an algorithm discovers some common objects / primitives (e.g., cars, pedestrians) on the road, these concepts about the driving scene can be directly used for tasks such as detection, segmentation, etc. to benefit the entire autonomous driving application.

[0047] Primitive Learning We propose an analysis-by-synthesis strategy for discovering primitives based on an autoencoder framework. The procedure is illustrated in Figure 2 according to an embodiment.

[0048] In this embodiment, the sampling module (210) takes a "snapshot" on the input point cloud by sampling on the input point cloud. It outputs a point cloud chunk, i.e., a subset of points in the input point cloud. In one implementation, the sampling module (210) can fetch a random point cloud chunk, which is a point cloud chunk placed at a random location with a random scale / size, from a random point cloud frame in the point cloud dataset. For example, to generate the jth point cloud chunk, the sampling module first fetches a random point cloud chunk at a random location P j Then, a random radius r j For point A in the point cloud, select

[0049]

number

[0050] Each "snapshot" chunk j For x, the analysis step is performed by an encoder network (220) in the autoencoder. This involves deriving a codeword CW in a feature space that describes a set of shapes. j (230). The synthesis step extracts the codeword CW j From point cloud chunk j This is performed by a decoder network (240) in the autoencoder, which reconstructs the original chunks. Then, we calculate a mismatch metric between the reconstructed point cloud chunks and the original chunks. During network training, the parameters of the encoder network (220) and the decoder network (240) are updated by minimizing the mismatch metric.

[0051] By repeating the analysis-synthesis process over the observed point cloud chunks, the autoencoder can gradually discover recurrent object shapes, i.e., primitives. In essence, such an autoencoding process becomes a primitive learning process, also called point cloud refinement. The learned feature space, i.e., the output of the encoder network (220), is expected to characterize / summarize some of the most frequently occurring object shapes.

[0052] In one embodiment, the encoder network (220) can be implemented as PointNet. Figure 3 shows an example of a detailed architecture of PointNet, where the input point cloud chunk j As an input (310), the codeword CW j It outputs (the transpose of) the output codeword CW. It consists of a set of shared MLPs (320) operating on each 3D point, followed by a global max pooling operation (330) to extract global features. It is then further processed (340) using another set of MLP layers to produce the output codeword CW j (The transpose of) is obtained.

[0053] Apart from PointNet, the encoder network can be any other existing encoder for 3D point clouds. In one embodiment, the encoder network (220) can be implemented as a PointConv operator, which approximates convolution in the continuous domain. In another embodiment, the encoder network (200) can be implemented as a KPConv operator, which takes into account neighboring points when computing codewords.

[0054] In one embodiment, the decoder network can be implemented as a FoldingNet, the detailed design of which is shown in Figure 4. It consists of two sequences of shared MLP layers. First, the replicated codeword CW j , and the two-dimensional grid are concatenated and fed into a first series of shared MLP layers (410). The first set of outputs of the MLPs and the replicated CW jare concatenated and processed by a second series of shared MLP layers (420) to obtain the final reconstructed point cloud chunk Rec j The dimension of is n×3 (where n is the number of points).

[0055] Note that the decoder network can be implemented as any other existing decoder for 3D point clouds. In one embodiment, the decoder network may be implemented as a LatentGAN decoder that simply consists of a sequence of MLP layers. In another embodiment, the decoder network can be a TearingNet that provides a reconstruction that preserves the point cloud topology.

[0056] Partitioning-Based Primitive Learning To make the primitive learning procedure more effective, in one embodiment, the sampling module is implemented by a point cloud segmentation / partitioning network (510) shown in Figure 5. The partitioning module selects a list of point cloud chunks, e.g., Chunk(s) j = 1, 2, ..., K. j It is assumed that K is a naive point cloud segmentation network that can output K. The value of K can be fixed or adaptive depending on the partitioning network. In one embodiment, the partitioning network is a learnable neural network module, which can be trained end-to-end using a subordinate autoencoder network.

[0057] In one example, the partitioning network (510) can be implemented as PointNet++, as shown in Figure 6. The partitioning network consists of a set abstraction layer (610, 620) that extracts features for a set of subsampled points. The set abstraction layer consists of Farthest Point Sampling (FPS) for subsampling, nearest neighbor search for grouping, followed by PointNet for feature extraction. The PointNet++ architecture further consists of an interpolation layer (630, 650) and a point-wise MLP layer (640, 660) to gradually extend the subsampled point set and associated features to finally obtain point-wise segmentation results. In another implementation, the partitioning network (510) may be implemented as VoteNet, a neural network architecture for 3D object detection or segmentation (see the paper "Deep Hough voting for 3D object detection in point clouds" by Charles R Qi et al., in Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 9277-9286, 2019).

[0058] If K point cloud partitions / chunks are output by the partitioning network, the autoencoder computes the K partitions for each point cloud partition Chunk. j Iterates over , and the output point cloud partitions are merged (520) to output a fully reconstructed point cloud frame. Note that if the input point cloud is partitioned into chunks, the chunk location information is sent to the merge module to fully reconstruct the point cloud. Finally, a mismatch metric between the reconstructed full point cloud and the input point cloud is calculated to supervise the neural network training process. In one embodiment, the mismatch metric can be the chamfer distance (CD). In another embodiment, it can be the earth mover distance (EMD).

[0059] Primitive Generation Using the proposed primitive learning, the feature space of the codeword CW is expected to represent the primitives to be discovered. From the primitive feature space, we further want to explicitly generate a list of primitive shapes.

[0060] As a preliminary step, we define a set of codewords,CW,for which the reconstruction quality of the relevant point cloud chunk exceeds a certain threshold. j Basically, each identified CW j The point cloud chunks in are instantiations from primitive shapes. Using this codeword set, we aim to output the primitive shapes shown in Figure 7.

[0061] First of all, the codeword CW j is a high-level description of a point cloud chunk, so different point cloud chunks sampled from the same primitive are j The codewords should form clusters in the feature space of CW. For example, different styles of chairs should have codewords that form one cluster, and different styles of cars should have codewords that form another cluster. Thus, when a set of codewords is passed through the clustering module (710), each CW j is assigned to one of a number (m) of clusters. In one embodiment, the clustering module may be a DBSCAN algorithm. In another embodiment, k-means clustering may be employed.

[0062] Then, we perform aggregation (720) to find the representative codewords for each cluster / class.

[0063]

number

[0064] Finally, to explicitly visualize the point cloud primitives, we send each representative primitive codeword to the point cloud chunk decoder (730). During primitive learning, we trained the decoder module so that, in the end, it generates the primitive shapes that we discovered.

[0065] segmentation As shown in Fig. 8, this embodiment aims to use learned primitives to aid the semantic segmentation process. Assuming there is an input point cloud, which may come from a training dataset or share similar characteristics with the training dataset, e.g., it is roughly the same type of scene as the training dataset, then this point cloud is first fed into a trained partitioning network (810) to obtain a set of point cloud chunks. Each point cloud chunk is then j is the corresponding codeword CW j The chunk codewords are then fed to a classifier (830), which classifies the previously obtained primitive codewords

[0066]

number

[0067]

number

[0068] Since each point cloud chunk output by the partitioning module is essentially an instance, our proposal can be easily applied to instance segmentation. Specifically, each point in the input is not only assigned a class label by the classifier, but also an instance label by the partitioning network. The instance label helps to distinguish between two object instances even if they belong to the same primitive class.

[0069] Object Detection In this embodiment, the learned primitives are used to perform the detection task, as shown in Figure 9. The goal of this detection pipeline is to find the target primitive (codeword

[0070]

number

[0071] Similar to the segmentation task, the input point cloud is first partitioned by a partitioning network (910) and then fed to an encoder (920) which generates a set of codewords,

[0072]

number

[0073]

number

[0074]

number

[0075]

number

[0076]

number

[0077] compression In this embodiment, we use our proposal to aid the point cloud compression (PCC) process. A real point cloud frame contains millions of points, which may be too many for a point cloud codec to encode / decode in one pass due to memory / computation limitations. Therefore, it is necessary to partition the input point cloud into several segments (or blocks) and encode / decode each partition separately. A simple approach is to uniformly divide the entire 3D space into cubes and consider each cube as a block. However, this approach puts correlated 3D points into two different blocks, which may prevent the point cloud codec from exploiting their correlation for compression. For example, an object may be divided into two parts in two different blocks, both of which the point cloud encoder does not have access to.

[0078] Separately, we propose to apply our method to partition the input point cloud frame. As illustrated in FIG. 10, on the encoder side, a trained point cloud partitioning network (1010) is applied to divide the input point cloud frame into K partitions. Then, each individual partition is fed to a PCC encoder (1020) to obtain K bitstreams. As illustrated in FIG. 11, on the decoder side, the received bitstream is first decoded by a PCC decoder (1110), and then the decoded K point cloud partitions are merged by a merge module (1120) to obtain the final decoded point cloud. In our approach, we discover and preserve the primitive structure within each point cloud partition; that is, highly correlated points are grouped together within a partition. Thus, it allows the point cloud codec to access the correlated points within each partition, which in turn improves the compression performance compared to the naive approach mentioned above.

[0079] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes and the described aspects are not limited to these specific values.

[0080] Various methods are described herein, each of which includes one or more steps or acts for achieving the described method. Unless a specific order of steps or acts is required for proper operation of the method, the order and / or use of specific steps and / or acts may be modified or combined. It is noted that terms such as "first", "second", etc. may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decode" and "second decode". The use of such terms does not imply any ordering to the modified operations, unless specifically required. Thus, in this embodiment, the first decode need not be performed before the second decode, but may occur, for example, before, during, or during a period overlapping with the second decode.

[0081] The implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The method may be implemented in, for example, an apparatus such as a processor, which generally refers to processing devices including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include, for example, communication devices such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0082] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation", as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation", as well as other variations thereof, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0083] It should be noted that the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0084] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0085] It should be noted that the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing the information or retrieving the information (e.g., from a memory). Furthermore, "receiving" typically involves in some way an operation, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0086] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first enumerated alternative (A), or the selection of only the second enumerated alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first enumerated alternative (A), or the selection of only the second enumerated alternative (B), or the selection of only the third enumerated alternative (C), or the selection of only the first and second enumerated alternatives (A and B), or the selection of only the first and third enumerated alternatives (A and C), or the selection of only the second and third enumerated alternatives (B and C), or the selection of all three alternatives (A and B and C). This can be expanded as many times as the items listed, as would be apparent to one of ordinary skill in this and related arts.

[0087] As will be apparent to one of ordinary skill in the art, implementations may generate a variety of signals formatted to carry information that may be, for example, stored or transmitted. Information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries may be, for example, analog information or digital information. The signal may be transmitted by a variety of different wired or wireless links, as is well known. The signal may be stored in a processor-readable medium.

Claims

1. 1. A method comprising: Accessing chunks of points from an input point cloud; generating, by a first neural network based module, a code word describing at least one shape in the point cloud chunk; reconstructing the point cloud chunks based on the codewords by a second neural network based module; obtaining a mismatch metric based on the reconstructed point cloud chunk and the point cloud chunk; and adjusting parameters of the first neural network based module and the second neural network based module based on the mismatch metric.

2. The method of claim 1 , wherein the point cloud chunks are portions of the input point cloud that are randomly sized and located at random positions within the input point cloud.

3. The method of claim 1 or 2, wherein the first neural network based module corresponds to PointNet.

4. The method according to any one of claims 1 to 3, wherein the second neural network based module corresponds to a FoldingNet.

5. the input point cloud is partitioned into a plurality of chunks by a third neural network based module, and the generating of code words and the reconstructing of the point cloud chunks are performed for each chunk of the plurality of chunks to obtain a plurality of code words and a plurality of reconstructed point cloud chunks; and reconstructing the input point cloud based on the plurality of reconstructed point cloud chunks, wherein the mismatch metric is obtained based on the input point cloud and the reconstructed point cloud; and adjusting parameters of the third neural network based module based on the mismatch metric.

6. The method of claim 5 , wherein the third neural network based module corresponds to PointNet++ or VoteNet.

7. The method of any one of claims 1 to 6, wherein said adjusting is performed by minimizing said mismatch metric.

8. Furthermore, identifying a set of code words from the plurality of code words; grouping the set of codewords into one or more clusters; and obtaining a representative codeword for each of said one or more clusters.

9. The method of claim 8 , wherein the set of codewords is identified based on a reconstruction quality of the plurality of reconstructed point cloud chunks.

10. Furthermore, 10. The method of claim 8 or 9, comprising using the second neural network based module to reconstruct primitives for corresponding representative codewords.

11. Furthermore, The method of any one of claims 1 to 7, further comprising performing classification on another cloud of points based on the second and third neural network based modules.

12. Furthermore, The method of any one of claims 1 to 7, further comprising performing object detection on another point cloud based on the second and third neural network based modules.

13. Furthermore, partitioning the other point cloud into another plurality of point cloud chunks based on the first neural network based module; and encoding each of the plurality of point cloud chunks into one or more bitstreams.

14. Furthermore, Decoding the one or more bitstreams to form another plurality of point cloud chunks; Reconstructing a further point cloud from said further plurality of point cloud chunks.

15. An apparatus comprising one or more processors and at least one memory coupled to said one or more processors, said one or more processors configured to execute a method according to any one of claims 1 to 14.

16. A non-transitory computer readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 14.

Images