Method and system for acquiring and processing point cloud

Abstract

The present invention relates to a method (100) and system (10) for point cloud acquisition and processing using a 3D sensor acquiring a continuous stream of 3D points spatially distributed across a scene. The method comprises the step of acquiring a continuous stream of points using a resolution scalable 3D sensor, where the continuous stream of points is partitioned in real time, each partition (2a-c) being used as an input to an encoder (3a-c) from the moment when said partition (2a-c) becomes available. The points of the continuous flow are obtained such that each subsequent partition provides an increased resolution compared to the previous partition. In the next step, the encoding of the point cloud to the feature vectors (4a-c) of each partition (2a-c) is performed using a deep learning-based point cloud encoder (3a-c), preferably a transformer architecture. Finally, a decoder (7a-c) is used to decode the feature vector (4a-c) of each partition (2a-c) into an output prediction.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to methods of point cloud acquisition and processing, in particular to methods using resolution scalable 3D sensors. BACKGROUND

[0002] 3D scene understanding is essential for many application domains such as virtual reality, autonomous driving and robotics, where timing is critical. For this, a basic building block is 3D semantic segmentation.

[0003] Deep learning has shown great potential in different domains such as compression, 6D pose estimation and semantic segmentation. State-of-the-art techniques focus on pure performance and how traditional methods can be improved. Little attention is given to practical features. One such important feature is to enable joint acquisition and processing with sensors, more specifically 3D sensors.

[0004] Most existing deep learning based point cloud processing methods employ an architecture where the first layer processes the entire input data completely [1][2][3][4][5]. Subsequently, the point cloud is down-sampled and processed again. This is repeated multiple times and is similar to most CNN architectures [1]. By doing so, the input is processed at different resolutions and the receptive field increases step by step. An important drawback of this approach is that it is time and computationally expensive. Furthermore, this regular processing paradigm requires the point cloud to be acquired completely at full resolution before its processing can start.

[0005] Modern state-of-the-art deep learning based point cloud processing methods such as PointTransformer [5] are based on transformers. Since they compute K-Nearest Neighbors (KNN) and attention scores on the full resolution input, they are very time and power consuming.

[0006] Furthermore, existing methods are not able to process additional points in the input point cloud that become available over time step by step. One example of a frame-based approach is disclosed in [6].

[0007] Recent developments in scalable 3D acquisition devices [7] enable the acquisition of point clouds whose density increases step by step over time. This resolution scalable 3D scanning device generates a low spatial resolution of the scene with minimal latency and increases the resolution of the acquired point cloud step by step over time. However, to meet performance requirements, existing techniques using such 3D scanning devices are power and computationally expensive.

[0008] Therefore, there is a need for a processing method that provides efficient 3D semantic segmentation that is efficient in time and computation.

[0009] [1] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton, “Imagenet classification with deep convolutional neural networks,” Communications of the ACM 60.6 (2017): 84-90.

[0010] [2] He, Kaiming, et al., “Deep residual learning for image recognition,” Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.

[0011] [3] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox, “U-net: Convolutional networks for biomedical image segmentation,” Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, Springer International Publishing, 2015.

[0012] [4] Vaswani, Ashish, et al., “Attention is all you need,” Advances in neural information processing systems 30 (2017).

[0013] [5] Zhao, Hengshuang, et al., “Point transformer,” Proceedings of the IEEE / CVF international conference on computer vision, 2021.

[0014] [6] Kangcheng Liu et al.: “FG-Net: Fast Large-scale LiDAR point clouds understanding network leveraging correlated feature mining and geometric-aware modelling”, Arxiv.org, Cornell University Library, 2020.

[0015] [7] van der Tempel, Ward et al., “Low power, low latency perception for XR”, Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) IV, Vol. 12449, SPIE, 2023. SUMMARY

[0016] It is an object of the present invention to provide an improved solution which alleviates the mentioned drawbacks of prior art devices. Furthermore, it is an object to provide a method for 3D acquisition and processing of point clouds using a resolution scalable 3D sensor which provides a temporally and computationally efficient semantic segmentation of point clouds.

[0017] The invention is defined by the independent claims appended hereto, embodiments being set forth in the dependent claims appended hereto, in the following description and in the drawings.

[0018] According to a first aspect of the present invention, there is provided a method for point cloud acquisition and processing using a 3D sensor acquiring a continuous stream of 3D points, said 3D points being spatially distributed over an entire scene. The method comprises the steps of: acquiring a continuous stream of points using a resolution scalable 3D sensor, wherein the continuous stream of points is partitioned in real-time, each partition being used as input to an encoder from the moment the partition becomes available, wherein the acquiring of the continuous stream of points is such that each subsequent partition provides an increased resolution compared to the previous partition; encoding a point cloud of each partition into a feature vector using a deep learning based point cloud encoder; and decoding the feature vector of each partition into an output prediction using a decoder.

[0019] A resolution scalable 3D scanning device can generate a low spatial resolution of a scene with minimal latency and increase the resolution of the acquired point cloud over time, enabling increasing the resolution over time. By using a resolution scalable 3D sensor, there can be no frame-wise acquisition output from the sensor, but rather a continuous stream of points providing increasing resolution over time. The scanning pattern of the scene read by the resolution scalable 3D sensor, which provides how the points are distributed, can be chosen such that the first points are acquired to provide spatial completeness with low resolution, without capturing a whole frame. Further points in the stream can then be acquired such that an increasingly complete capture is provided, increasing the resolution stepwise. Thus, a spatially complete representation can be obtained even with a small amount of points, and the resolution can be increased by adding additional points of the obtained stream.

[0020] The scanning pattern of the 3D sensor can be used to acquire the points in the continuous stream. The scanning pattern can be fixed, predetermined, or selected according to a particular scene. The scanning pattern can be chosen so as to provide a spatially complete coverage of the scene also in the first partition with low resolution. After the first partition, additional points can be acquired, which are distributed such that they form a next partition of the scene with higher resolution, i.e. again a spatially complete representation but with increased resolution. The points of the continuous stream can be distributed over the scene to increase the resolution of subsequent partitions. The continuous stream of points continuing according to the scanning pattern can provide a continuously increasing coverage of the scene, providing subsequent partitions with increasing resolution. The points of each subsequent partition can not overlap with the points of any previous partition.

[0021] The acquired partitions of the continuous stream can allow different and separate handling of different partitions, increasing potential parallelization. Since the point stream continuously provides an increasingly complete capture, the partitions can sequentially include increasingly more points of the frame and provide increasingly higher resolution of the subsequent partitions. This can enable an improvement in processing time with comparable quality to known methods. Also, the first partition can be processed while still capturing other partitions, which also results in a time gain. Each partition can be processed in a separate branch. By sequentially processing the partitions, the processing memory (CPU or GPU) can be reduced compared to processing the whole data stream at once. Less data can need to be handled at once. By using the invention, a process can be implemented 30-60% faster compared to a baseline process using the highest spatial resolution. This can be due to a reduction in data processing complexity and the continuous acquisition of point cloud data.

[0022] The continuous stream of points can be acquired in at least two, at least three or at least four or more partitions with subsequently increasing resolution. Each partition can acquire points of the same scene by the 3D sensor but with different resolution. The characteristics of the 3D sensor can be adapted for each subsequent partition to provide increasing resolution. Partitioning the continuous stream of points can mean that the partitions are sequentially formed with correspondingly increasing resolution.

[0023] The method can be utilized with any point cloud semantic segmentation network employing an encoder-decoder architecture. The encoder-decoder architecture can be a transformer-based architecture. One example of such an architecture is PointTransformer. The encoder can process the input point cloud and transform it into a latent representation, which can be considered as high-level features relevant to the task. The decoder can employ these high-level features to predict the segmentation class for each input point, thereby providing the output prediction. Each encoder-decoder structure or branch can be trained in an end-to-end fashion on a training dataset consisting of point clouds and their associated per-point ground truth segmentation labels. The deep learning-based encoder-decoder architecture can be trained branch by branch. When training for an additional partition, all weights from previous branches can be frozen and the loss function can be computed based on the results from the current partition only. Each branch can be trained using a predetermined number of epochs and with a predetermined batch size.

[0024] According to one embodiment, the partitions of the continuous stream can sequentially increase in size. Depending on the employed 3D sensor, the sequentially increasing partition size can result in higher resolution point cloud partitions without overlapping points between different partitions. These partitions can be sequentially acquired in time.

[0025] As a result of the size-increasing partitions, the first partition can be smaller (in terms of number of points) than the subsequent partitions. Therefore, the earlier partition(s) can be processed much faster and with very low latency, enabling very low latency and early decisions. The difference in resolution between different partitions can increase the diversity of the input data, thereby improving the performance of the processing. It can allow for more flexible resource management, where smaller partitions can be processed by fast but small computing resources, while larger partitions can be processed by larger computing resources.

[0026] According to one embodiment, different encoder-decoder structures can be employed in parallel branches, each branch processing a different partition. For each partition processed, a separate encoder-decoder structure can be used. In other words, each partition can be processed in a separate encoder and decoder, in a separate branch. This can allow for progressive encoding, where each partition can be processed individually, allowing for greater parallelization of processing, faster processing, easier distribution of processing across multiple computing resources. By performing the branches sequentially, a substantial reduction in power consumption can be achieved.

[0027] According to one embodiment, the encoder of each branch can be performed immediately after the partition is obtained from the continuous stream. Thus, the encoders from different branches can be performed in an asynchronous manner. This can eliminate the need to wait until a dense point cloud is obtained. Once a first coarse representation of the scene is captured in a first partition, it can be processed. This can allow for joint point cloud acquisition and processing and starting processing with acquisition time, providing faster and time-efficient processing. It can also result in a more power-efficient solution, as not all encoders have to be started at the same time, but rather spread out over time.

[0028] According to one embodiment, the feature vector of the second partition obtained by the encoder with a second resolution can be fused with the feature vector from the first partition with a first resolution, the first resolution being lower than the second resolution.

[0029] The feature vectors resulting from the encoding of different resolution point clouds from different partitions in different branches can be fused into one multi-resolution feature vector. This can provide performance improvements without eliminating advantages related to, for example, time efficiency. This embodiment can be power and time efficient, as the fusion occurs on feature vectors with reduced dimensionality compared to the input to the branches.

[0030] Each subsequent partition can have an increased resolution of the point cloud obtained in the continuous stream. The first partition in the embodiment can be the partition with the lowest resolution, or a subsequent partition with an increased resolution. The feature vector of the first partition can be the feature vector output from the encoder of the first partition in the first branch, or a multi-resolution feature vector resulting from a previous fusion between two or more feature vectors.

[0031] According to one embodiment, the feature vector of the third partition obtained by the encoder with a third resolution can be fused with the multi-resolution feature vector from the second partition, the third resolution being higher than the second resolution.

[0032] According to one embodiment, the feature vector of the third partition can also be fused with the feature vector of the first partition, wherein the feature vector of the first partition and the multi-resolution feature vector of the second partition can be combined using concatenation before being fused with the feature vector from the encoder of the third partition.

[0033] The third branch can process a third sequential resolution, i.e. a third partition, of the continuous stream. For the third partition and each subsequent partition having a sequentially higher resolution, the feature vector obtained from the encoder of the third or higher branch can be fused with both the multi-resolution feature vector of the previous branch and the (multi-resolution) feature vector of the branch before that. In other words, the feature vector of partition n can be fused with the multi-resolution feature vector of partition n-1 and the (multi-resolution) feature vector of partition n-2. Before fusing these three feature vectors, the multi-resolution feature vector of partition n-1 and the (multi-resolution) feature vector of partition n-2 can be combined using concatenation. In one embodiment, the feature vector of partition n can be fused with all previous feature vectors of partitions n-1, n-2, n-3,... 1. For all previous partitions for which a multi-resolution feature vector has been encoded, such multi-resolution feature vector can be used for fusing with the feature vector of partition n. The n-th partition can have an n-th resolution which is higher than the resolution of each previous partition. In one embodiment, the n-th feature vector can be fused with the feature vector of the first partition and each multi-resolution feature vector of all the remaining previous partitions.

[0034] According to one embodiment, the feature vectors can be fused based on a K-Nearest Neighbor (KNN) algorithm. Each feature of a partition can be fused with features of earlier partitions of the continuous stream that are associated to the K nearest points. The K nearest points can be computed on points that are associated to each feature having a feature vector of reduced dimension compared to the input partition. The temporal impact can thus be limited. This can further provide performance improvements and be power efficient as it uses previously computed feature vectors from other partitions, thereby reducing the complexity required by the encoder.

[0035] According to one embodiment, the input of the second or third decoder can be the fused second or third multi-resolution feature vector, respectively.

[0036] In each branch for which the feature vectors have been fused into a multi-resolution feature vector, the decoder of such branch can receive the multi-resolution feature vector for decoding to generate an output prediction. The decoder can thus decode the multi-resolution feature vector. This can improve performance as the decoder can receive a larger amount of information (fused from different resolutions) compared to the output feature vector of the encoder, without negatively impacting the time.

[0037] According to one embodiment, the decoders can be executed immediately upon computing the fused multi-resolution feature vector.

[0038] Thus, the decoders in different branches can be executed in an asynchronous manner. This can eliminate the need to wait until a dense point cloud is obtained and processed, which is typical for conventional processing paradigms. Once a first coarse representation of a scene is captured in a first partition and encoded as a feature vector, the fused feature vector of a second partition can be decoded. The obtained fused feature vector of the second partition can in turn allow the decoding of the fused feature vector of a third partition. This can continue until the obtained fused feature vector of the (n-1)th partition in turn allows the decoding of the fused feature vector of the nth partition. It can also allow joint point cloud acquisition and processing and starting processing with acquisition time and provide faster and time-efficient processing. Since not all decoders have to be started at the same time, but can be spread out over time, a more power-efficient solution can be achieved.

[0039] According to one embodiment, the outputs of the decoders of the branches are obtained at different time instants. This can improve joint point cloud acquisition and deep learning based encoder processing. The need to wait until a dense point cloud is obtained can be eliminated, further enabling early decisions since first results are obtained very quickly. In addition, it gives flexibility in terms of required confidence and the computation of subsequent partitions can be interrupted when the required confidence is reached. First processing results can be obtained while the acquisition process is ongoing.

[0040] According to one embodiment, the outputs of the decoders of the branches can be refined by an update step that conveys information from the outputs of later branches.

[0041] The outputs of each branch can thus be improved using information from later branches. An increase in performance can be achieved while removing the need to recompute the outputs of earlier branches, since they can be updated by a fast and simple refinement step. However, this update will impact the power and time consumption at a low level. In one embodiment, the update of the prediction of the output of a previous branch can be provided by replacing the previous predicted output with the majority vote result in the group of nearest neighbors of the later prediction. In another embodiment, the update can be achieved by combining the predicted output of a previous branch with the predicted outputs of one or more later branches. Thus, the predicted output of a previous branch can be improved based on the predicted outputs of later branches.

[0042] According to a second aspect of the invention, there is provided a computer program product configured to perform the method according to any one of the embodiments herein when executed on a processing device.

[0043] According to a third aspect of the application, a readable computer medium comprising the computer program product is provided.

[0044] The processing unit can comprise one or more computer units, data processing units and / or general purpose processors configured to perform the steps of the method according to any one of the embodiments herein. The processing unit can comprise a computer program product or software configured to perform the steps of the method when executed. The processing unit can comprise or be communicatively connected to a computer readable storage medium comprising the computer program product or software.

[0045] According to a fourth aspect of the application, a system for point cloud acquisition and processing is provided, comprising: a resolution scalable 3D sensor configured to acquire a continuous stream of 3D points, the 3D points being spatially distributed over an entire scene; a partitioning unit configured to partition the continuous stream of points in real-time, each partition being used as input to an encoder from the moment the partition becomes available, wherein the continuous stream of points increases resolution such that each subsequent partition provides an increased resolution compared to a preceding partition; a deep learning based point cloud encoder for each partition, each encoder being configured to encode the point cloud of the partition into a feature vector; and a decoder for each partition configured to decode the feature vector of the partition into an output prediction.

[0046] The system can be configured to perform the method according to any one of the embodiments described above. The system can further comprise at least one fusion block configured to operate according to any one of the embodiments of the fusion block described herein. The system can comprise a processing unit configured to operate according to one or more of the entities of the disclosed system. BRIEF DESCRIPTION OF DRAWINGS

[0047] The application will be described in more detail in the following with reference to the enclosed drawings, in which:

[0048] Figure 1 A schematic block diagram of a process according to an embodiment of the application is shown.

[0049] Figure 2 A schematic block diagram of a fusion block according to an embodiment of the application is shown.

[0050] Figure 3 A performance result comparison plot for an embodiment of the application is shown.

[0051] Figure 4 A timing result comparison plot for an embodiment of the application is shown.

[0052] Figure 5 A visual result for an embodiment of the application is shown.

[0053] Figure 6aA visualization of the scan pattern of a resolution-scalable 3D sensor is shown that can be used in embodiments of the present application.

[0054] Figure 6b A visualization of the distribution of different partition centers is shown according to embodiments of the present application.

[0055] Figure 7 A visualization of the scan pattern of a resolution-scalable 3D sensor is shown that can be used in embodiments of the present application.

[0056] Figure 8 A visualization of the scan pattern of a resolution-scalable 3D sensor is shown that can be used in embodiments of the present application.

[0057] Figure 9 A flowchart of a method according to embodiments of the present application is shown. DETAILED DESCRIPTION

[0058] The present application will be described hereinafter more fully with reference to the accompanying drawings, in which preferred embodiments of the present application are shown. The present application may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the application to those skilled in the art. In the drawings, like numbers refer to like elements throughout.

[0059] Figure 1 A processing pipeline of a system 10 implementing a method according to embodiments of the present application is illustrated.

[0060] A scene 1 is being captured by a resolution-scalable 3D acquisition device. The resulting continuous stream of points is partitioned 2 in real-time into successive and non-overlapping point cloud partitions 2a-c of increasing size and thus resolution. The point cloud partitions 2a-c are encoded by deep learning based point cloud encoders 3a-c in parallel branches A, B, C. Each branch A-C comprises an encoder 3a-c. The encoders 3a-c are for example transformer based. The encoders 3a-c for the different partitions 2a-c are executed in an asynchronous manner, whereby an encoder 3a-c is executed as soon as a partition 2a-c is available. While the encoders of a partition are executed, the point cloud of the next partition is being acquired. The encoders 3a-c encode each input point cloud partition 2a-c in a feature vector 4a-c at that particular resolution.

[0061] The encoder-decoder architecture can be PointNet++ based [7, 11], point convolution based [8-10] or transformer based [5, 12-13]. Thus, the skilled person understands that other architectures can also be suitable.

[0062] [7] Qi, Charles Ruizhongtai et al., “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” Advances in neural information processing systems 30 (2017).

[0063] [8] Wang, Yue et al., “Dynamic graph cnn for learning on point clouds,” ACM Transactions on Graphics (tog) 38.5 (2019): 1-12.

[0064] [9] Thomas, Hugues et al., “Kpconv: Flexible and deformable convolution for point clouds,” Proceedings of the IEEE / CVF international conference on computer vision, 2019.

[0065]

[10] Zhao, Hengshuang et al., “Pointweb: Enhancing local neighborhood features for point cloud processing,” Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019.

[0066]

[11] Qian, Guocheng et al., “Pointnext: Revisiting pointnet++ with improved training and scaling strategies,” Advances in Neural Information Processing Systems 35 (2022): 23192-23204.

[0067]

[12] Wu, Xiaoyang et al., “Point transformer v2: Grouped vector attention and partition-based pooling”, Advances in Neural Information Processing Systems 35 (2022): 33330-33342.

[0068]

[13] Lai, Xin et al., “Stratified transformer for 3d point cloud segmentation”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022.

[0069] The multi-resolution feature vector fusion block 5b-c is configured to fuse the obtained feature vector 4b-c of a particular partition with the feature vector 4a-c from a previous partition. To this end, it can employ a K-Nearest Neighbors (KNN) technique, where features from nearby points are employed in the fusion process. The resulting multi-resolution feature vector 6b-c contains information from different partitions and thus from different resolutions.

[0070] More specifically, in one embodiment, the second branch B comprises a fusion block 5b configured to fuse the feature vector 4a from the first partition 2a with the feature vector 4b of the second partition 2b, resulting in a multi-resolution feature vector 6b. Additionally, the third branch C can comprise a fusion block 5c configured to fuse the feature vector 4c of the third partition 2c with the multi-resolution feature vector 6b resulting from the fusion process in the second branch B. In one embodiment, the multi-resolution feature vector 6b is combined with the feature vector 4a of the first partition 2a using concatenation before being fused with the feature vector 4c of the third partition 2c. This process is repeated for any number n of resolution layers. In the nth branch, the fusion process can be performed between the feature vector of the nth branch and the encoded feature vector and / or multi-resolution feature vector of each of the previous branches n-1, n-2, n-3,... 1. The feature vector of the nth branch can be fused with the feature vector of the first branch and the multi-resolution feature vector of each of the remaining previous branches.

[0071] The feature vectors (or feature vectors 4a or multi-resolution feature vectors 6b-c) are decoded using a point cloud decoder 7a-c based on deep learning and output a prediction 8a-c for the partitions 2a-c in branches A-C. The decoders 7a-c can use a transformer architecture such as Point Transformer. The decoders 7a-c can be executed in an asynchronous manner, whereby the decoders 7a-c are executed as soon as the feature vectors 4a, 6b-c are available. The resulting predictions 8 are obtained asynchronously. For example, the prediction 8a on the first partition 2a is obtained while the acquisition and processing of the other partitions 2b-c is still ongoing. The process is similarly generalized for any number n of branches, each branch comprising a decoder that decodes an output prediction based on the input partition and the feature vectors of the previous branches.

[0072] In one embodiment, a progressive update step 9 can be applied. Given the later, higher quality predictions that incorporate more information, the update process iterates back and improves the quality of the previous prediction outputs 8a-c.

[0073] In more detail, the partitions 2a-c can be mathematically represented as where and s is the number of partitions and N is the total number of points in the final dataset, is the sum of points of all partitions. In the example shown, the first partition 2a comprises points , the second partition samples points and the third partition samples points The partitions are sequential along time t.

[0074] The first partition 2a, also denoted as Xi, is processed by the first encoder 3a and decoder 7a, resulting in a prediction output 8a, mathematically represented as . Due to the number of points (N is the number of points in the final dataset), the first partition with the first lowest resolution has low computational complexity and a fast prediction can be obtained.

[0075] The second partition 2b, partitioned as X2, is processed by a separate encoder 3b to provide a feature vector 4b as where is the number of (downsampled) points, F features in the feature vector 4b are computed for each of them. To improve performance, the features of the feature vector 4b are fused by a fusion module 5b with the features of the feature vector 4a of the previous partition 2a. The resulting multi-resolution features of the multi-resolution feature vector 6b are taken by the decoder 7b to obtain an output prediction 8b, also denoted as . Once all s partitions have been processed, the is obtained.

[0076] In terms of computational complexity, the invention provides an advantage over methods that process all sampled data at once. When using an attention mechanism that requires computation of K-Nearest Neighbors (KNN), the complexity of processing the sampled input as a whole can be expressed as where , in the case of s partitions with increasing resolution. In the invention, the attention mechanism can be applied in parallel on partitions 2a-c, thus providing a complexity of Compared to non-scalable methods, the invention greatly reduces the complexity by a factor proportional to

[0077] As the number of partitions s is larger, the effect of reducing complexity becomes more pronounced as the partition size becomes smaller. In addition, the sequential processing of the partitions also provides computational redundancy, although KNN is the most computationally expensive operation. For large N and a large number of partitions, the effect of double product elimination becomes significant.

[0078] To further describe the fusion block, may be the stitched features from lower resolution partitions, where is the number of points of the stitch in the feature space. Given and , the fusion block can combine the multi-resolution information into a single feature vector for decoding.

[0079] The deep learning based encoder-decoder structure can be trained on a partition by partition basis. When training for an additional partition, all weights from previous partitions can be frozen and the loss function can be computed based on the results from the current partition only. This can allow the backbone architecture to maximize results for each resolution.

[0080] Figure 2 A multi-resolution feature vector fusion block 5 is illustrated. In general, the fusion block 5 retrieves relevant features from previous partitions and fuses them with the features of the current partition. The fusion block 5 includes a Query And Group (QAG) block 10 configured to retrieve the feature vectors 4 and / or multi-resolution feature vectors 6 from previous partitions that are most relevant to the features of the current partition. More specifically, the QAG block can employ a KNN algorithm to retrieve the features of the K nearest points that belong to the points associated to the features of the current partition. Since these features originate from partitions of different resolutions, the retrieved feature vectors contain multi-resolution information.

[0081] ​​​The multi-resolution feature vector fusion block 5 also comprises a deep learning based processing block 11 configured to encode the K-nearest neighbors of these features into a single feature vector. The processing block 11 can be configured to process the K-nearest neighbors by a Conv1D procedure followed by a MaxPool layer.

[0082] The multi-resolution feature vector fusion block 5 is also configured to concatenate the features of the current partition with the resulting feature vector from the processing block 11, which is processed into a single multi-resolution feature vector 6 for the partition, combining the multi-resolution information.

[0083] Figure 3 Performance results of the disclosed method are illustrated. On the x-axis 13, the number of points is displayed, where N i is the number of points comprised in the i-th partition. The performance of the semantic segmentation methods under study is compared in terms of mean intersection over union (mloU) 14. As a baseline 15, the backbone semantic segmentation network has been used to process the points in a non-progressive way. Thus, compared to the described method, the baseline method deals with the complete point cloud at each partition, requiring all data to be available before execution. It does not work on different partitions, lacks the ability to process data step by step and does not generate any intermediate results. The results of the described method are represented as 16. Since this method works in a progressive approach, the points are interrelated and dependent on each other. Label 17 indicates the results according to the method according to the embodiment without the update step 9. Label 18 indicates the results according to the method according to the embodiment without the multi-resolution fusion 5 and the update step 9.

[0084] Figure 4 Timing results during inference are illustrated. On the y-axis, the required inference time 19 is provided. Label 20 indicates the upper bound of the inference time. This is the case where data acquisition is instantaneous and thus all latencies are generated by the processing. Label 21 indicates the lower bound of the inference time. This is the case where the processing is faster than the acquisition of the different partitions. The area 22 is where all operations take place, bounded by the upper and lower bounds. It can be noted that even operating on the upper bound, the present invention can achieve a reduction of 31% of the inference time with respect to the baseline on the highest partition. On the lower bound, the reduction can be 61%. As discussed above, the reduction of the inference time can be achieved mainly due to the reduced complexity.

[0085] Figure 5 Visual results of the described method on a scene are illustrated, including the results on one of the lowest partitions 23, the results on the intermediate partitions 24 and the results on the later partitions 25.

[0086] Figure 6aFig. 6 illustrates an example of a scan pattern that can be used in a 3D sensor for the distribution of points of a scene in a continuous stream of points. For an exemplary description of the scan pattern, the upper left graph of Fig. 6 can visualize the scan pattern for acquiring points in a first partition. A spatial complete coverage of the scene can be achieved, but with a very low resolution. Next, in the upper right graph, the scan pattern continues to cover more points to be acquired, enabling the selection of a second partition formed by the acquired additional points and displayed with an increased resolution. The lower left and right graphs illustrate further distributions when more points are acquired, and provide an increased resolution for these subsequent partitions. Over time, the number of points distributed over the scene can increase, providing an increased resolution for subsequent partitions.

[0087] Figure 6b Fig. 7 illustrates how the scan pattern in Fig. 6 can be used to form partitions, where there is no overlap between the points covered in a subsequent partition and the points covered in a previous partition. It also illustrates that each subsequent partition includes points providing an increased resolution compared to a previous partition. In each subsequent partition, an increased number of points can be included compared to a previous partition. The distribution of points in each partition can follow the scan pattern used by the resolution scalable 3D sensor. Figure 6a

[0088] Figure 7 Fig. 8 illustrates another example of a scan pattern that similarly can provide an increased resolution in subsequent partitions over time.

[0089] Figure 8 Fig. 9 illustrates the visual result of the method on a scene. Figure 8 a illustrates the acquired points, the semantic labels, and the underlying 3D scene from which the points are acquired. Figure 8 b illustrates the result of a first partition, where points are acquired in a distribution pattern. Figure 8 c illustrates the result of a first subsequent partition, with additional points acquired at a higher resolution compared to the first partition. Figure 8 d illustrates the result of a second subsequent partition, with points acquired at a higher resolution compared to the previous partition.

[0090] Figure 9 ​A method 100 according to an embodiment of the application is illustrated. The method 100 comprises a step of acquiring 102 a continuous stream of 3D points of a scene. The continuous stream is acquired using a resolution-scalable 3D sensor. The continuous stream is partitioned 104 in real-time. Each partition is used as input to an encoder 3. For each partition, the continuous stream is acquired at a sequentially increasing resolution. Each subsequent partition thereby provides an increased resolution compared to the previous partition. The method further comprises a step of encoding 106 the point cloud of each partition into a feature vector 4 using a deep learning based point cloud encoder 3. In addition, the method 100 comprises a step of decoding 108 the feature vector 4 of each partition into an output prediction 8.

[0091] In one embodiment, the method 100 can further comprise a step of fusing 107 the feature vector 4 of the second partition with the feature vector 4 of the first partition into a multi-resolution feature vector 6. The first partition can have a first resolution and the second partition can have a second resolution, the first resolution being lower than the second resolution.

[0092] In the drawings and specification, there have been disclosed preferred embodiments and examples of the application, although specific terms are employed, but they are used in a generic and descriptive sense only and not for purposes of limitation, as the scope of the application in therewith set forth in the following claims.

Claims

1. A method for point cloud acquisition and processing using a 3D sensor, wherein the 3D sensor acquires a continuous stream of 3D points, the 3D points being spatially distributed across the entire scene, the method comprising the following steps: a) A continuous stream of points is acquired using a resolution-scalable 3D sensor, wherein the continuous stream of points is partitioned in real time, and each partition (2a-c) is used as the input of an encoder (3a-c) from the moment the partition becomes available, wherein the points of the continuous stream are acquired such that each subsequent partition (2a-c) provides an increased resolution compared to the previous partition; b) Encode the point cloud of each partition into a feature vector using a deep learning-based point cloud encoder (3) (4a-c); c) Use the decoder (7a-c) to decode the feature vector of each partition (2a-c) into the output prediction (8a-c).

2. The method according to claim 1, wherein the partitions (2a-c) of the continuous flow increase in size sequentially.

3. The method according to claim 1 or 2, wherein different encoder-decoder structures are employed in the parallel branches (A, B, C), and each branch processes a different partition (2a-c).

4. The method of claim 3, wherein the encoder (3a-c) of each branch (A, B, C) is executed immediately after obtaining the partition (2a-c) from the continuous stream.

5. The method according to any one of claims 1-4, wherein the feature vector (4b) of the second partition (2b) with a second resolution obtained by the encoder (3b) is fused with the feature vector (4a) of the first partition (2a) with a first resolution to form a multi-resolution feature vector (6b), the first resolution being lower than the second resolution.

6. The method of claim 5, wherein the feature vector of the nth partition with the nth resolution obtained by the encoder is fused with the feature vectors of all previous partitions and / or multi-resolution feature vectors, the nth resolution being higher than the previous resolutions.

7. The method according to claim 6, wherein the feature vector (4c) from the third partition (2c) is further fused with the feature vector (4a) from the first partition (2a), wherein the feature vector (4a) from the first partition (2a) and the multi-resolution feature vector (6b) from the second partition (2b) are combined by splicing before being fused with the feature vector (4c) from the encoder (3c) of the third partition (2c).

8. The method according to any one of claims 5-7, wherein the features The vectors are fused based on the K-Nearest Neighbor (KNN) algorithm.

9. The method according to any one of claims 5-8, wherein the inputs to the second decoder or the third decoder (7b-c) are the fused second multi-resolution feature vector or the third multi-resolution feature vector (6b-c), respectively.

10. The method of claim 9, wherein the decoder (7b-c) is executed immediately after calculating the fused multi-resolution feature vector (6b-c).

11. The method according to any one of the preceding claims, wherein the output of the partitioned decoder (7a-c) is obtained at different times.

12. The method according to any one of the preceding claims, wherein the output (8a-c) of the decoder (7a-c) of the branch is refined by an update step (9) that transmits information from the output (8b-c) of the subsequent partition.

13. A computer program product configured to perform the method according to any one of the preceding claims when executed on a processing device.

14. A readable computer medium comprising the computer program product according to claim 12.

15. A system (10) for point cloud acquisition and processing, comprising: A resolution-scalable 3D sensor configured to acquire a continuous stream of 3D points spatially distributed across the entire scene. A partitioning unit is configured to partition a continuous stream of points in real time, each partition (2a-c) being used as input to an encoder (3a-c) from the moment the partition becomes available, wherein points of the continuous stream are acquired such that each subsequent partition (2a-c) provides an increased resolution compared to the previous partition; For each partition (2a-c), a deep learning-based point cloud encoder (3a-c) is provided, each encoder being configured to encode the point cloud of the partition into a feature vector (4a-c); and For each partition (2a-c), the decoder (7a-c) is configured to decode the feature vector (4a-c) of the partition (2a-c) into an output prediction (8a-c).

Images