Method and system for acquiring and processing point clouds.

Abstract

The present invention relates to a method (100) and system (10) for acquiring and processing a point cloud using a 3D sensor that acquires a continuous stream of 3D points, wherein the 3D points are spatially distributed across the entire scene. The method includes the step of acquiring a continuous stream of points using a resolution-scalable 3D sensor, wherein the continuous stream of points is divided in real time, and each division (2a-2c) is used as input to an encoder (3a-3c) as soon as the division (2a-2c) becomes available. The continuous stream of points is acquired such that each subsequent division has an increasing resolution compared to the preceding division. In the next step, a step is performed in which the point cloud of each division (2a-2c) is encoded into feature vectors (4a-4c) using a deep learning-based point cloud encoder (3a-3c), preferably a transformer architecture. Finally, a step is performed in which the feature vectors (4a-4c) of each division (2a-2c) are decoded into output predictions using a decoder (7a-7c).

Description

[Technical Field]

[0001] This disclosure relates to a method and system for acquiring and processing point clouds, and more particularly to a method using a resolution-scalable 3D sensor. [Background technology]

[0002] Understanding 3D scenes is crucial in many application areas where timing is critical, such as virtual reality, autonomous driving, and robotics. Therefore, the fundamental component is 3D semantic segmentation.

[0003] Deep learning has shown great potential in various fields such as compression, 6D pose estimation, and semantic segmentation. Currently known techniques focus on pure performance and how they can improve upon conventional methods. Less attention has been paid to practical features. One such important feature is the ability to enable joint acquisition and processing using sensors, more specifically 3D sensors.

[0004] Most existing deep learning-based point cloud processing methods employ an architecture in which the first layer processes the entire input data in one go [1][2][3][4][5]. The point cloud is then subsampled and processed again. This process is repeated multiple times, similar to most CNN architectures [1]. In this way, the input is processed at different resolutions and the receptive field is progressively expanded. However, a major drawback of this approach is its time and computational cost. Furthermore, this conventional processing paradigm requires that the point cloud be fully acquired at full resolution before processing can begin.

[0005] State-of-the-art deep learning-based point cloud processing methods, such as PointTransformer[5], are transformer-based. These are very time-consuming and power-intensive because they compute K-nearest neighbors (KNN) and attention scores for full-resolution inputs.

[0006] Furthermore, existing methods cannot process additional points in the input point cloud that become available over time in a stepwise manner. For example, a frame-based method is disclosed in [6].

[0007] Recent advancements in scalable 3D acquisition systems [7] have made it possible to acquire point clouds with progressively increasing density over time. Such resolution-scalable 3D scanners generate low spatial resolution scenes with extremely low latency and progressively increase the resolution of the acquired point cloud over time. However, existing technologies using such 3D scanners require significant power and computational costs to meet performance requirements.

[0008] Therefore, there is a need for a processing method that provides 3D semantic segmentation with excellent time and computational efficiency. [1] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “Imagenet classification with deep convolutional neural networks.” Communications of the ACM 60.6 (2017): 84-90. [2] He, Kaiming, et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. [3] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. “U-net: Convolutional networks for biomedical image segmentation.” Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18.Springer International Publishing, 2015. [4] Vaswani, Ashish, et al. “Attention is all you need.” Advances in neural information processing systems 30 (2017). [5] Zhao, Hengshuang, et al. “Point transformer.” Proceedings of the IEEE / CVF international conference on computer vision. 2021. [6] Kangcheng Liu et al: “FG-Net: Fast Large-scale LiDAR point clouds understanding network leveraging correlated feature mining and geometric-aware modelling”, Arxiv.org, Cornell University Library, 2020. [7] van der Tempel, Ward, et al. “Low power, low latency perception for XR.” Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) IV. Vol.12449. SPIE, 2023. [Overview of the project]

[0009] The object of the present invention is to provide an improved solution that mitigates the aforementioned drawbacks of conventional devices. Furthermore, the object of the present invention is to provide a method for 3D acquisition and processing of point clouds using a resolution-scalable 3D sensor, the method providing semantic segmentation of point clouds that is excellent in terms of time efficiency and computational efficiency.

[0010] The present invention is defined by the appended independent claims, and embodiments are shown in the appended dependent claims, the following description and drawings.

[0011] According to a first aspect of the present invention, a method for acquiring and processing a point cloud using a 3D sensor that acquires a continuous stream of 3D points spatially distributed across an entire scene is provided. The method includes the step of acquiring a continuous stream of points using a resolution-scalable 3D sensor. The continuous stream of points is divided in real time. Each division is used as input to an encoder from the time it becomes available. Points in the continuous stream are acquired such that each subsequent division has an increased resolution compared to the preceding division. The method includes the step of encoding the point cloud of each division into feature vectors using a deep learning-based point cloud encoder. The method includes the step of decoding the feature vectors of each division into output predictions using a decoder.

[0012] Resolution-scalable 3D scanners can generate scenes with extremely low spatial resolution and gradually increase the resolution of the acquired point cloud over time, thereby enabling resolution to increase over time. Using a resolution-scalable 3D sensor, instead of obtaining frame-by-frame acquisition output from the sensor, a continuous stream of points with progressively increasing resolution over time may be provided. The scanning pattern (defining the distribution of points) of the scene read by the resolution-scalable 3D sensor may be selected so that the first point is acquired to provide spatial completeness at low resolution without capturing the entire frame. Subsequent points in the stream can then be acquired to provide more complete capture, thereby progressively increasing the resolution. Therefore, a spatially complete representation can be obtained even with a small number of points, and the resolution can be increased by adding points to the acquired stream.

[0013] Points in a continuous stream can be acquired using a scanning pattern of a 3D sensor. The scanning pattern may be fixed, predetermined, or selected depending on the specific scene. The scanning pattern may be selected to provide spatially complete coverage across the entire scene at a low resolution even in the first division. After the first division, additional points may be acquired, distributed to form the next division with a higher resolution of the scene, i.e., increased resolution while maintaining spatially complete representation. Points in the continuous stream may be distributed across the entire scene to improve the resolution in subsequent divisions. A continuous stream of points that continues to be distributed according to the scanning pattern continuously increases the coverage of the scene and provides increased resolution in subsequent divisions. Points in each subsequent division do not necessarily overlap with points in any preceding division.

[0014] By splitting the acquired continuous stream, different splits can be handled individually and in different ways, thereby increasing the potential for parallelization. As the stream of points continuously provides more complete capture, the splits sequentially include more points in the frame, and subsequent splits provide higher resolution. This allows for improvements in processing time at a comparable quality to known methods. Furthermore, the first split can be processed while other splits are still capturing data, which also contributes to the time gain. Each split can be processed in a separate branch. Processing splits sequentially reduces processing memory (CPU or GPU) compared to processing the entire data stream at once. The amount of data that needs to be handled at one time is reduced. By using this invention, processing can be achieved 30-60% faster than baseline processing when using the highest spatial resolution. This may be due to the reduced complexity of data processing and the sequential acquisition of point cloud data.

[0015] Points in a continuous stream may be acquired by at least two, at least three, or at least four or more divisions, each with a successively increasing resolution. Each division acquires points from the same scene by the 3D sensor, but has a different resolution. The characteristics of the 3D sensor may be adapted so that the resolution increases with each subsequent division. Division of a continuous stream of points may mean that the divisions are formed sequentially so that the resolution increases with each division.

[0016] This method may be used in any point cloud semantic segmentation network using an encoder-decoder architecture. The encoder-decoder architecture may be a transformer-based architecture. An example of such an architecture is a point transformer. The encoder may process the input point cloud and transform it into latent representations that can be considered higher-order features relevant to the task. The decoder may use these higher-order features to predict a segmentation class for each input point and provide an output prediction. Each encoder-decoder structure or branch may be trained end-to-end with a training dataset consisting of the point cloud and its corresponding point-by-point ground-truth segmentation labels. Deep learning-based encoder-decoder architectures may be trained branch by branch. When training additional partitions, all weights from previous branches may be frozen, and the loss function may be calculated only for the results from the current partition. Each branch may be trained using a predetermined number of epochs and a predetermined batch size.

[0017] According to one embodiment, the division of the continuous stream may be progressively increased in size. Depending on the 3D sensor used, progressively increasing the division size can result in higher-resolution point cloud division without points overlapping between different divisions. These divisions may be acquired in chronological order.

[0018] As the partition size increases, the first partition can be smaller (in terms of the number of points) than subsequent partitions. Therefore, earlier partitions can be processed very quickly and with extremely low latency, enabling early decision-making with extremely low latency. The difference in resolution between different partitions increases the diversity of the input data, improving processing performance. This allows for more flexible resource management, where smaller partitions can be processed quickly with smaller computing resources, while larger partitions can be processed with larger computing resources.

[0019] According to one embodiment, different encoder-decoder structures are used in parallel branches, and each branch processes a different partition. A separate encoder-decoder structure may be used for each partition to be processed. That is, each partition may be processed by a separate encoder and decoder in a separate branch. This enables stepwise encoding in which each partition is processed individually, facilitating improved parallelization of processing, higher processing speed, and easy distribution of processing to a plurality of computing resources. Significant reduction in power consumption can be achieved by executing branches sequentially.

[0020] According to one embodiment, the encoder of each branch may be executed immediately after the partition from the continuous stream is obtained. Therefore, encoders of different branches may be executed asynchronously. This eliminates the need to wait until a dense point cloud is obtained. As soon as an initial coarse representation of a scene is obtained in a first partition, said representation can be processed. This enables simultaneous execution of point cloud acquisition and processing, allows processing to be started using the acquisition time, and achieves faster and more time-efficient processing. In addition, since it is not necessary to activate all encoders at the same time and they can be distributed over time, a more power-efficient solution can be obtained.

[0021] According to one embodiment, the feature vector obtained by an encoder for a second partition having a second resolution may be fused with a feature vector from a first partition having a first resolution lower than the second resolution to form a multi-resolution feature vector.

[0022] Feature vectors obtained by encoding point clouds of different resolutions, that is, different partitions in different branches, may be fused into one multi-resolution feature vector. This achieves improved performance without losing, for example, the advantage related to time efficiency. Embodiments are excellent in power and time efficiency because the fusion is performed on feature vectors whose dimensions are reduced compared to the input to the branches.

[0023] Each subsequent split may have an increased resolution of the point cloud obtained in the continuous stream. The first split in the present embodiment may be a split having the lowest resolution, or may be a subsequent split with increased resolution. The feature vector of the first split may be a feature vector output from the encoder of the first split in the first branch, or may be a multi-resolution feature vector obtained by previous fusion between two or more feature vectors.

[0024] According to one embodiment, a feature vector obtained by an encoder for a third split having a third resolution higher than the second resolution may be fused with the multi-resolution feature vector from the second split.

[0025] According to one embodiment, the feature vector of the third split may be further fused with the feature vector of the first split, and the feature vector of the first split and the multi-resolution feature vector of the second split may be combined by concatenation before being fused with the feature vector from the encoder of the third split.

[0026] The third branch may process the third-rank resolution of the continuous stream, i.e., the third split. For the third split and each subsequent split having a higher-rank resolution, the feature vector obtained from the encoder of the third or subsequent branch may be fused with both the multi-resolution feature vector of the preceding branch and the (multi-resolution) feature vector of the branch even further ahead. In other words, the feature vector of split n may be fused with the multi-resolution feature vector of split n-1 and the (multi-resolution) feature vector of split n-2. Before merging the three feature vectors, the multi-resolution feature vector of split n-1 and the (multi-resolution) feature vector of split n-2 may be joined by concatenation. In one embodiment, the feature vector of split n may be fused with all the preceding feature vectors of splits n-1, n-2, n-3, ... 1. For all preceding splits in which a multi-resolution feature vector is encoded, such a multi-resolution feature vector may be used for merging with the feature vector of split n. The nth split may have an nth resolution higher than the resolution of each preceding split. In one embodiment, the nth feature vector may be merged with the feature vector of the first division and each of the remaining multi-resolution feature vectors of all preceding divisions.

[0027] According to one embodiment, the feature vector may be merged based on a K-nearest neighbors (KNN) algorithm. Each feature of a given partition may be merged with features corresponding to the K nearest neighbors from a previous partition in a continuous stream of points. The K nearest neighbors may be calculated based on the points corresponding to each feature of the feature vector. Since the feature vector has a reduced dimensionality compared to the input partition, the temporal impact can be limited. This results in improved performance, and because it utilizes feature vectors already calculated from other partitions, the complexity required for the encoder is reduced, and it is also more power efficient.

[0028] According to one embodiment, the input to the second or third decoder is a fused second or third multi-resolution feature vector, respectively.

[0029] At each branch where the feature vector is fused into a multi-resolution feature vector, the decoder for such a branch may receive the multi-resolution feature vector to be decoded in order to generate an output prediction. Thus, the decoder may decode the multi-resolution feature vector. This allows the decoder to receive more information (information fused from different resolutions) compared to the encoder's output feature vector, improving performance without negatively impacting time.

[0030] According to one embodiment, the decoder may be executed immediately after the fused multi-resolution feature vector is calculated.

[0031] Therefore, decoders at different branches may be executed asynchronously. This eliminates the need to wait until a high-density point cloud is acquired and processed, thus eliminating the waiting that was typical in conventional processing paradigms. As soon as the initial coarse representation of the scene is captured and encoded into a feature vector in the first division, the fused feature vector of the second division may be decoded. The obtained fused feature vector of the second division may then enable the decoding of the fused feature vector of the third division. This may continue in a manner in which the acquisition of the fused feature vector of the (n-1)th division enables the decoding of the fused feature vector of the nth division, and so on. This allows for simultaneous acquisition and processing of point clouds, enabling processing to begin using the acquisition time, resulting in faster and more time-efficient processing. Since it is not necessary to start all decoders simultaneously and they can be distributed over time, a more power-efficient solution is obtained.

[0032] According to one embodiment, the output of the branch decoder is obtained at different time instances. This improves the simultaneous execution of point cloud acquisition and deep learning-based encoder processing. It eliminates the need to wait until a high-density point cloud is acquired, and the initial results are obtained very quickly, enabling early decision-making. Furthermore, it provides flexibility according to the required confidence level, and the calculation of subsequent partitions can be interrupted once the required confidence level is achieved. The initial processing results can be obtained even while the acquisition process is in progress.

[0033] According to one embodiment, the output of the branch decoder may be refined by an update step that transfers information from the output of a later branch.

[0034] This allows the output of each branch to be improved using information from subsequent branches. Performance improvements can be achieved because the output of previous branches can be updated in a quick and simple refinement step without requiring recalculation of the output of earlier branches. Such updates can be implemented with minimal impact on power and time consumption. In one embodiment, updating the output prediction at a preceding branch may be performed by replacing the preceding prediction output with the result of a majority vote in the nearest neighbor point cloud in a later prediction. In another embodiment, the update may be implemented by combining the prediction output of a preceding branch with the prediction outputs of one or more subsequent branches. Thus, the prediction output of a preceding branch may be improved based on the prediction outputs of subsequent branches.

[0035] According to a second aspect of the present invention, a computer program product is provided which, when executed on a processing unit, is configured to perform a method according to any embodiment described herein.

[0036] According to a third aspect of the present invention, a readable computer medium containing the computer program product is provided.

[0037] The arithmetic unit may include one or more computer units, data processing units and / or general-purpose processors configured to perform each step of the method according to any embodiment described herein. The arithmetic unit may include a computer program product or software configured to perform each step of the method at runtime. The arithmetic unit may include a computer-readable storage medium containing the computer program product or software, or may be communicated to such medium.

[0038] A fourth aspect of the present invention provides a system for acquiring and processing point clouds. The system comprises a resolution-scalable 3D sensor configured to acquire a continuous stream of 3D points spatially distributed across an entire scene, and a splitting unit configured to split the continuous stream of points in real time. Each split is used as input to an encoder from the moment it becomes available. The continuous stream of points is continuously increased in resolution such that each subsequent split has a higher resolution than a preceding split. The system further comprises a deep learning-based point cloud encoder corresponding to each split. Each encoder is configured to encode the point cloud of the split into feature vectors. The system further comprises a decoder corresponding to each split. Each decoder is configured to decode the feature vectors corresponding to the split into output predictions.

[0039] The system may be configured to perform a method according to any of the embodiments described above. The system may further have at least one fusion block configured to operate according to any embodiment relating to the fusion blocks described herein. Furthermore, the system may have a computing unit configured to operate according to one or more components of the disclosed system. [Brief explanation of the drawing]

[0040] The present invention will be described in more detail below with reference to the accompanying drawings. [Figure 1] Figure 1 is a schematic block diagram of a process relating to one embodiment of the present invention. [Figure 2] Figure 2 is a schematic block diagram of a fusion block according to one embodiment of the present invention. [Figure 3] Figure 3 is a comparison chart of the performance results of one embodiment of the present invention. [Figure 4] Figure 4 is a comparison chart of time results for one embodiment of the present invention. [Figure 5] Figure 5 shows the visual results of one embodiment of the present invention. [Figure 6a] Figure 6a shows a visualization of the scanning pattern of a resolution-scalable 3D sensor that can be used in one embodiment of the present invention. [Figure 6b] Figure 6b is a diagram showing the visualization of the distribution of points in different divisions according to one embodiment of the present invention. [Figure 7] Figure 7 shows a visualization of the scanning pattern of a resolution-scalable 3D sensor that can be used in one embodiment of the present invention. [Figure 8] Figure 8 shows the visual results of one embodiment of the present invention. [Figure 9] Figure 9 is a flowchart of a method relating to one embodiment of the present invention. [Modes for carrying out the invention]

[0041] The present invention will be described in more detail below with reference to the accompanying drawings. The drawings show preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described herein and can be implemented in a variety of forms. Rather, these embodiments are provided to make this disclosure complete and to fully convey the scope of the invention to those skilled in the art. In the drawings, the same reference numerals indicate the same elements.

[0042] Figure 1 shows the processing pipeline of a system 10 that implements a method according to one embodiment of the present invention.

[0043] Scene 1 is acquired by a resolution-scalable 3D acquisition device. The resulting continuous stream of points is divided in real time into continuous and non-overlapping point cloud partitions 2a-2c, the resolution of which increases with increasing size. Point cloud partitions 2a-2c are encoded in parallel branches A, B, and C by deep learning-based point cloud encoders 3a-3c. Each branch A-C contains an encoder 3a-3c. Encoders 3a-3c are, for example, transformer-based. Encoders 3a-3c for different partitions 2a-2c are executed asynchronously, by being executed as soon as partitions 2a-2c become available. While the encoder for one partition is being executed, the point cloud for the next partition is acquired. Encoders 3a-3c encode each input point cloud partition 2a-2c into feature vectors 4a-4c corresponding to its specific resolution.

[0044] The encoder-decoder architecture may be PointNet++-based[7,11], point convolution-based[8-10], or transformer-based[5,12-13]. Therefore, those skilled in the art will understand that other architectures are equally applicable. [7] Qi, Charles Ruizhongtai, et al. “Pointnet++: Deep hierarchical feature learning on point sets in a metric space.” Advances in neural information processing systems 30 (2017). [8] Wang, Yue, et al. “Dynamic graph cnn for learning on point clouds.” ACM Transactions on Graphics (tog) 38.5 (2019): 1-12. [9] Thomas, Hugues, et al. “Kpconv: Flexible and deformable convolution for point clouds.” Proceedings of the IEEE / CVF international conference on computer vision. 2019.

[10] Zhao, Hengshuang, et al. “Pointweb: Enhancing local neighborhood features for point cloud processing.” Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019.

[11] Qian, Guocheng, et al. “Pointnext: Revisiting pointnet++ with improved training and scaling strategies.” Advances in Neural Information Processing Systems 35 (2022): 23192-23204.

[12] Wu, Xiaoyang, et al. “Point transformer v2: Grouped vector attention and partition-based pooling.” Advances in Neural Information Processing Systems 35 (2022): 33330-33342.

[13] Lai, Xin, et al. “Stratified transformer for 3d point cloud segmentation.” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022.

[0045] The multi-resolution feature vector fusion blocks 5b-5c are configured to fuse the feature vectors 4b-4c obtained for a particular partition with the feature vectors 4a-4c from a previous partition. This fusion process may use the K-nearest neighbor (KNN) technique, which utilizes the features of neighboring points. The resulting multi-resolution feature vectors 6b-6c contain information from different partitions and therefore from different resolutions.

[0046] More specifically, in one embodiment, the second branch B includes a fusion block 5b configured to fuse the feature vector 4a from the first division 2a with the feature vector 4b from the second division 2b, resulting in the generation of a multi-resolution feature vector 6b. Furthermore, the third branch C may include a fusion block 5c configured to fuse the feature vector 4c from the third division 2c with the multi-resolution feature vector 6b, which is the result of the fusion process in the second branch B. In one embodiment, the multi-resolution feature vector 6b may be concatenated with the feature vector 4a from the first division 2a before fusing with the feature vector 4c from the third division 2c. This process is repeated for any number n resolution layers. At the nth branch, a fusion process may be performed between the feature vector of the nth branch and the encoded feature vectors and / or multi-resolution feature vectors of the preceding branches n-1, n-2, n-3, ..., 1. The feature vector of the nth branch may be fused with the feature vector of the first branch and the multi-resolution feature vectors of all the remaining preceding branches.

[0047] The deep learning-based point cloud decoders 7a to 7c are used to decode feature vectors that are the feature vector 4a or the multi-resolution feature vectors 6b to 6c, and output output predictions 8a to 8c of the segments 2a to 2c in branches A to C. The decoders 7a to 7c may adopt a transformer architecture such as a point transformer. The decoders 7a to 7c may be executed asynchronously by being executed immediately after the feature vectors 4a, 6b to 6c become available. The resulting output prediction 8 is obtained asynchronously. For example, the prediction output 8a for the first segment 2a is obtained while the acquisition and processing of other segments 2b to 2c are in progress. This process can be similarly generalized to any number n of branches, where each branch includes a decoder that decodes an output prediction based on the input segment and the feature vectors of preceding branches.

[0048] In one embodiment, a stepwise updating step 9 may be applied. This updating step is performed iteratively such that more information is fused to improve the quality of preceding output predictions 8a to 8c based on higher-quality subsequent predictions.

[0049] More specifically, the segments 2a to 2c can be mathematically expressed as X_i∈R N_i×C (In this mathematical expression, the subscript i attached to symbol Z is expressed as "Z_i". The same applies hereinafter. Also, hereinafter, for the summation symbol Σ, the addition of summing mathematical expression terms Z_k from the first term to the n-th term (k=1,...,n) is expressed as Σ{k=1~n}Z_k). Here, i∈[1,s], N_1<…<N_s<N, s is the number of segments, N is the total number of points in the final dataset, which is the sum of points of all segments. In the illustrated example, the first segment 2a includes points P_1 to P_i_1, the second segment 2b samples points P_i_1+1 to P_i_2, and the third segment 2c samples points P_i_2+1 to P_i_3. The segmentation is performed sequentially along time t.

[0050] The first division 2a (also denoted as X_1) is processed by the first encoder 3a and the first decoder 7a, thereby obtaining a prediction output 8a (mathematically, Y_1∈R N_1 ). Since N_1<<N (where N is the number of points in the final dataset), the first division has the lowest resolution, low computational complexity, and enables rapid prediction.

[0051] The second division 2b (divided as X_2) is processed by a separate encoder 3b, and the feature vector 4b is α_2∈R N_2´×F obtained as . Here, N_2' is the number of points (after downsampling) for which F features are respectively calculated in the feature vector 4b. To improve performance, the features of the feature vector 4b are fused by the fusion module 5b with the features of the feature vector 4a of the preceding division 2a. The multi-resolution feature α_2 of the multi-resolution feature vector 6b obtained thereby f is used by the decoder 7b to obtain an output prediction 8b (also denoted as Y_2). When all s divisions have been processed, Y={Y_1,Y_2,…,Y_s}∈R N is obtained.

[0052] From the perspective of computational complexity, the present invention has advantages over methods that handle all sampling data collectively. When using an attention mechanism that requires K-nearest neighbor (KNN) calculation, the computational complexity of collectively processing the entire sampled input is O(N 2 )=O((N_1 +…+N_s) 2 = O(N_1 2 +…+N_s 2 + 2Σ{k=1~s}Σ{p=1~s,p≠k}N_kN_p). Here, N=N_1+…+N_s, and s is the number of divisions with increasing resolution. On the other hand, in the present invention, since the attention mechanism can be applied to the divisions 2a to 2c in parallel, the computational complexity is O(N_1 2 +…+N_s 2) . Compared to a non-scalable approach, the present invention substantially reduces computational complexity by the amount of the factor proportional to Σ{k=1~s}Σ{p=1~s,p≠k}N_kN_p.

[0053] As the number of partitions s increases, the size of each partition decreases, making the effect of reducing computational complexity more pronounced. Furthermore, sequential partitioning can also provide computational redundancy, but KNN is the most computationally expensive process. When N is large and the number of partitions is large, the effect of removing the double product term can become significant.

[0054] Further explanation of the fusion block. Connected feature α_(i-1) from low-resolution partitioning. c ∈ R N_j´×F In this case, N_j' is the number of connected points in the feature space. α_(i-1) c Given α_i, the fusion block integrates the multi-resolution information into a single feature vector, which is then used for decoding.

[0055] The deep learning-based encoder-decoder structure may be trained for each partition. When training an additional partition, all weights from the previous partition may be frozen, and the loss function may be calculated only for the result from the current partition. This allows the backbone architecture to achieve maximum performance for each resolution.

[0056] Figure 2 shows a multi-resolution feature vector fusion block 5. Generally, fusion block 5 takes relevant features from a previous split and fuses them with the features of the current split. Fusion block 5 includes a query and group (QAG) block 10, which is configured to take the feature vector 4 and / or multi-resolution feature vector 6 from the previous split that are most relevant to the features of the current split. More specifically, the QAG block takes α_(i-1) belonging to points in the K neighborhoods of a point corresponding to a feature α_i. cThe KNN algorithm can be used to extract features from the data. Since these features are obtained from partitions at different resolutions, the resulting feature vectors contain multi-resolution information.

[0057] The multi-resolution feature vector fusion block 5 further includes a deep learning-based processing block 11 configured to encode the features of these K neighbors into a single feature vector. The processing block 11 may be configured to process the K neighbors by a one-dimensional convolution (Conv1D) operation followed by a maximum pooling (MaxPool) layer.

[0058] The multi-resolution feature vector fusion block 5 is further configured to concatenate the feature α_i of the current division with the feature vector obtained from the processing block 11, and to generate a single multi-resolution feature vector 6 that incorporates multi-resolution information for the division.

[0059] Figure 3 shows the performance results of the disclosed method. The X-axis 13 shows the number of points, and N_i is the number of points included in the i-th division. The performance of each method for semantic segmentation is compared in terms of mean Intersection over Union (mIoU) 14. As a baseline 15, a backbone semantic segmentation network is used to process the point cloud non-stepwise. Therefore, the baseline method, in contrast to the methods described above, processes the complete point cloud in each division and therefore requires all data to be available before execution. The baseline method does not work with different divisions, does not have the ability to process data stepwise, and does not produce intermediate results. The results of the method described above are shown in 16. Because this method works stepwise, the points are interconnected and dependent on each other. Furthermore, the results of the method according to an embodiment that does not include update step 9 are shown in 17. The results of the method according to an embodiment that does not include multi-resolution fusion 5 and update step 9 are shown in 18.

[0060] Figure 4 shows the time results during inference. The y-axis shows the required inference time 19. The upper limit of the inference time is shown at 20. The upper limit is when data acquisition is instantaneous and all delays are caused by processing. The lower limit of the inference time is shown at 21. The lower limit is when processing is faster than acquiring individual partitions. Region 22 is demarcated by the upper and lower limits, and all operations occur here. Note that even when operating at the upper limit, the present invention can reduce the inference time at the highest partition by approximately 31% compared to the baseline. At the lower limit, a reduction of approximately 61% can be achieved. These reductions in inference time are mainly achieved by the reduction in complexity described above.

[0061] Figure 5 shows the visual results of this method for a scene. The figure includes result 23 for one of the lowest divisions, result 24 for an intermediate division, and result 25 for a later division.

[0062] Figure 6a shows an example of a scan pattern that can be used in a 3D sensor to distribute points in a scene as a continuous stream of points. As an example of a scan pattern, the graph in the upper left of Figure 6 may show a scan pattern used to acquire points in the first division. In this case, spatially complete coverage across the entire scene is achieved, but the resolution is very low. Next, in the graph in the upper right, the scan pattern further covers the range of points to be acquired, and the additional points acquired make it possible to select a second division with increased resolution. The graphs in the lower left and lower right show the distribution when even more points are acquired, resulting in higher resolution in subsequent divisions. Over time, the number of points distributed across the entire scene increases, and the resolution in subsequent divisions increases.

[0063] Figure 6b shows how the scanning pattern in Figure 6a is used to form the divisions, demonstrating that points in subsequent divisions do not overlap with points in preceding divisions. It also shows that each subsequent division contains points that increase the resolution compared to the preceding division. Each subsequent division may contain more points than the preceding division. The distribution of points in each division may follow the scanning pattern used by a resolution-scalable 3D sensor.

[0064] Figure 7 shows an example of another scanning pattern in which the resolution can be increased over time in subsequent divisions.

[0065] Figure 8 shows the visual results of this method for a scene. Figure 8a shows the acquired points, semantic labels, and the underlying 3D scene from which these points were acquired. Figure 8b shows the results of the first split, in which points were acquired according to a distribution pattern. Figure 8c shows the results of the first subsequent split, in which additional points were acquired at a higher resolution compared to the first split. Figure 8d shows the results of the second subsequent split, in which points were acquired at an even higher resolution compared to the preceding split.

[0066] Figure 9 shows a method 100 according to one embodiment of the present invention. Method 100 includes a step 102 of acquiring a continuous stream of 3D points of a scene. The continuous stream is acquired using a resolution-scalable 3D sensor. The continuous stream is divided 104 in real time. Each division is used as input to an encoder 3. For each division, the continuous stream is acquired so that the resolution increases sequentially. Thus, each subsequent division provides a higher resolution compared to the preceding division. The method further includes a step 106 of encoding the point cloud of each division into feature vectors 4 using a deep learning-based point cloud encoder 3. Method 100 further includes a step 108 of decoding the feature vectors 4 of each division into output predictions 8.

[0067] In one embodiment, method 100 may further include step 107 of fusing the feature vector 4 of the second division with the feature vector 4 of the first division to generate a multi-resolution feature vector 6. The first division may have a first resolution, and the second division may have a second resolution. The first resolution is lower than the second resolution.

[0068] Preferred embodiments and examples of the present invention are disclosed in the drawings and specification, and certain terms are used, but these are used only in a general and descriptive sense and not to limit the invention. The scope of the present invention is defined by the following claims.

Claims

1. A method for acquiring and processing a point cloud using a 3D sensor that acquires a continuous stream of 3D points spatially distributed across the entire scene, A step of acquiring a continuous stream of points using a resolution-scalable 3D sensor, wherein the continuous stream of points is divided in real time, each division (2a-2c) is used as input to an encoder (3a-3c) from the time the division becomes available, and the points of the continuous stream are acquired such that each subsequent division (2a-2c) has an increased resolution compared to the preceding division. The steps include: encoding the point cloud of each division into feature vectors (4a to 4c) using a deep learning-based point cloud encoder (3); The steps include decoding the feature vectors of each division (2a to 2c) into output predictions (8a to 8c) using decoders (7a to 7c), including, Methods for acquiring and processing point clouds.

2. In the method for acquiring and processing point clouds according to claim 1, The division of the aforementioned continuous stream (2a to 2c) is such that the size increases sequentially. Methods for acquiring and processing point clouds.

3. In the method for acquiring and processing point clouds according to claim 1 or claim 2, Different encoder-decoder structures are used in parallel branches (A, B, C), and each branch processes a different division (2a to 2c). Methods for acquiring and processing point clouds.

4. In the method for acquiring and processing point clouds according to claim 3, The encoders (3a to 3c) of each branch (A, B, C) are executed immediately after the division (2a to 2c) is obtained from the continuous stream. Methods for acquiring and processing point clouds.

5. A method for acquiring and processing a point cloud according to any one of claims 1 to 4, The feature vector (4b) obtained by the encoder (3b) for the second division (2b) having a second resolution is fused with the feature vector (4a) from the first division (2a) having a first resolution lower than the second resolution to become a multi-resolution feature vector (6b). Methods for acquiring and processing point clouds.

6. In the method for acquiring and processing point clouds according to claim 5, The feature vector obtained by the encoder for the nth division, which has a higher resolution than the previous nth division, is merged with the feature vectors of all previous divisions and / or multi-resolution feature vectors. Methods for acquiring and processing point clouds.

7. In the method for acquiring and processing point clouds according to claim 6, The feature vector (4c) from the third division (2c) is further fused with the feature vector (4a) from the first division (2a). The feature vector (4a) of the first division (2a) and the multi-resolution feature vector (6b) of the second division (2b) are combined by concatenation before being merged with the feature vector (4c) from the encoder (3c) of the third division (2c). Methods for acquiring and processing point clouds.

8. A method for acquiring and processing a point cloud according to any one of claims 5 to 7, The aforementioned feature vectors are merged based on the K-nearest neighbors (KNN) algorithm. Methods for acquiring and processing point clouds.

9. A method for acquiring and processing a point cloud according to any one of claims 5 to 8, The inputs to the second or third decoder (7b-7c) are, respectively, the fused second or third multi-resolution feature vectors (6b-6c). Methods for acquiring and processing point clouds.

10. In the method for acquiring and processing point clouds according to claim 9, The decoders (7b to 7c) are executed immediately after the fused multi-resolution feature vectors (6b to 6c) are calculated. Methods for acquiring and processing point clouds.

11. A method for acquiring and processing a point cloud according to any one of claims 1 to 10, The outputs of the aforementioned divided decoders (7a to 7c) are obtained at different time instances. Methods for acquiring and processing point clouds.

12. A method for acquiring and processing a point cloud according to any one of claims 1 to 11, The outputs (8a-8c) of the branch decoders (7a-7c) are refined by an update step (9) which transfers information from the outputs (8b-8c) of the subsequent branch. Methods for acquiring and processing point clouds.

13. A computer program product configured to perform the point cloud acquisition and processing method described in any one of claims 1 to 12 when executed on a processing device.

14. A readable computer medium comprising the computer program product described in claim 12.

15. A system (10) for acquiring and processing point clouds, A resolution-scalable 3D sensor configured to acquire a continuous stream of 3D points spatially distributed across the entire scene, A splitting unit configured to split a continuous stream of points in real time, wherein each split (2a to 2c) is used as input to an encoder (3a to 3c) from the time the split becomes available, and the points of the continuous stream are acquired such that each subsequent split (2a to 2c) has an increased resolution compared to the preceding split. Each division (2a to 2c) is represented by a deep learning-based point cloud encoder (3a to 3c) configured to encode the point cloud of the division into feature vectors (4a to 4c), A decoder (7a-7c) is configured to correspond to each division (2a-2c) and to decode the feature vectors (4a-4c) corresponding to the division (2a-2c) into output predictions (8a-8c), including, system.

Images