A computer-implemented method for semantic segmentation of point cloud data and applications thereof.

Abstract

A computer-implemented method for semantic segmentation of sparse point cloud data 20 comprises providing point cloud data 20 that include coordinate data indicative of the spatial location and veloci

Description

A computer-implemented method for semantic segmentation of point cloud data and applications thereof TECHNICAL FIELD The invention relates to a computer-implemented method for semantic segmentation of point cloud data. Furthermore, the invention relates to a perception system for a vehicle, preferably an autonomous vehicle. BACKGROUND An autonomous driving (AD) perception system usually requires multiple sensors working together to achieve optimal results. Therefore, it is necessary to actively explore the perception upper bound of each sensor individually. Currently, there is much research work on cameras and LIDAR sensors, but relatively little work on RADARs to achieve autonomous driving. Instead, radar (specifically doppler millimetre-wave radar) has been used for Advanced Driving Assistance System (ADAS) applications such as collision avoidance and Adaptive Cruise Control only. Compared with Lidar and camera sensors, radar is able to detect long-range objects and may perform more robustly in adverse weather conditions such as foggy, rainy or snowy weather. However, in autonomous driving, radar still usually performs rather simple primary sensing tasks in ADAS applications, in which the system does not need to understand the surrounding environment since humans will do the primary decision-making. Despite how commonplace the sensor is, it is rarely used for semantic segmentation, or object detection tasks. Since the basic information of radar is the point cloud in 3D space, similar to a Lidar, it seems intuitive that algorithms determining a 3D bounding box of each object should be used. However, it has been found that it is challenging to predict precise and useful 3D bounding boxes with radar point clouds because of the radar point cloud’s sparsity and typical lack of accurate height information. Point cloud semantic segmentation has long been a field of interest in order to enable encoding 3D point clouds for downstream tasks such as detection and segmentation. [1] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space,” Jun. 2017, available: http: / / arxiv.org / abs / 1706.02413 proposes a sampling and grouping technique in an attempt to partition points within a region. However, radar point clouds are significantly sparser, and key information may be lost when the point cloud is sampled and grouped. Due to the sparsity of radar point cloud, radar and some other sensors are being used to perform perception tasks jointly to ensure a certain level of accuracy. Y. Kim, J. W. Choi, and D. Kum, “Grif net: Gated region of interest fusion network for robust 3d object detection from radar point cloud and monocular image,” in 2020 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 10 857-10 864, proposes extracting two Region of Interest (Rol) features from camera and radar, then uses a designed explicit gating mechanism to adaptively select the appropriate data when one sensor is abnormal, for 3D object detection. In the above-mentioned document, despite the so-called fusion, radar still behaves as an auxiliary sensor, while all main features are provided by camera. SUMMARY OF THE INVENTION It is the object of the invention to improve perception systems for autonomous driving, in particular under challenging weather. The invention provides a computer-implemented method for semantic segmentation of sparse point cloud data, the method comprising: a) providing point cloud data that include a plurality of cloud data points, wherein each cloud data point is associated with coordinate data, that are indicative of the spatial location of the cloud data point, and velocity data that are indicative of the velocity of the cloud data point; b) feeding the point cloud data to a transformer module that is trained for processing the point cloud data for semantic segmentation, wherein the transformer module outputs transformed point cloud data; c) feeding the transformed point cloud data to a fully connected layer module comprising a plurality of fully connected layers, wherein a last fully connected layer along the feed forward direction of the plurality of fully connected layers outputs preclassification data that, for each cloud data point, include an associated probability value for each semantic class of a predetermined number of semantic classes of the semantic segmentation; d) feeding the pre-classification data to a classification module that comprises a classification layer and a concatenation layer, wherein the classification layer classifies each cloud data point into one of the semantic classes based on the preclassification data and outputs classified point cloud data, wherein the concatenation layer concatenates the unprocessed point cloud data to the classified point cloud data and outputs them as concatenated point cloud data; e) refining spatial clustering of the concatenated point cloud data based on the concatenated point cloud data, thereby resulting in semantically segmented point cloud data. The semantic segmentation of sparse point cloud data is enabled by analysing not only coordinate data but also velocity data of each cloud data point. With the additional velocity data it is possible to achieve enough precision for semantic segmentation even when the point cloud data is sparse. The transformer module allows the analysis of sequences of point cloud data by evolving the movement of the coordinate points forward to a predetermined point in time, thereby effectively increasing the density of datapoints at that point in time. With these measures a reliable semantic segementation of point cloud data can be achieved without additional information based solely on the sparse point cloud data. Preferably, in step a) the point cloud data are provided as radar point cloud data. Radar point cloud data allow a direct and more precise measurement of velocity data for each cloud data point, e.g. via doppler effect. This allows an improvement in accuracy and possibly reliability of the semantic segmentation, in particular in border regions, where different semantic regions meet. Preferably, in step a) the point cloud data include for each cloud data point radar cross section (res) data that are indicative of the radar cross section. The res is typically related to geometry and / or material of the object that is scanned. Since it can be assumed that a similar material or geometry belongs to the same object, the res allows further improvement of the accuracy of the semantic segmentation. Preferably, in step b) the transformer module is configured as a point transformer module that comprises at least one point transformer block. Preferably, in step b) the transformer module is configured as an autoencoder. Preferably, in step b) the transformer module comprises a feature encoder portion. Preferably, the transformer module comprises a feature decoder portion. Preferably, in step b) the feature encoder portion and / or the feature decoder portion each include at least one point transformer block. The point transformer module is better equipped to handle point cloud data, in particular in the autoencoder configuration, which is particularly suitable for semantic segmenatation tasks. Preferably, in step b) the transformer module comprises a feature encoder portion (26), and the feature encoder portion includes at least one point transformer block and at least one encoder block that is fed with the point cloud data or with an output of a previous block along the feed forward direction of the transformer module, wherein each encoder block consists of a transition down block followed by the transformer block. The feature encoding portion reduces dimensionality of the point cloud data while being able to extract features. Preferably, in step b) the transformer module comprises feature decoder portion, and the feature decoder portion includes at least one point transformer block and at least one decoder block that is fed with an output the feature encoder portion, preferably with an output of a last encoder block along the feed forward direction of the feature encoder portion, wherein each decoder block consists of a transition up block followed by the transformer block. The feature decoder portion increases dimensionality based on encoded features, thereby allowins a spatial association between the extracted features and where they are located in the point cloud data. Preferably, one of the encoder blocks and one of the decoder blocks are associated with each other, and the output of the associated encoder block, preferably the output of the transformer block of the associated encoder block, is fed to the transition up block of the associated decoder block. With this the decoding process can be more accurate. Preferably, in step c) the fully connected layer module comprises a first fully connected layer along the feed forward direction that is fed with the transformed point cloud data, wherein the number of features per cloud data point of the first fully connected layer and the number of features per cloud data point of the transformed point cloud data are the same. With this the transformed point cloud data is prepared for subsequent classification, which allows an improvement in classification accuracy. Preferably, the fully connected layer module comprises a second fully connected layer along the feed forward direction that has a larger number of neurons than the number of features per cloud data point, preferably double the number of neurons than the number of features per cloud data point. With this the data is further processed for subsequent classification, which allows an improvement in classification accuracy. Preferably, in step c) the probability value is determined by a softmax function. Using the probability of a cloud data point having a certain semantic class allows for improved accuracy. Preferably, in step d) the classification layer determines the semantic class by selecting the semantic class having the largest probability value. Preferably, in step e) the spatial refinement is performed by a rules based model or a k-nearest neighbours method. Spatial refinement allows for a better resolution of fringe cases, in particular where there are graduated or fuzzy borders between semantic classes. Preferably, in step a) the point cloud data are provided by a computer-readable medium and / or by a radar sensor apparatus that is configured to measure location and velocity of an object and output point cloud data that are indicative of a relative location and a relative velocity of the object relative to the radar sensor apparatus. This method can be applied to pre-collected data or real-time collected data by sensors. The invention provides a method for controlling a vehicle or drone by performing a previously described method and generating a control signal for controlling the vehicle or drone based on the semantically segmented point cloud data. The invention provides a data processing apparatus comprising means for carrying out a previously described method. The invention provides a vehicle or drone comprising a data processing apparatus and a radar sensor apparatus for carrying out the disclosed method. The invention provides a computer program comprising instructions which, when the program is executed by a data processing apparatus, cause the data processing apparatus or the data processing apparatus and a radar sensor apparatus to carry out the disclosed method. The invention provides a computer readable storage medium or a data carrier signal that includes the computer program. One idea is that radar sensors may use the doppler effect to accurately estimate the velocities of detected objects, without any processing of temporal information. These advantages and low cost makes the radar a popular choice in autonomous driving system. The velocity data generated by the radar sensors are used to improve the results of the segmentation task. The method is proposed as the first point semantic segmentation method which accepts only radar point cloud data input (i.e. no other input from cameras is necessary) and performs well on public benchmark without additional data. In other words, the proposed method is the first method that allows reliable semantic segmentation of radar point cloud data without resorting to additional data. The method can make full use of multiple radar point attributes such as velocity and radar cross section (RCS), whose specific effects are may significantly improve the results of the semantic segmentation. Furthermore, a radar point post-correction module is proposed to improve the semantic segmentation performance. One approach is to focus on the radar point cloud semantic segmentation in Bird-Eye-View (BEV) space. A specific aim of the method is to determine where each radar point belongs to, i.e. to which object the radar point belongs. To the best of the inventors’ knowledge, the disclosed radar point semantic segmentation method, named RadSeg, is at the time of filing the only one performing reliably on state-of-the-art benchmark datasets and in real world testing. The method is performed by a pipeline that includes a modified point transformer network as the backbone preferably followed by an output head, that preferably includes fully connected layers and a classifier. The output head is followed by a spatial refinement step. The existing point transformer backbone is modified to accept velocity data and optionally RCS data or dynaprop data in addition to position information. The base network may make use of coordinates x, y, z; velocities: vxcomp, vycomp; and radar cross section (res) values, x, y, z refer to the 3D coordinate of the cloud data point, respectively, vxcomp, vycomp refer to the x and y components of the compensated radial velocities of each cloud data point. The radial velocities are compensated with the velocity of the ego vehicle and motion (i.e. the compensated radial velocities are such relative to the ego vehicle). The res value measures how detectable an object is by radar. In addition, the velocity and motion of the ego vehicle may be used to transform cloud data points observed at t-5 to t-1 to the frame t. In other words, the presented method is able to use multiple point clouds as input rather than just a single one. The point clouds may be accumulated for a time interval, which allows for a denser and richer point cloud for semantic segmentation. The output head of the point transformer network is preferably modified by adding fully connected layers to the MLP with a ReLU activation layer. The output of the network is the prediction scores of these classes for each point given as N,C where C refers to the number of classes and N is the number of points. Another idea involves a spatial refinement step to update the predicted classification results from the network. It is assumed that radar points of a certain class should have similar properties. A k-nearest neighour (kNN) method is used within a given radius to performed the refinement. Cloud data points within a certain radius, r, and with similar properties are given the same class assignment. BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention are described in more detail with reference to the accompanying schematic drawings. Fig. 1 depicts an embodiment of a vehicle; Fig. 2 depicts an embodiment of a pipeline; Fig. 3 depicts an embodiment of a transformer module; Fig. 4 depicts an embodiment of a transformer block; Fig. 5 depicts an embodiment of a transformer layer; Fig. 6 depicts an embodiment of a transition down block; Fig. 7 depicts an embodiment of a transition up block; Fig. 8A and Fig. 8B compile experimental results; Fig. 9 illustrates the effects of spatial refinement on sparse data. DETAILED DESCRIPTION OF EMBODIMENT Fig. 1 depicts a vehicle 10. The vehicle 10 is preferably an autonomous vehicle. The vehicle 10 comprises a vehicle control unit 12. The vehicle control unit 12 is configured to control the vehicle 10. The vehicle control unit 12 may be configured to assist a driver of the vehicle 10 or may be capable of full autonomous driving of the vehicle 10. The vehicle control unit 12 is an example of a data processing apparatus. The vehicle 10 includes a radar sensor apparatus 14. The radar sensor apparatus 14 may include a plurality of radar sensors 16 that are distributed around the circumference of the vehicle 10. Ideally, the radar sensors 16 are arranged such that the radar sensor apparatus 14 cover the entire 360° surroundings of the vehicle 10. The radar sensor apparatus 14 has a sensing area 17 in which an object 18 can be detected. The radar sensor apparatus 14 is configured to generate point cloud data 20 based on the location and velocity of the object 18 relative to the vehicle 10. In addition the point cloud data 20 may include radar cross-section data and / or dynaprop value data. The point cloud data 20 are fed to the vehicle control unit 12 for processing and generating a control signal based on the point cloud data 20. The control signal may be a warning signal that warns the driver of obstacles. The control signal may be a steering signal which causes the vehicle 10 to avoid the object 18. It is also possible that the control signal causes the vehicle 10 to accelerate or decelerate in order to keep a predetermined distance from the object 18, for example. Another possible control signal is an emergency brake signal that causes the vehicle 10 prepare for collision and to decelerate at the maximum safest deceleration in order to avoid a collision with the object 18. Fig. 2 depicts an embodiment of a pipeline that is executed by the vehicle control unit 12. The pipeline comprises an input layer 22. The input layer 22 is configured to accept the point cloud data 20. The point cloud data 20 includes a plurality of cloud data points that represent the object 18. Each cloud data point has associated with it coordinate data x, y, z and velocity data vx, vy. The x-y plane is thereby typically defined to be parallel to the ground, whereas the z-axis represents the height. The velocity data vx and vy may also be designated radial velocity data. The velocity data vx, vy are determined by the radar sensor apparatus 14 as relative velocities to the vehicle 10, i.e. the velocity data vx, vy are indicative of the velocity of the object 18 relative to the vehicle 10. The pipeline comprises a transformer module 24. The transformer module 24 receives the point cloud data 20. The transformer module 24 is substantially configured as an autoencoding transformer. The transformer module 24 comprises a feature encoder portion 26 and a feature decoder portion 28. Referring to Fig. 3, the transformer module 24 may include an optional transformer input block 30. The transformer input block 30 preferably consists of a multilayer perceptron (MLP) input block 32 that is followed by a point transformer block 34. The feature encoder portion 26 includes at least one encoder block 36. Each encoder block 36 preferably consists of a transition down block 38 followed by the point transformer block 34. The transformer module 24 may include an optional transformer center block 40. The transformer center block 40 preferably consists of an MLP block 42 followed by the point transformer block 34. The transformer center block 40 is preferably arranged between the feature encoder portion 26 and the feature decoder portion 28 along the feed forward direction. The feature decoder portion 28 includes at least one decoder block 44. Preferably, the feature decoder portion 28 includes one decoder block 44 for each encoder block 36. Each decoder block 44 preferably consists of a transition up block 46 followed by the point transformer block 34. The last decoder block 44 outputs transformed point cloud data 48 for further processing. The point transformer block 34 of each encoder block 36 preferably feeds its output also to the transition up block 46 of the corresponding decoder block 44, as depicted in Fig. 3. In other words, the first point transformer block 34 feeds to the last transition up block 46, the second point transformer block 34 feeds to the second to last transition up block 46, and so on. The pipeline comprises a fully connected layer module 50. The fully connected layer module 50 is configured to output pre-classification data 52. The pre-classification data 52 includes a probability value for each predetermined semantic class of the semantic segmentation. Each cloud data point is associated with one probability value for each semantic class. The probability value is preferably determined using a softmax function. The fully connected layer module 50 may comprise a plurality of fully connected layers 54. A first fully connected layer 56 is adapted to receive the transformed point cloud data 48 and preferably has 32 output features. A second and third fully connected layer 58, 60 preferably have a larger number of neurons as the first fully connected layer 56, e.g. 64 neurons. A fourth fully connected layer 62, which in this example is the output layer, has a number of outputs according to a number of predetermined semantic classes for semantic segmentation, in this embodiment C=9 output layers. The fourth fully connected layer 62 preferably includes a softmax-nonlinearity in order to obtain a probability value for each semantic class. The pipeline comprises a classification module 64. The classification module 64 receives the pre-classification data 52. The classification module 64 preferably consists of a classification layer 66 followed by a concatenation layer 68. The classification layer 66 classifies the pre-classification data 52 such that each cloud data point is associated with exactly one of the predetermined semantic classes. The classifier 66 preferably selects the semantic class having the highest probability value. The classifier 66 outputs classified point cloud data. The concatenation layer 68 concatenates the point cloud data 20 to the classified point cloud data thereby obtaining concatenated point cloud data 70 for further processing. The pipeline comprises a spatial refinement module 72. The spatial refinement module 72 is configured to refine the clustering of the concatenated point cloud data 70. Based on the premise that cloud data points belonging to the same semantic class, the spatial refinement module 72 performs a k-nearest neighbours algorithm, e.g. by selecting a cloud data point from the concatenated point cloud data and assigning to the selected cloud data point the semantic class that is associated with the majority of the k = 3 nearest neighbour cloud data points within a predetermined radius r. The spatial refinement module 72 finally outputs the semantically segmented point cloud data 74. The semantically segmented point cloud data 74 are further processed by the vehicle control unit 12 that depending thereon generates a control signal. For example, in a parking situation, the object 18 may be recognized as another vehicle coming closer, then the vehicle control unit 12 may generate the control signal to warn of the closing other vehicle. It is also possible to detect a parking space between neighbouring vehicles and the vehicle control unit 12 may generate control signals that cause the vehicle 10 to navigate into the parking space. In another application, the vehicle control unit 12 may generate a control signal that causes the vehicle 10 to keep a more or less constant and / or safe distance from a leading vehicle or a passing vehicle. Referring to Fig. 4, the point transformer block 34 is described in more detail. The point transformer block 34 includes a first linear layer 76 and a second linear layer 78 with an interposed transformer layer 80 (also called a point transformer layer). The first and second linear layer 76, 78 are of the usual form y = x AT + b, where y is the output feature vector, x is the input feature vector, A is a weight matrix and b a bias vector. The training parameters are the weight matrix and the bias. The symbol T indicates transposition. The point transformer block 34 furthermore includes a bypass for concatenating the input data to the output data of the point transformer block 34. Referring to Fig. 5, the transformer layer 80 is illustrated in more detail. The transformer layer 80 comprises a linear layer 82, an MLP layer 84 and a linear layer 86. The input of the transformer layer 80 is fed to all those layers 82-86 in parallel. The outputs of the linear layer 82 and the MLP layer 84 are fed to a first combination means 88, where those outputs are combined. The outputs of the MLP layer 84 and the linear layer 86 are fed to a second combination means 90, where those outputs are combined. The output of the first combination means 88 is fed through another MLP layer 92 into an aggregation layer 94. The output of the second combination means 90 is directly fed to the aggregation layer 94. For greater details, reference is made to H. Zhao, L. Jiang, J. Jia, P. Torr, and V. Koltun, “Point Transformer,” Sep. 2021, arXiv:2012.09164 [cs], [Online], Available: http: / / arxiv.org / abs / 2012.09164, section “3.2. Point Transformer Layer” the disclosure of which is incorporated herein by reference. Referring to Fig. 6, the transition down block 38 is illustrated in more detail. The transition down block 38 includes a farthest point sampling layer 96, an MLP layer 98 and a local max pooling layer 100. The farthest point sampling layer 96 is configured to perform an iterative farthest point sampling algorithm in order to subsample a set of K points from the input. Typically, at each iteration, a cloud point is selected which has the largest nearest neighbor distance to any of the already selected cloud points. The selected cloud points are then fed to the MLP layer 98, which is preferably configured for performing a k-nearest neighbours algorithm, and subsequently to the local max pooling layer 100, which are well known, respectively. Referring to Fig. 7, the transition up block 46 is illustrated in more detail. The transition up block 46 includes a first linear layer 102 and a second linear layer 104. The first linear layer 102 is fed from the transformer center block 40 or a previous decoder block 44. The second linear layer 104 is fed from the point transformer block 34 of the corresponding encoder block 36. The first linear layer 102 is followed by an interpolation layer 106 that performs interpolation to according to one of the usual algorithms, such as linear, bilinear, bicubic or trilinear interpolation. The transition up block 46 comprises a summation layer 108 that sums the outputs of the interpolation layer 106 and the second linear layer 104 and outputs that result. The pipeline can be trained in the usual manner as subsequently described. The spatial refinement module 72 is usually not part of the training loop, since it is preferably a rules based module. In the following results of the experimental validation of the disclosed method are presented. The method was validated using the nuScenes dataset as the main benchmark dataset. Information on nuScenes can be found in H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and 0. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in CVPR, 2020. As one of the largest multi-modal autonomous driving dataset currently, nuScenes includes radar sensor inputs as well as a LiDAR point semantic segmentation tasks. Here the focus is on radar point cloud semantic segmentation. This dataset contains 5 synchronized and calibrated radars with object labels at 2 Hz. The radar data format is a point cloud with 18 feature dimensions (including x, y, z, vx, vy, RCS, dynaprop etc.) for each cloud data point. In the disclosed method, use is made of the nuScenes training and validation split -there are 700 scenes for training and 150 scenes for validation. Existing bounding boxes labels are used and radar points are appended with class labels for training the point cloud semantic segmentation task. For example, the following values may be set: N = 1024, C = 9 and r = 2, where N is the number of cloud data points, C is the predetermined number of semantic classes and r is the radius used for spatial refinement. Training was done using PyTorch with two 1080Ti graphic cards. The Adam optimiserwas used with typical parameters. The learning rate was 0.001. The selected loss function for training is cross entropy loss. Since radar point cloud semantic segmentation has few methods available, the performance of the disclosed method, named RadSeg, with other semantic segmentation techniques such as PointNet++ and Point Transformer that were retrained for radar point clouds. The disclosed method is also benchmarked with respect to an existing detection method, namely CenterPoint. Information on CentrePoint can be found in T. Yin, X. Zhou, and P. Krahenbuhl, “Center-based 3d object detection and tracking,” CVPR, 2021. For CenterPoint, the points are segmented based on the detection bounding box results. RadSeg is able to segment a greater number of points of the given class. The results on nuScenes dataset show that RadSeg performs much better for car and bicycle classes, which is a huge advantage for applications in autonomous driving. The RadarScenes dataset (available from https: / / radar-scenes.com / dataset / about / ) includes only dynamic objects being labelled. Therefore, RadSeg is able to adapt well and usually outperforms other methods (PointNet++, CenterPoint and Pointformers). In the “car” class RadSeg achieves + 25 % improvement compared to the second highest PointFormers. This can be attributed to RadSeg having velocity values as input. Fig. 8A shows a table of results using the nuScenes dataset for training and validation. The RadSeg method according to the disclosed invention shows significant improvement in accuracy for almost all classes and a substantial improvement of 6.3 up to almost 57 percentage points in the average accuracy. The class mean intersection over union (rnlOU) is also significantly improved from 1.3 up to almost 23 percentage points compared to the usual methods. Fig. 9A shows a table of results using the RadarScenes dataset for training and validation. The RadSeg method according to the disclosed invention again shows significant improvement in accuracy for all benchmark values. The accuracy for individual classes is improved between 1.2 percentage points (for background) up to 34.7 percentage points (for cars). The overall accuracy was improved by 1 percentage point. It should be noted that the improvement from 96.4 % to 97.4 % is considered a substantial improvement, since it is notoriously difficult to improve this value any further, especially for large accuracies above 95 %. The class mlOll is also significantly improved by at least 6.1 percentage points compared to the next best method. RadSeg was also tested in a real world environment to perform radar point cloud semantic segmentation using data collected from a Continental ARS540 sensor. The experiments are conducted in the Robot Operating System (ROS) environment. Based on the knowledge at the time of filing, the disclosed method is one of very few showing performance on the typical 3D radar and latest 4D radar sensors. The results are shown in Fig. 9. The left column illustrates the actual picture / driving situation from which sparse radar point cloud data 20 was obtained. The middle column illustrates the classification of the radar point cloud data before spatial refinement. The right column illustrates the radar point cloud data 20 after spatial refinement. In all examples, the spatial refinement module is able to improve the results of the overall semantic segmentation in the scene. The technique updates the results by associating radar points of similar values, in particular similar velocity values. This has helped to significantly improve the results under sparse radar conditions. Further qualitative analysis of the performance of RadSeg against the known methods shows that overall the radar res values and velocity values are beneficial for the semantic segmentation task. However, the dynamic property value (dynaprop) seems to have less effect. The performance of RadSeg on public large-scale autonomous driving datasets and real-world data proves its ability to perform point cloud semantic segmentation in autonomous driving scenarios. It provides another possibility for vehicle on-board low-resolution radar to perform high-level perception tasks. REFERENCE SIGNS 10 vehicle 12 vehicle control unit (data processing apparatus) 14 radar sensor apparatus 16 radar sensor 17 sensing area 18 object 20 point cloud data 22 input layer 24 transformer module 26 feature encoder portion 28 feature decoder portion 30 transformer input block 32 multilayer perceptron (MLP) input block 34 point transformer block 36 encoder block 38 transition down block 40 transformer center block 42 MLP block 44 decoder block 46 transition up block 48 transformed point cloud data 50 fully connected layer module 52 pre-classification data 54 fully connected layers 56 first fully connected layer 58 second fully connected layer 60 third fully connected layer 62 fourth fully connected layer 64 classification module 66 classification layer 68 concatenation layer 70 concatenated point cloud data 72 spatial refinement module 74 semantically segmented point cloud data 76 first linear layer 78 second linear layer 80 transformer layer 82 linear layer 84 MLP layer 86 linear layer 88 first combination means 90 second combination means 92 MLP layer 94 aggregation layer 96 farthest point sampling layer 98 MLP layer 100 local max pooling layer 102 first linear layer 104 second linear layer 106 interpolation layer 108 summation layer

Claims

1. A computer-implemented method for semantic segmentation of sparse point cloud data (20), the method comprising:a) providing point cloud data (20) that include a plurality of cloud data points, wherein each cloud data point is associated with coordinate data (x,y,z), that are indicative of the spatial location of the cloud data point, and velocity data (vxcomp, vycomp) that are indicative of the velocity of the cloud data point;b) feeding the point cloud data (20) to a transformer module (24) that is trained for processing the point cloud data (20) for semantic segmentation, wherein the transformer module (24) outputs transformed point cloud data;c) feeding the transformed point cloud data (48) to a fully connected layer module (50) comprising a plurality of fully connected layers (54), wherein a last fully connected layer (62) along the feed forward direction of the plurality of fully connected layers (54) outputs pre-classification data (52) that, for each cloud data point, include an associated probability value for each semantic class of a predetermined number of semantic classes of the semantic segmentation;d) feeding the pre-classification data (52) to a classification module (64) that comprises a classification layer (66) and a concatenation layer (68), wherein the classification layer (66) classifies each cloud data point into one of the semantic classes based on the pre-classification data (52) and outputs classified point cloud data, wherein the concatenation layer concatenates the unprocessed point cloud data (20) to the classified point cloud data and outputs them as concatenated point cloud data (70);e) refining spatial clustering of the concatenated point cloud data (70) based on the concatenated point cloud data, thereby resulting in semantically segmented point cloud data (74).

2. The method according to claim 1, wherein in step b) the transformer module (24) comprises a feature encoder portion (26), and the feature encoder portion (26) includes at least one point transformer block (34) and at least one encoder block (36) that is fed with the point cloud data (20) or with an output of a previous block along the feed forward direction of the transformer module (24), wherein each encoderblock (36) consists of a transition down block (38) followed by the transformer block (34).

3. The method according to any of the preceding claims, wherein in step b) the transformer module (24) comprises feature decoder portion (28), and the feature decoder portion (28) includes at least one point transformer block (34) and at least one decoder block (44) that is fed with an output the feature encoder portion (26), preferably with an output of a last encoder block (36) along the feed forward direction of the feature encoder portion (26), wherein each decoder block (44) consists of a transition up block (46) followed by the transformer block (34).

4. The method according to claim 2 and 3, wherein one of the encoder blocks (36) and one of the decoder blocks (44) are associated with each other, and the output of the associated encoder block (36), preferably the output of the transformer block (34) of the associated encoder block (36), is fed to the transition up block (46) of the associated decoder block (44).

5. The method according to any of the preceding claims, wherein in step c) the fully connected layer module (50) comprises a first fully connected layer (56) along the feed forward direction that is fed with the transformed point cloud data (48), wherein the number of features per cloud data point of the first fully connected layer (56) and the number of features per cloud data point of the transformed point cloud data (48) are the same.

6. The method according to claim 5, wherein the fully connected layer module (50) comprises a second fully connected layer (58) along the feed forward direction that has a larger number of neurons than the number of features per cloud data point, preferably double the number of neurons than the number of features per cloud data point.

7. The method according to any of the preceding claims, wherein in step c) the probability value is determined by a softmax function.

8. The method according to any of the preceding claims, wherein in step d) the classification layer (66) determines the semantic class by selecting the semantic class having the largest probability value.

9. The method according to any of the preceding claims, wherein in step e) the spatial refinement is performed by a rules based model or a k-nearest neighbours method.

10. The method according to any of the preceding claims, wherein in step a) the point cloud data (20) are provided by a computer-readable medium and / or by a radar sensor apparatus (14) that is configured to measure location and velocity of an object (18) and output point cloud data (20) that are indicative of a relative location and a relative velocity of the object (18) relative to the radar sensor apparatus (14).

11. A method for controlling a vehicle (10) or drone by performing a method according to any of the preceding claims and generating a control signal for controlling the vehicle (10) or drone based on the semantically segmented point cloud data.

12. A data processing apparatus (12) comprising means for carrying out a method according to any of the claims 1 to 10.

13. A vehicle (10) or drone comprising a data processing apparatus (12) and a radar sensor apparatus (14) for carrying out the method of claim 11.

14. A computer program comprising instructions which, when the program is executed by a data processing apparatus (12), cause the data processing apparatus (12) or the data processing apparatus (12) and a radar sensor apparatus (14) to carry out the method according to any of the claims 1 to 11.

15. A computer readable storage medium or a data carrier signal that includes the computer program according to claim 14.

Images