Method for processing a point cloud

Abstract

According to various embodiments, a method for processing a point cloud is provided, comprising receiving a point cloud containing at least one feature value for each point in a spatial region, mapping the point cloud onto a grid with a plurality of grid cells, each grid cell being assigned a respective sub-region of the spatial region, in each case according to a first mapping rule and according to a second mapping rule, determining a grid representation of the point cloud by combining, for each grid cell for which a feature value was determined according to the first mapping rule and according to the second mapping rule, the feature value determined according to the first mapping rule and the feature value determined according to the second mapping rule, and determining a processing result of the point cloud by processing the determined grid representation.

Description

[0001] The present disclosure relates to methods for processing a point cloud.

[0002] Advanced driver assistance systems and autonomous driving require a precise representation of the vehicle's surroundings. For this purpose, cameras and high-resolution (image) point-based sensors such as LiDAR or imaging radar are typically used. These sensors provide sensor data in the form of a point cloud, which is then processed to obtain information about the environment. Effective grid-based approaches exist for such processing, i.e., approaches that expect input data in a grid representation. To do this, the respective point cloud must first be converted into a grid representation, for which efficient approaches (especially with low information loss) are desirable.

[0003] The paper AH Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “PointPillars: Fast Encoders for Object Detection from Point Clouds,” May 2019, arXiv: 1812.05784, hereinafter referred to as Reference 1, describes the “PointPillars” method for converting a point cloud into a grid representation.

[0004] The publication D. Köhler, M. Quach, M. Ulrich, F. Meinl, B. Bischoff, H. Blume “Improved Multi-Scale Grid Rendering of Point Clouds for Radar Object Detection Networks”, May 2023, arXiv:2305.15836, , hereinafter referred to as reference 2, describes the “KPBEV” method for converting a point cloud into a grid representation.

[0005] According to various embodiments, a method for processing a point cloud is provided, comprising • Receiving a point cloud containing at least one feature value for points in a spatial area, mapping the point cloud onto a grid having a plurality of grid cells, each grid cell being assigned a respective sub-area of ​​the spatial area, in each case according to a first mapping rule and according to a second mapping rule, wherein ◯ the first mapping rule for determining at least one feature value of a grid cell that is assigned to a sub-area of ​​the spatial area in which several points of the point cloud are located, determines the at least one feature value depending on feature values ​​of only those points of the point cloud that are located in the sub-area and ◯ wherein the second mapping rule for determining at least one feature value of a grid cell which is assigned to a sub-area of ​​the spatial area in which at least one point of the point cloud is located, determines the at least one feature value depending on feature values ​​of the at least one point which is located in the sub-area and one or more points outside the sub-area which are located within a predetermined neighborhood of the sub-area, • Determining a grid representation of the point cloud by combining, for each grid cell for which a feature value was determined according to the first mapping rule and according to the second mapping rule, the feature value determined according to the first mapping rule and the feature value determined according to the second mapping rule and • Determining a processing result of the point cloud by processing the determined grid representation.

[0006] In other words, according to various embodiments, different methods for converting point clouds into a grid representation (such as PointPillars, see Reference 1, and KPBEV, see Reference 2) are combined to provide input for further processing, such as an object detection network. This can improve the quality of the final results (e.g., object detection).

[0007] For example, • Less information loss when converting point clouds to grids compared to using a single conversion method. • Higher detection performance and estimation of regression parameters, e.g. better detection performance for vehicles (e.g. trucks, buses) due to improved information flow from the point cloud to the grid compared to using only PointPillars or KPBEV. • Reduced conversion and scaling errors due to better capture of the local context from the point cloud. • Better scalability through a grid representation for sparse data, which limits the calculation to only relevant locations.

[0008] The process can be used for various applications, e.g. • in automated assembly systems, e.g. for detecting components and their alignment to determine the gripping point • in automated lawnmowers, e.g. for detecting objects (obstacles) • for automatic access control, e.g. for the detection and identification of persons during automatic door opening • for monitoring locations or buildings, e.g. for the detection, testing and classification of dangerous goods • for traffic monitoring with stationary radar sensors • for the detection and classification of road users in an assistance system for bicycles or other two-wheelers (motorcycles, mopeds, etc.).

[0009] Various examples of implementation are given below.

[0010] Embodiment 1 is a method for processing a point cloud as described above.

[0011] Embodiment 2 is a method according to embodiment 1, wherein the processing of the grid representation comprises processing by a neural convolution network for sparse grids.

[0012] The grid representation is typically sparse, as it originates from a point cloud (e.g., lidar or radar), and can therefore be efficiently processed using a sparse grid convolutional neural network.

[0013] Embodiment 3 is a method according to embodiment 1 or 2, wherein the first mapping rule maps the point cloud to a first compressed representation of a sparse grid and the second mapping rule maps the point cloud to a second compressed representation of a sparse grid and the grid representation of the point cloud is determined by combining the first compressed representation and the second compressed representation on a grid cell-by-grid basis.

[0014] This allows the point cloud to be converted into the grid representation in a memory-efficient manner.

[0015] Embodiment 4 is a method according to any one of embodiments 1 to 3, wherein the point cloud is a radar point cloud.

[0016] Especially for radar point clouds, efficient mapping to sparse grids is of interest, since they have a relatively small number of points.

[0017] Embodiment 5 is a method according to one of the embodiments 1 to 4, wherein the first mapping rule and / or the second mapping rule have trainable parameters and are trained by means of processing results predetermined for point clouds of the processing of the grid representations determined for the point clouds.

[0018] In other words, at least one of the mapping rules can be trainable and trained in a supervised manner, possibly together with the processing (e.g., implemented by one or more neural networks). This allows for learning an effective extraction of information from the point cloud for the respective processing (e.g., object detection).

[0019] Embodiment 6 is a data processing device configured to perform a method according to any one of embodiments 1 to 5.

[0020] Embodiment 7 is a computer program including instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 5.

[0021] Embodiment 8 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 5.

[0022] In the drawings, like reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings. Fig. 1 shows a vehicle. Fig. 2 shows an architecture for object detection according to one embodiment. Fig. Figure 3 illustrates the conversion of a point cloud into a grid representation using a combination of PointPillars and KPBEV. Fig. 4 shows a flowchart illustrating a method for processing a point cloud according to one embodiment.

[0023] The following detailed description refers to the accompanying drawings, which, by way of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0024] Various examples are described in more detail below.

[0025] Fig. 1 shows a vehicle 101.

[0026] In the example of Fig. 1, a vehicle 101, for example a car or truck, is provided with a vehicle control device (also referred to as an electronic control unit, e.g. a control device, e.g. an Electronic Control Unit (ECU)) 102.

[0027] The vehicle control device 102 has data processing components, e.g., a processor (e.g., a CPU (central processing unit)) 103 and a memory 104 for storing control software 107 according to which the vehicle control device 102 operates, and data processed by the processor 103. The processor 103 executes the control software 107.

[0028] For example, the stored control software (computer program) includes instructions which, when executed by the processor, cause the processor 103 to perform driver assistance functions (i.e., the function of an ADAS (Advanced Driver Assistance System)) or even to control the vehicle autonomously (AD (Autonomous Driving)).

[0029] The control software 107 is transferred, for example, from a computer system 105, e.g., via a network 106 (or also with the aid of a storage medium such as a memory card), to the vehicle 101. This can also occur during operation (or at least when the vehicle 101 is at the user's location), since the control software 107 is updated to new versions over time, for example.

[0030] The control software 107 can be trained, for example, using machine learning (ML). This means that the control software 107 implements, for example, a neural network (NN) 108 that is trained based on training data, in this example, from the computer system 105. The computer system 105 thus implements an ML training algorithm for training the one or more ML models 108.

[0031] The control software 107 determines control actions for the vehicle (such as steering actions, braking actions, etc.) from input data available to it that contain information about the environment or from which it derives information about the environment (such as by detecting other road users, e.g., other vehicles). This data is, for example, sensor data from one or more sensor devices 109, such as a camera of the vehicle 101, which are connected to the vehicle control device 102 via a communication system 110 (e.g., a vehicle bus system such as CAN (Controller Area Network)).

[0032] Advanced driver assistance systems (ADAS systems) and autonomous driving (AD) require an accurate representation of the vehicle's surroundings (i.e., an environmental model). For this purpose, sensor devices 109 (possibly among others) with high-resolution (image) point-based sensors such as LiDAR or imaging radar are used.

[0033] These sensors provide measurements in the form of point clouds, i.e., points in three-dimensional space with one or more feature values. For example, a lidar sensor characterizes each point by a Cartesian coordinate (x, y, z) and the reflection intensity (as a feature), while a radar sensor provides a polar coordinate (range, azimuth angle) and (as feature values) other properties such as signal strength, radar cross section (RCS), elevation angle, etc.

[0034] Based on such data (i.e., a point cloud), the position, orientation, class, and possibly other properties of relevant objects (e.g., cars, trucks, or pedestrians) can be determined (e.g., by the control software 107), e.g., within the framework of a perception algorithm. A perception algorithm typically includes a tracking step (e.g., based on a Kalman filter), followed by a classification of the detected objects into object types. For tracking, for example, a model is used that describes what the objects look like in the sensor data (e.g., a reflection model for radar sensors or an L-shaped vehicle model for lidar sensors). With the advent of deep learning, traditional perception algorithms are increasingly being replaced by neural networks, and in particular by networks for object detection, i.e.,The control software 107 uses a (deep) neural network as the ML model 108 for object detection. The neural network outputs, for example, OBBs (Oriented Bounding Boxes) with the probability of existence, size, orientation, and class of the detected objects. The detected OBBs are then tracked over time. While perception using such neural object detection networks is already widespread in lidar sensors, radar object detection networks are not yet widespread due to the smaller number of points.

[0035] Object detection networks for point-based sensors can be roughly divided into two groups: point-based methods and grid-based methods. • Grid-based methods adapt the neural networks used in image processing for object detection methods. Neural networks have long been used in image processing to detect objects and have already achieved high performance. For example, an object detection network can be used to detect faces in camera images. Grid-based methods use similar architectures to object detection networks for image processing. However, converting the point cloud into a grid representation is an essential preprocessing step. This is done, for example, by projecting all points of the point cloud onto a Cartesian grid. If a grid cell contains a point, its feature(s) is / are adopted into the grid cell (i.e., adopted as feature(s) of the grid cell).If several points fall into a grid cell, their features are aggregated, for example, using a pooling function or a PointNet. A common architecture in such grid networks is to first process the input in a feature pyramid network using a backbone network. The feature pyramid network is / is trained to represent the input data at different scales. The information represented at the individual scales is then combined before detection is performed. The neural network for object detection is trained so that each grid cell (often after the resolution of the grid has been reduced by pooling layers of the backbone network) can generate one or more object hypotheses. Each grid cell is subjected to a classification, i.e., it is determined whether it belongs to one of the objects to be detected, and regression parameters (e.g.Longitude, latitude, orientation) are estimated to characterize the OBB. • In contrast, a point-based neural object detection network operates with a point set representation, i.e., a list of points. Each point therefore contains an anchor (point) that can generate an OBB. While grid-based methods can rely on proven convolutional layers from image processing, point-based methods require other means of exchanging information between the points (such as PointNet, PointNet++, neural graph networks, continuous convolutions, or kernel point convolutions).

[0036] Currently, grid-based approaches and hybrid approaches (which leverage the strengths of both a grid-based and a point-based network) have proven to be the most efficient for radar data and are therefore crucial for radar object detection. The following describes implementation examples using a grid-based approach.

[0037] In a grid-based approach, convolutional neural networks (CNNs) are typically used in the backbone network to process grid-based data (i.e., grid-represented data from the point cloud). For example, images are dense and have a grid-like structure. However, for grid-represented data from a (lidar or radar) point cloud, the grid is typically sparse. Sparse convolutional networks are a type of neural network architecture designed to process sparse data with a small number of active input points (i.e., populated grid cells) and an irregular occurrence in space and time. They offer several advantages over conventional convolutional networks (i.e., convolutional neural networks).A key advantage is memory efficiency, as sparse convolutional networks can significantly reduce memory usage by processing only non-zero input values. This is especially useful for large datasets. Another advantage is computational efficiency. By processing only non-zero input values, sparse convolutional networks can reduce the number of computations required in both the forward and backward passes. This results in shorter training and inference times, making them well-suited for real-time and resource-constrained applications. Sparse convolutional networks also excel at processing irregularly structured data.They are ideal for processing data from point clouds, graphs, or sparse time series, where capturing and exploiting the underlying spatial or temporal structure is crucial. This enables more accurate and efficient activation. Furthermore, convolutional networks for sparse data can also be generalized to dense data. They can treat dense data as a special case of sparse data with few spaces (e.g., zeros), allowing them to effectively process both sparse and dense inputs. This flexibility makes them a versatile approach applicable to various data types.

[0038] As described above, for grid-based and hybrid approaches, converting the point cloud provided by the respective sensor into a grid representation is a crucial step. Point clouds typically contain numerous additional features and can represent fine-grained details of the environment. However, converting point clouds to a grid representation typically results in a significant amount of information being lost, as a single grid cell can contain multiple points. In this case, the information within a grid cell may be too dense or difficult to aggregate.

[0039] There are several methods for converting a point cloud into a grid representation, such as PointPillars from Reference 1 and KPBEV from Reference 2. While PointPillars only considers the median of the features of all points falling within the same grid cell to calculate the feature value(s) of the grid cell, neglecting the structural order in the point cloud, KPBEV succeeds in extracting the ordering information of the points using kernel point convolutions (KPConvs), thus reducing information loss. While KPBEV extracts more information from the point cloud and transfers it to the grid (since it also considers features of points outside the current grid cell), the type of information is different from PointPillars, as KPBEV calculates a distance-based correlation function between each kernel point and the points within a certain radius of the point cloud center.

[0040] According to various embodiments, different methods for converting point clouds to a grid representation (such as PointPillars and KPBEV) are combined. For example, both PointPillars and KPBEV are used to transform an input point cloud into the same (ultimately sparse) grid. However, PointPillars and KPBEV are only examples here, and the described approach may include a combination of other conversion methods (even more than two) for point clouds to a grid representation.

[0041] In the following, an example of an object detection is described in detail.

[0042] Fig. 2 shows an architecture for object detection, which is implemented, for example, by the control software 107 for detecting objects in the environment of the vehicle 101.

[0043] First, the input data for object detection (e.g., by one or more of the sensor devices 109) is provided in the form of a point cloud 201, which, as described above, assigns one or more feature values ​​to points in space (or in the plane, e.g., around the vehicle 101). The point cloud 201 may, but does not have to, have been preprocessed (e.g., by one or more neural networks). As described above, several different methods 203, 204 for converting the point cloud 201 into a grid representation are combined in a conversion module 202 (where the methods use the same grid) to obtain more information about the point cloud in the grid representation 205, which is supplied to the neural object detection network (here consisting of backbone network 206, feature pyramid network 207, and detection head 208).

[0044] In this example, the methods 203, 204 are PointPillars 203 and KPBEV 204. The result of combining the results of these two is the grid representation 205, i.e. a representation of a sparse grid in which each grid cell contains information from the features of PointNet and KPBEV for a particular active point, as shown in Fig. 3 is shown.

[0045] Fig. 3 illustrates the conversion of a point cloud 301 (corresponding to point cloud 201) into a grid representation 312 (corresponding to grid representation 205) using a combination of PointPillars 302 and KPBEV 303.

[0046] Using PointPillars 302 and KPBEV 303, the points of the point cloud are assigned to grid cells of the (specified) grid. The result is a respective sparse grid 304, 305. The respective sparse grid 304, 305 is then converted into a respective representation (for sparse grids) 306, 307, which contains the features for occupied grid cells 308, 309 and their (grid cell) indices 310, 311. Since the same point cloud is processed on both paths (via PointPillars and via KPBEV), the set of indices 310, 311 is identical, and the features can be added (index-wise) to obtain the grid representation 312.

[0047] The combination of the two conversion methods 203, 204 improves the flow of information from the point cloud 201 to the grid representation 205, since PointPillars and KPBEV capture different features for the same grid cell. In this way, information losses during the conversion to the grid representation can be avoided (or at least reduced). The methods 203, 204 can have trainable parameters (as is the case with PointPillars and KPBEV), and the conversion module 202 can learn to extract the most important information for each grid cell. Furthermore, the features are converted into a grid representation 205 (according to a sparse grid), which can be processed by a submanifold convolutional neural network as the backbone network 206, which is computationally more efficient for sparse data such as radar point clouds.

[0048] The point cloud 201, for example, is a list of unordered points. The order is irrelevant. Each point is characterized by one or more properties (one or more feature values), e.g., its radar cross section (RCS) or the radial velocity, which is compensated by the radar's own motion. The output of the object detection network, which processes the grid representation 205 (i.e., the information from the point cloud projected onto the grid (point by point)), is a list of object hypotheses. For each object hypothesis, for example, a classification of the object type, the position of the object, and the dimensions (length, width, height, orientation) of an OBB surrounding the object are predicted. The input of the conversion module 202 is thus a list of unordered points.Its output (grid representation 205) is a representation of an n-dimensional (typically 2- or 3-dimensional) sparse grid covering a certain (spatial or plane) area. Sparse means that some (few) cells of the grid contain information, while some other cells remain empty. The resolution of the grid determines the (2D or 3D) area of ​​each grid cell and thus the number of (grid) cells of the grid. The output of the conversion module 204 is, in this example (as shown in . Fig. 3 right) a (compressed) representation for sparse grids, with a first matrix 313 in which each entry represents the feature of an occupied grid cell and a second matrix 314 which specifies the indices of the grid cells with the entries from the first matrix 313.

[0049] In PointPillars 302, points are grouped into spatial columns (i.e., cuboid-shaped pillars), with the points of each column being assigned to a respective cell of a 2D grid. The feature(s) of each grid cell are individually determined (embedded) by a fully connected neural network. If multiple points fall into the same column (i.e., the same grid cell), mean pooling is applied across all points within the column to obtain a feature vector of a fixed length. KPBEV 303 receives the same input point cloud 301 as PointPillars 302, but also considers points from neighboring grid cells when calculating features for a grid cell, which allows for better capture of the local context: For each non-empty (i.e., active) grid cell, an anchor point is placed in the center of the grid cell. For each anchor point, a spherical neighborhood is encoded with a set of kernel points.These kernel points carry learnable weight matrices W. k and are evenly distributed around the anchor point. For each kernel point, a multiplication with the weight matrix of the respective kernel point and a distance-based correlation function is calculated. The resulting features (per kernel point) are then combined (added) and assigned to the anchor point in the center of the cell.

[0050] The results of the conversion processes 302, 303 are brought into a compressed form, ie a respective grid representation 306, 307, with a first matrix 308, 309, in which each entry represents a non-zero grid cell of the respective conversion process, and a second matrix 310, 311, which indicates the indices of the occupied grid cells in the grid.

[0051] To combine the grid representations 306, 307 into the grid representation 312 of the point cloud, the feature matrices 308, 309 are normalized, and each value of the feature matrix 308 of the PointPillars method is added to the value at the corresponding position of the feature matrix 309 of the KPBEV method, resulting in a feature matrix 313 (with entries for the same indices 314 as the feature matrices 308, 309) for the grid representation 312, which corresponds to the addition of the PointPillars results to the KPBEV results. However, the combination can also be performed in a more complex way as a simple addition, e.g., as a weighted sum, etc.

[0052] The abstract features extracted from the point cloud 201, 301 are then processed by the object detection network, which, for example, is or contains one or more submanifold convolutional networks for sparse grids (e.g., as backbone network 206). The backbone network 206 is, for example, a U-network. The detection head 208 (or multiple detection heads 208), e.g., an additional neural convolutional network, estimates an object probability between 0 and 1 for each grid cell (of the respective feature map it receives as input) and the regression parameters for an OBB (position, length, width, height, orientation). Several of these detection heads 208 are used to detect different object types, each of which is responsible for predicting an object type class, i.e., object types with similar properties such as trucks and buses.These detection heads 208 use feature maps with appropriate resolution as input according to the object types to be detected. For example, a feature map with a higher resolution is used for small objects such as pedestrians than for large objects such as trucks. Since an object can extend across multiple grid cells, according to one embodiment, the generated object hypotheses are filtered by voting. This is done, for example, by non-maximum suppression (NMS). For each object, the spatially overlapping object hypotheses are filtered according to the one with the highest object probability. The filtered object hypotheses in the form of an OBB are the output of the detection network.

[0053] The output of the object detection network can also include semantic segmentation, self-localization, dynamic and stationary environment, e.g., drivable areas, an occupancy map, or a visibility grid.

[0054] Backbone 206, feature pyramid network 207 and / or detection head(s) 208 can be chosen differently for any application (as long as the resulting network can process a sparse grid as input).

[0055] The grid can have different dimensions depending on the conversion method 203, 204, e.g. 2D bird's eye view, spherical (distance-azimuth) or 3D Cartesian.

[0056] In summary, according to various embodiments, a method is provided as described in Fig. 4 shown.

[0057] Fig. 4 shows a flowchart 400 illustrating a method for processing a point cloud according to one embodiment.

[0058] In 401, a point cloud (determined by one or more sensors) (ie a sensor data point cloud) is received, for example from a control device of a vehicle, which contains at least one feature value for each point in a spatial area.

[0059] In 402, the point cloud is mapped onto a (predetermined) grid having a plurality of grid cells according to a first mapping rule and a second mapping rule, each grid cell being assigned a respective sub-area (e.g. cubes or “columns” in the case of projection onto a 2D grid) of the spatial area.

[0060] The first mapping rule (e.g. PointPillars) determines at least one feature value of a grid cell (i.e. an anchor point of the grid cell) that is assigned to a sub-area of ​​the spatial area in which several points of the point cloud are located, the at least one feature value depending on feature values ​​of only those points of the point cloud that are located in the sub-area (i.e. the result for the grid cell depends only on the features of the points that are projected onto this grid cell).

[0061] The second mapping rule (e.g. KPBEV) determines at least one feature value of a grid cell (i.e. an anchor point of the grid cell) that is assigned to a sub-area of ​​the spatial area in which at least one point of the point cloud is located, the at least one feature value depending on feature values ​​of the at least one point that is located in the sub-area and one or more points outside the sub-area that are located within a predetermined neighborhood of the sub-area (i.e. it is determined whether one or more points outside the sub-area are present that are located within the predetermined neighborhood of the sub-area (which extends around the sub-area) and, if necessary, taken into account when calculating the feature for the grid cell).

[0062] In 403, a grid representation of the point cloud is determined by combining, for each grid cell for which a feature value was determined according to the first mapping rule and according to the second mapping rule, the feature value determined according to the first mapping rule and the feature value determined according to the second mapping rule.

[0063] In 404, a processing result of the point cloud is determined by processing the obtained grid representation (ie, the point cloud is processed by first converting it into the grid representation, which is then further processed, e.g., by a neural object detection network).

[0064] The procedure of Fig. 4 may be performed by one or more computers having one or more data processing units. The term “data processing unit” may be understood as any type of entity that enables the processing of data or signals. The data or signals may, for example, be handled according to at least one (i.e., one or more than one) specific function performed by the data processing unit. A data processing unit may include or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA) integrated circuit, or any combination thereof.Any other way of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the method steps described in detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0065] According to various embodiments, the method is therefore particularly computer-implemented.

[0066] After training, the neural network can be applied to sensor data acquired by at least one sensor. The output of the neural network thus provides a result about a physical state of an environment of the at least one sensor and / or the at least one sensor itself, or the method can include using the output of the trained neural network, which it provides in response to an input of sensor data, as such a result.

[0067] In other words, the method may comprise inferring or predicting the physical state of an existing real object from measurements of physical properties (i.e., sensor data concerning the object) by means of the trained neural network (i.e., its output in response to the sensor data).

[0068] The approach of Fig.4, for example, is used to generate a control signal for a robotic device. The term "robotic device" can be understood as referring to any technical system (with a mechanical part whose movement is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system. A control rule for the technical system is learned, and the technical system is then controlled accordingly.

[0069] For example, after training, the neural network is used to select a control action and generate a control signal for the vehicle. A control action for which one or more control signals are generated could be, for example, a distance (e.g., moving one meter to the right), a speed, an acceleration, etc.

[0070] Various embodiments may receive and use (e.g., time series of) sensor data from various sensors such as video, radar, LiDAR, ultrasound, motion, thermal imaging, etc. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] AH Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “PointPillars: Fast Encoders for Object Detection from Point Clouds,” May 2019, arXiv: 1812.05784

[0003] D. Köhler, M. Quach, M. Ulrich, F. Meinl, B. Bischoff, H. Blume „Improved Multi-Scale Grid Rendering of Point Clouds for Radar Object Detection Networks“, May 2023, arXiv:2305.15836

[0004]

Claims

[1] A method for processing a point cloud (201, 301), comprising: Receiving a point cloud (201, 301) containing at least one feature value for points in a spatial area; Mapping the point cloud onto a grid having a plurality of grid cells, each grid cell being assigned a respective sub-area of the spatial area, in each case according to a first mapping rule (203, 302) and according to a second mapping rule (204, 303), wherein the first mapping rule (203, 302) for determining at least one feature value of a grid cell which is assigned to a sub-area of the spatial area in which several points of the point cloud (201, 301) lie, determines the at least one feature value depending on feature values of only those points of the point cloud (201, 301) which lie in the sub-area; and wherein the second mapping rule (204, 303) for determining at least one feature value of a grid cell which is assigned to a sub-area of the spatial area in which at least one point of the point cloud (201, 301) lies, determines the at least one feature value depending on feature values of the at least one point which lies in the sub-area and one or more points outside the sub-area which are located within a predetermined neighborhood of the sub-area; Determining a grid representation (205, 312) of the point cloud (201, 301) by combining, for each grid cell for which a feature value was determined according to the first mapping rule (203, 302) and according to the second mapping rule (204, 303), the feature value determined according to the first mapping rule (203, 302) and the feature value determined according to the second mapping rule (204, 303); and Determining a processing result of the point cloud (201, 301) by processing the determined grid representation (205, 312). [2] The method of claim 1, wherein the processing of the grid representation (205, 312) comprises processing by a sparse grid convolutional neural network. [3] Method according to claim 1 or 2, wherein the first mapping rule (203, 302) maps the point cloud (201, 301) to a first compressed representation (306) of a sparse grid and the second mapping rule (204, 303) maps the point cloud (201, 301) to a second compressed representation (307) of a sparse grid and the grid representation (205, 312) of the point cloud (201, 301) is determined by combining the first compressed representation and the second compressed representation on a grid cell-by-grid basis. [4] Method according to one of claims 1 to 3, wherein the point cloud (201, 301) is a radar point cloud. [5] Method according to one of claims 1 to 4, wherein the first mapping rule (203, 302) and / or the second mapping rule (204, 303) have trainable parameters and are trained by means of processing results predetermined for point clouds (201, 301) of the processing of the grid representations (205, 312) determined for the point clouds (201, 301). [6] Data processing device (102) which is arranged to carry out a method according to one of claims 1 to 5. [7] A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 5. [8] A computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 5.

Images