Method for information extension and method for processing a point cloud of a radar sensor

Abstract

The invention relates to a method (10) for information expansion of a point feature (12) of each point (14) of a point cloud (16) of at least one radar sensor (18), comprising the following steps: providing (20) the point cloud (16) consisting of the points (14) having the respective point feature (12), at least including distance information (d) and velocity information (v); encoding (22) the point (14) into at least one d-v space (24) expanded by at least the distance information (d) and the velocity information (v); calculating an extended point feature (40) for each of the points (14) on the basis of the points (14) encoded into the d-v space (24); the expanded point feature (40) is added (50) to an existing point feature (12) of the point (14) of the point cloud (16). The invention further relates to a method (52) for processing a point cloud (16) of a radar sensor (18).

Description

Technical Field

[0001] This invention relates to a method for information expansion. Furthermore, this invention also relates to a method for processing point clouds from radar sensors. Background Technology

[0002] Driver assistance systems or autonomous driving functions used in vehicles require accurate representation of the vehicle's environment. To this end, in addition to camera devices, lidar and radar sensors are used, providing point clouds with spatial coordinates and point features such as signal strength, elevation angle, and azimuth angle. Environment detection models, using the point cloud as input data, then calculate, for example, the directed bounding boxes (OBBs) of environmental objects, their location, orientation, or object category. Here, the environment detection model is based on trained artificial neural networks, particularly object recognition networks, which can be further subdivided into grid-based and point-based object recognition networks.

[0003] Grid-based object recognition networks typically operate on Cartesian grids and project points onto a two-dimensional grid, while point-based object recognition networks utilize spatial distances to exchange information.

[0004] It is known that the measurement signals from radar sensors are processed as follows: first, the measurement signals are transformed into a dv spectrum (relative range / relative velocity); then, the reflections identified in this spectrum are estimated in terms of azimuth and elevation. While the estimates of relative range and relative velocity are reliable, angle estimation is often prone to error. Summary of the Invention

[0005] According to the present invention, a method for information expansion is proposed, the method comprising the following steps:

[0006] Provides a point cloud composed of points, each with its own point features, including at least distance and velocity information.

[0007] The points are encoded into at least one dv space expanded from at least the distance information and the velocity information.

[0008] Based on the points encoded into the dv space, the extended point features of each point are calculated.

[0009] The extended point features are added to the existing point features of the points in the point cloud.

[0010] Therefore, point clouds can be expanded using reliable and more accurate information. Using point clouds as input data provides more accurate and reliable results.

[0011] Radar sensors can be configured for environmental detection. They can be used in automated driving functions and / or driver assistance systems for vehicles; in automated assembly systems, such as to identify components and their orientation to determine gripping points; in automated lawnmowers, such as to identify objects (obstacles); in automated access control, such as to identify and verify personnel for automatic door opening; in object monitoring, such as to identify, inspect, and classify hazardous materials; and in traffic monitoring and / or in assistance systems for bicycles or other two-wheeled vehicles to identify and classify traffic participants.

[0012] A radar sensor can have multiple radar antennas arranged in an antenna array. The radar sensor can be an FMCW sensor. The radar sensor can also perform environmental detection at a certain azimuth and / or elevation angle.

[0013] A point cloud can indicate at least one environmental object. Environmental objects can be vehicles, organisms, plants, devices, objects, and / or buildings.

[0014] A point cloud can consist of measurements taken by a radar sensor at a single point in time, or a combination of multiple measurements taken by the radar sensor at different times. A point cloud can also consist of multiple measurements from one or more radar sensors.

[0015] Points in a point cloud can exist as an unordered list. These points are calculated based on reflections taken by a radar sensor during measurements, by performing distance calculations and angle estimations on these reflections.

[0016] In addition to spatial coordinates, such as Cartesian or polar coordinates, point features may also include radar cross-section, azimuth and / or elevation.

[0017] Range information can correspond to the relative distance from the dv spectrum of the radar sensor. Velocity information can correspond to the relative velocity from the dv spectrum of the radar sensor. Range and velocity information can be calculated, for example, based on the FMCW signal of the radar sensor. Range information can be the relative distance of the reflection assigned to that point relative to the radar sensor, and velocity information can be the relative velocity of the reflection assigned to that point relative to the radar sensor.

[0018] Encoding from point to dv space can be a transmission from point to dv space, for example, by using distance and velocity information attributes already assigned to each point.

[0019] Extended point features can be added to existing point features by attaching extended point features to existing point features.

[0020] If the method for information expansion is applied to multiple radar sensors, it can be executed for each individual radar sensor as described. The point clouds of these radar sensors, containing both point features and expanded point features, can then be merged. Alternatively, in the case of forming a common point cloud, the method can be applied to pre-merged points in the point clouds of these radar sensors. Here, the assignment of points to their respective radar sensors can be achieved through additional point features, such as point features with sensor location or sensor ID.

[0021] In a preferred embodiment of the invention, it is advantageous that the encoding of a point into at least one dv space comprises: first, transforming the point into a latent feature space consisting of multiple feature vectors; and then, mapping these feature vectors to their respective dv spaces. The feature vectors may have a dimension corresponding to the number of points.

[0022] Points can be transformed into a latent feature space using a fully connected network.

[0023] In a particular embodiment of the invention, it is advantageous to map the feature vectors into the dv space based on the distance and velocity information assigned to each point with point features. Therefore, the original distance and velocity information are also incorporated into the feature space.

[0024] In an advantageous embodiment of the invention, the corresponding dv space is subdivided into spatial cells in a grid manner, and the point-to-dv space encoding includes: assigning the encoded point to a spatial cell in the dv space corresponding to the distance and velocity information from the point features. Point-to-dv space encoding may include: assigning the feature vector value assigned to each point to a spatial cell in the dv space corresponding to the distance and velocity information from the point features.

[0025] In an advantageous embodiment of the invention, if at least two points should be assigned to the same spatial cell when mapping feature vectors to the dv space, then these points are merged. This merging can be achieved using pooling techniques known from convolutional neural networks, particularly mean pooling or max pooling.

[0026] A preferred embodiment of the invention is advantageous in which extended point features for each point are computed by applying a trained artificial neural network, using points encoded in the dv space as input data. The trained artificial neural network can be a convolutional neural network or a sub-form, such as a PointPillar, graph neural network, or kernel-point convolution. The extended point features for each point can be computed by applying the trained artificial neural network, using feature vectors mapped into the dv space as input data.

[0027] In a preferred embodiment of the present invention, feature vectors encoded into the corresponding dv space are configured to form input data. This input data can exist as a feature map unfolded in the dv space.

[0028] In an advantageous embodiment of the invention, a feature map is created by applying a trained artificial neural network, and then extended point features are extracted for each point based on this feature map. The feature map can be expanded along the dimensions of distance and velocity information. The number of feature maps can be pre-defined, specifying the number of extended point features for each point.

[0029] Extraction can be performed as follows: In the feature map, identify and extract the spatial cells corresponding to the points.

[0030] In an advantageous embodiment of the invention, the extended point features are calculated by applying at least one learning algorithm to the encoded points in at least one dv space. Distances between points in the dv space can be considered in this learning algorithm. The calculation of extended point features for each point can be performed by applying a trained artificial neural network, using points encoded in the dv space as input data, through a convolutional neural network, a graph neural network, and / or a Transformer model that respectively utilizes distances in the dv space. In the case of a convolutional neural network, the feature map is unfolded in the dv dimension, thereby exchanging information through convolutional kernels that consider the spatial structure of points in the dv space. In the case of a graph neural network, points are connected to a certain number, possibly a finite number, of spatially nearest points. Information exchange in the learning algorithm occurs locally between the connected points. In the case of a Transformer model with local self-attention / cross-attention, associations with a finite number of spatially nearest points are utilized. The learning algorithm is based on associations or attention to a finite number of spatially nearest points in the dv space.

[0031] According to the present invention, a method for performing the processing is also provided, the method comprising the following steps:

[0032] Provides a point cloud composed of points, each point having its own point features and its own extended point features added by the method for information extension according to the present invention.

[0033] By applying at least one object detection model based on a trained artificial neural network, at least one object parameter of an environmental object is calculated using point cloud as input data.

[0034] Object parameters can be assigned to semantic segmentation, object classification, and / or object recognition of environment objects. Object parameters can indicate the bounding box of the environment object, the object's location, at least one dimension, especially length, width, or height, or the object's orientation.

[0035] Object detection models can include convolutional neural networks, Transformer models, or graph neural networks.

[0036] The method for information expansion and / or the method for processing may be computer-implemented. Furthermore, a computer program is proposed having machine-readable instructions executable on at least one computer, which, when executed, run the method for information expansion and / or the method for processing. Additionally, a memory unit is proposed, implemented in a machine-readable manner and accessible by at least one computer, on which the aforementioned computer program is stored.

[0037] Other advantages and advantageous configurations of the invention will become apparent from the description and drawings. Attached Figure Description

[0038] The invention will now be described in detail with reference to the accompanying drawings. Specifically, the drawings show:

[0039] Figure 1 A method for expanding information in a specific embodiment of the present invention.

[0040] Figure 2 A method for performing processing in a specific embodiment of the present invention. Detailed Implementation

[0041] Figure 1 A method for information expansion is illustrated in a specific embodiment of the present invention. The method 10 for expanding the information of point features 12 of individual points 14 of a point cloud 16 of at least one radar sensor 18 includes: providing a point cloud 16 consisting of 20 points 14 each having its own point features 12, wherein at least spatial Cartesian coordinates x, y, z, distance information d corresponding to the relative distance from the dv spectrum of the radar sensor 18, and velocity information v corresponding to the relative velocity from the dv spectrum of the radar sensor 18, and further particularly including a radar cross-section A. The point cloud 16 may be calculated based on measurement signals from a radar sensor 18 having multiple radar antennas.

[0042] Subsequently, point 14 is encoded 22 into at least one dv space 24 expanded from at least the distance information d and the velocity information v. Here, the encoding 22 of point 14 into at least one dv space 24 includes: first, transforming point 14 26 into a latent feature space 30 composed of multiple feature vectors 28; then, mapping each of these feature vectors 28 32 into its own dv space 24. Specifically, this is achieved by the PointPillar module 33, which maps each of the feature vectors 28 32 into the dv space 24 based on the distance information d and the velocity information v assigned to each point 14 with point feature 12.

[0043] The corresponding dv space 24 is subdivided into spatial cells 34 in a grid manner, and the encoding 22 of point 14 into dv space 24 includes: assigning the encoded point 14 to 36 spatial cells 34 in dv space 24 corresponding to the distance information d and velocity information v from point feature 12, respectively; and if at least two points 14 should be assigned to the same spatial cell 34 when mapping feature vector 28 to dv space 24, these points can be merged, for example, by pooling known from convolutional neural networks, especially mean pooling or max pooling.

[0044] Furthermore, based on the point 14 encoded in the dv space 24, specifically based on the feature vector 28 encoded in the corresponding dv space 24, the extended point features 40 of each point 14 are computed. The extended point features 40 of each point 14 are computed by applying a trained artificial neural network 42, using the point 14 encoded in the dv space 24 and, here, the feature vector 28 encoded in the corresponding dv space 24, as input data 44.

[0045] By applying neural network 42, for example, a convolutional neural network, a feature map F is created as the output. The feature map has the same dimensions as the input data 44, specifically distance information d and velocity information v. Here, the convolutional neural network can perform two-dimensional convolution, batch normalization, and ReLU activation.

[0046] Subsequently, based on the feature map F, 48 extended point features 40 are extracted for each point 14, in particular by identifying and extracting the spatial cells 34 corresponding to point 14 in the feature map F according to the association relationships 49 created during mapping 32. Here, the number n of feature maps F corresponds to the number n of extended point features 40 for each point 14.

[0047] Subsequently, the extended point feature 40 is added to the existing point feature 12 of point 14 in point cloud 16 by concatenation.

[0048] Figure 2 This illustrates a method for processing in a specific embodiment of the invention. The method 52 for processing a point cloud 16 of a radar sensor for detecting environmental objects, having a plurality of points 14, includes: providing 54 a point cloud 16 composed of points 14, each point having its own point feature 12 and its own pass, for example as... Figure 1 The method 10 for information expansion adds extended point features 40; and calculates at least one object parameter 56 of the corresponding environmental object by applying at least one object detection model 58 based on a trained artificial neural network, with point cloud 16 as input data 60.

[0049] First, points 14 of the point cloud 16 are encoded into a space 62 that is expanded by spatial coordinates x and y and subdivided into spatial cells 34 in a grid-like manner, for example, by encoding into a feature map 64 via a Point Pillar module. The feature map 64 forms the input to the Residual Network (ResNet) and the Feature Pyramid Network 66 so that features can be extracted from these feature maps 64 at different resolutions 68.

[0050] In the classification head 70, a convolutional neural network estimates the object probability between 0 and 1 for each spatial cell 34 of the feature map 64, as well as regression parameters for the directed bounding box (OBB), namely position, length, width, height, and orientation. To identify different object types, multiple classification heads 70 are used, where each classification head is responsible for predicting a specific object category, i.e., several object types with similar characteristics, such as trucks and buses.

[0051] Since an environment object may span multiple spatial cells 34, the generated object hypotheses are filtered by voting. This is achieved, for example, using non-maximum suppression (NMS). For each environment object, spatially overlapping object hypotheses are filtered based on the object hypothesis with the highest object probability. The filtered object hypotheses 72, here in OBB form, are the results of object detection model 58.

Claims

1. A method (10) for information extension of point features (12) of individual points (14) of a point cloud (16) of at least one radar sensor (18), the method having the following steps: providing (20) the point cloud (16) consisting of the points (14) with their respective point features (12), which at least include distance information (d) and velocity information (v), encoding (22) the points (14) into at least one d-v space (24) unfolded by at least the distance information (d) and the velocity information (v), calculating an extended point feature (40) for each of the points (14) on the basis of the points (14) encoded into the d-v space (24), adding (50) the extended point feature (40) to the existing point feature (12) of the points (14) of the point cloud (16).

2. The method (10) for performing information extension according to claim 1, characterized in that: The encoding (22) of the points (14) into at least one d-v space (24) includes first converting (26) the points (14) into a latent feature space (30) consisting of a plurality of feature vectors (28) and subsequently mapping (32) the feature vectors (28) respectively into their own d-v space (24).

3. The method (10) for performing information extension according to claim 2, characterized in that: The mapping (32) of the feature vectors (28) respectively into the d-v space (24) is based on the distance information (d) and the velocity information (v) assigned to each of the points (14) with the point features (12).

4. The method (10) for performing information extension according to any of the preceding claims, characterized in that: The respective d-v space (24) is subdivided in a grid-like manner into spatial cells (34) and the encoding (22) of the points (14) into the d-v space (24) includes assigning (36) the encoded points (14) to spatial cells (34) in the d-v space (24) corresponding to the distance information (d) and the velocity information (v) from the point features (12).

5. The method (10) for performing information extension according to claim 2 or 3 and 4, characterized in that: If at least two points (14) should be assigned to the same spatial cell (34) when mapping (32) the feature vectors (28) respectively into the d-v space (24), these points are merged.

6. The method (10) for performing information extension according to any of the preceding claims, characterized in that: The extended point feature (40) for each of the points (14) is calculated by applying a trained artificial neural network (42) with the points (14) encoded into the d-v space (24) as input data (44).

7. The method (10) for performing information extension according to any one of claims 2, 3 or 5 and 6, characterized in that: The feature vectors (28) encoded into the respective d-v space (24) form the input data (44).

8. The method (10) for performing information extension according to any of the preceding claims, characterized in that: The extended point feature (40) for each of the points (14) is extracted (48) from a feature map (F) created by applying the trained artificial neural network (42).

9. The method (10) for performing information spreading according to any of the preceding claims, characterized in that: The extended point feature (40) is calculated by at least one learning algorithm applied on the encoded points (14) in at least one d-v space (24).

10. A method (52) for processing a point cloud (16) having a plurality of points (14) of a radar sensor (18) for detecting at least one environmental object, the method having the following steps: providing (54) the point cloud (16) consisting of the points (14) having a respective point feature (12) and a respective extended point feature (40) added by a method (10) for information extension according to any one of the above claims, calculating at least one object parameter (56) of the environmental object by applying at least one object detection model (58) established on the basis of a trained artificial neural network with the point cloud (16) as input data (60).

Images