Point cloud processing, object detection and environmental sensor system

Abstract

The invention relates to a point cloud processing system (22) for processing point clouds (24), comprising the steps of providing a point cloud (24) that has several points (26) which in turn are calculated from sensor data (28) of an environmental sensor for an environment having at least one environmental object and have several features (34), grid subdivision (38) of the point cloud (24) into a point cloud grid (40) which is spanned by several grid cells (42) having the points (26), output data calculation (46) with which, for at least several of the grid cells (42), a statistically aggregated representation (49, 66, 70) is carried out over selected points (48) that comprise at least the points (26) of the grid cell (42), cell-related at the respective grid cell (42') and for at least several of the features (34, 54, 56), feature-related at the respective feature (34, 54, 56).The output data (20) is used to generate input data (18) for an object detection model (16) for object detection (10) of at least one environmental object. The invention further relates to an object detection system (10) and an environmental sensor system.

Description

[0001] The invention relates to point cloud processing according to claim 1. Furthermore, the invention relates to object detection and an environmental sensor system. State of the art

[0002] Automated driver assistance systems and autonomous driving require a precise representation of the vehicle's surroundings, which is generated by environmental sensors such as cameras and radar sensors. Point clouds, consisting of points associated with radar reflections, can be calculated from the sensor data of a radar sensor. Each point can be specified by spatial coordinates and may have other characteristics.

[0003] The processing of these point clouds is achieved through perception using object detection models to determine object parameters of the surrounding objects, such as position, orientation, classification, and other properties. With the introduction of deep learning, traditional algorithms are increasingly being replaced by neural networks that represent the object parameters of the surrounding objects as Oriented Bounding Boxes (OBBs).

[0004] Point clouds from radar sensor data show similarities to point clouds from other sensors such as LiDAR sensors, but differ in their lower point density and heterogeneous assignment of features to the points, i.e., varying availability of features at the points. Disclosure of the invention

[0005] According to the present invention, point cloud processing with the features of claim 1 is proposed. This allows structured, aggregated information from multiple points in the point cloud to be captured and processed, although the information is limited to a selection of all points in the point cloud. Missing features at the points have less of an impact on the output data. The robustness of the output data with respect to missing features at the points can be increased. Furthermore, measurement errors and signal noise have less influence on the output data. Additionally, a large number of environmental objects can be captured in the point cloud.

[0006] The environmental sensor can be a radar sensor, a LiDAR sensor, a camera, an ultrasonic sensor, or another environmental sensor that provides point cloud data.

[0007] The environmental sensor can be arranged in a device, in particular a vehicle, for example a motor vehicle or a bicycle, preferably a mobile device, in particular a robot or a stationary device, in particular a traffic infrastructure device, for example for traffic monitoring.

[0008] The environment can be the environment of the device containing the environmental sensor, for example, a vehicle environment.

[0009] The surrounding object can be an item, a device, a facility, a building, a living being, a plant, or a vehicle.

[0010] The points can correspond to reflections of the sensor signal from the environmental sensor in the surroundings. The points can be calculated from spectral data, such as spectral distance-velocity information, in particular a range-Doppler spectrum, from the environmental sensor. The spectral data can be calculated from raw signals, such as time-series signals, from the environmental sensor.

[0011] The features of the points can specify the properties of the associated reflection or of the surrounding object causing the reflection. The features can include at least one spatial coordinate, in particular a Cartesian or polar coordinate, for example an elevation angle, a distance, a velocity, in particular a Doppler velocity, a radar cross-sectional area, a signal strength, and / or other information.

[0012] The points can exhibit a heterogeneous distribution of characteristics among themselves. This heterogeneous distribution contrasts with a homogeneous distribution, where all points have the same number of characteristics. A heterogeneous distribution can also manifest as a varying availability of characteristics among the points. For example, a heterogeneous distribution might occur if some points have only one characteristic, while others have a second characteristic and / or no characteristic at all.

[0013] The point cloud grid can be one-dimensional, two-dimensional, three-dimensional, or higher-dimensional. At least one dimension of the point cloud grid can be a spatial coordinate. The point cloud grid can comprise at least two dimensions, each with a spatial coordinate. At least one dimension of the point cloud grid can be a velocity, in particular a Doppler velocity.

[0014] The grid cells of the point cloud grid can all be of the same size.

[0015] The grid subdivision can also be referred to as grid rendering.

[0016] The features of the points can be scaled or normalized before the representation is performed.

[0017] The aggregated representation is a compact, structured statistic about the characteristics of the selected points. The statistic can specify statistical properties of the point characteristics, such as a frequency distribution or means.

[0018] The representation can be performed either cell-based for the relevant grid cell among several grid cells, or feature-based for the relevant feature among several features of the selected points. The representation can be performed for all features of the selected points within the relevant grid cells. Alternatively, the representation can be performed for all features of the point cloud grid within the relevant grid cells. The total number of representations performed can be the number of relevant grid cells multiplied by the number of relevant features.

[0019] The representations can thus aggregate selected points with varying numbers of points. Specifying a maximum number of points, as in conventional point-based algorithms, is unnecessary. Furthermore, prior grouping of the points, independent of any grid subdivision, is not required for the representation.

[0020] If there are no points in the grid cell for which the representation is performed, the output data can be zero. If a point in the grid cell lacks the feature for which the representation is performed, the representation can reduce the influence of the missing feature on the output data. No default values ​​need to be used for the missing feature, as is the case with conventional point-based algorithms. Furthermore, it is unnecessary to specify a fixed number of features that must be fulfilled for all selected points, as is the case with conventional point-based algorithms.

[0021] The output data can be assigned to the respective grid cell. The output data of each grid cell can collectively form total output data. The total output data can have an output dimension with a dimensionality that depends on the number of grid cells for which the representation was calculated. One dimensionality of the output dimension can correspond to the number of grid cells in the point cloud grid.

[0022] The output data can have two or more dimensions.

[0023] Object acquisition can include object recognition, object classification and / or semantic segmentation.

[0024] The object detection model can be a neural network. The object detection model can be trained using deep learning. The object detection model can be a convolutional neural network (CNN), a recurrent neural network (RNN), a feedforward network, in particular a multilayer perceptron (MLP), a graph neural network (GNN), an autoencoder, or a transformer model. The neural network can have multiple layers, including an input layer, at least one intermediate layer, and an output layer. The neural network can have at least one fully connected layer, in which the neurons are fully connected to neurons in a preceding and / or subsequent layer. The neural network can employ at least one pooling operation. An activation function, such as ReLU, sigmoid, or Tanh, can be applied in the layers.

[0025] In a preferred embodiment of the invention, it is advantageous if the output data has a first dimension with a first dimension size that depends on the number of features in which a representation is performed. The sensor data can have more features than are considered for the respective representations. The sensor data can have as many features as are considered for the respective representations.

[0026] The first dimension size can be the number of values ​​in the first dimension. The first dimension size can correspond to the number of features for which a representation is performed.

[0027] The first dimension can specify the values ​​of the representation assigned to the respective characteristic.

[0028] In a particular embodiment of the invention, it is advantageous if the representation is a histogram divided into intervals. The number of intervals can be independent of the number of points at the selected points. The number of intervals in the histogram can be globally defined. The number of intervals between the histograms of the respective grid cells can remain constant.

[0029] A first representation for a first feature can be a first histogram with a first number of intervals, and a second representation for a second feature can be a second histogram with a second number of intervals. The first number can be the same as or different from the second number.

[0030] The representation can also be a vector with statistical characteristics and a vector length. The statistical characteristics can be a mean and / or a variance.

[0031] In a specific embodiment of the invention, it is advantageous if the histogram indicates a frequency distribution of the relevant feature of the selected points over the intervals. An example of a histogram is the frequency distribution of a Doppler velocity as a feature of the selected points. The histogram can summarize the values ​​in a fixed number of intervals (bins).

[0032] In a particular embodiment of the invention, it is advantageous if the output data has a second dimension that depends on the number of intervals in the histogram. This second dimension is, in particular, independent of the number of points at the selected points. The histograms for each of the respective features of the selected points in the relevant grid cell can have the same or a different number of intervals. The second dimension can be a constant number of intervals. Alternatively, the second dimension can correspond to the maximum number of intervals across the histograms.

[0033] The second dimension size can be a number of values ​​in the second dimension.

[0034] The second dimension can specify the values ​​of the representation assigned to the respective interval.

[0035] If the representation is a vector with statistical characteristics and a vector length, the second dimension can depend on the vector length, in particular be formed by the vector length.

[0036] In a preferred embodiment of the invention, the output data comprises an output matrix spanning the first and second dimensions. The output matrix is ​​assigned to the respective grid cell. The output data for each grid cell can be calculated from the representations by transferring the values ​​of the representations into the output matrix. The output data corresponds, in particular, to the output matrix.

[0037] Alternatively, the output data can be formed as an output vector derived from the output matrix by vectorization. The output vector can have a vector length corresponding to the product of the first dimension and the second dimension.

[0038] In a specific embodiment of the invention, it is advantageous if the output data, as total output data, comprises all output matrices of the multiple grid cells of the point cloud grid, in each of which the representation is performed. The total output data can form the input data for the object acquisition model. Alternatively, the output matrices of the multiple grid points assigned to a feature can form input data as partial output data for an object acquisition model individually assigned to that feature. There can be as many object acquisition models as there are corresponding features.

[0039] The output matrices of the total output data can have a consistent first dimension size and second dimension size.

[0040] In an advantageous embodiment of the invention, the selected points include, in addition to the points of the grid cell, further points from at least one surrounding grid cell. The selected points can be limited to those lying within the grid cell. The points can include further points from neighboring grid cells. In this context, all points located in at least one neighboring grid cell can be included as further points.

[0041] The additional points can be captured via a selection area that covers at least the grid cell but also includes at least some adjacent grid cells. The selection area can be defined, particularly from the center of the grid cell in question, by a maximum distance, at least in one dimension of the point cloud grid, and in particular in all dimensions of the point cloud grid.

[0042] The maximum distance can be defined by a distance norm. This distance norm can be an L_p norm, a generalized distance norm with 1 ≤ p ≤ ∞, where p = 2 is the Euclidean norm. Alternatively, the distance norm can be a maximum norm, where the maximum distance in all dimensions defines the maximum distance. Finally, the distance norm can be a weighted norm, where different dimensions are scaled using weighting factors.

[0043] The features of the points can be supplemented, particularly before representation and especially after grid subdivision, by further features that depend on an arrangement and / or distribution relative to surrounding points and / or the relevant grid cell. Representation can also be performed for at least one such additional feature. These additional features can be, with respect to at least one dimension of the point cloud grid, a distance of the point in question to neighboring points, a distance of the point in question to a center of the relevant grid cell, and / or a distance of the point in question to a mean position of the selected points.

[0044] The first dimension size can correspond to a number of features, including further features, for each of which a representation is performed.

[0045] According to the present invention, an object detection system with the features of claim 10 is further proposed. The object parameters can be at least a distance, a dimension, a direction of movement, a speed, an object category, a material property, or other parameters of the at least one surrounding object.

[0046] In addition to the input data calculated through point cloud processing, the object acquisition model can process further input data, particularly from additional sensor data. This additional sensor data can include spectral data, such as spectral distance-velocity information, from the environmental sensor and / or sensor data from another environmental sensor.

[0047] According to the present invention, an environmental sensor system with the features according to claim 11 is further proposed.

[0048] Point cloud processing and / or object acquisition can be a computer-implemented method.

[0049] Furthermore, a computer program is proposed that contains machine-readable instructions executable on at least one computer, the execution of which performs point cloud processing and / or object acquisition.

[0050] Furthermore, a storage unit is proposed that is machine-readable and accessible by at least one computer and on which the aforementioned computer program is stored.

[0051] Further advantages and advantageous embodiments of the invention will become apparent from the description of the figures and the illustration. Character description

[0052] The invention is described in detail below with reference to the illustration. Fig.Figure 1 shows an object acquisition process with point cloud processing in a specific embodiment of the invention. The object acquisition process 10 for acquiring environmental objects, for example, in the environment of a vehicle, comprises a calculation 12 of object parameters 14 of the environmental objects by an object acquisition model 16 depending on input data 18. The object parameters 14 can be at least a distance, a dimension, a direction of movement, a velocity, an object category, a material property, or other parameters of the environmental objects. The object acquisition model 16 has a neural network 19 that is trained by deep learning. The object acquisition model 16 calculates an output from the input data 18 through several layers. At least some of the layers perform a linear transformation with weights and bias on the respective inputs of these layers, in particular followed by an activation function.The input data is propagated, for example, from the input layer through several intermediate layers to the output layer and then forms the output of the object acquisition model 16.

[0053] The input data 18 are formed by output data 20 of a point cloud processing system 22. The point cloud processing system 22 for processing point clouds 24 comprises providing a point cloud 24 that has several points 26, which in turn are calculated from sensor data 28 of an environmental sensor relating to an environment containing at least one environmental object, in particular the vehicle environment. The points 26 can be calculated from spectral data 30, for example, spectral distance-velocity information, of the environmental sensor. The spectral data 30, in turn, can be calculated from raw signals 32, for example, time series signals, of the environmental sensor.

[0054] Point cloud 24, for example, can be two-dimensional. The dimensions can be spatial coordinates, such as Cartesian coordinates, in particular an x-coordinate and a y-coordinate.

[0055] The points 26 exhibit several features 34 and a heterogeneous assignment 36 of these features 34 to one another. The features 34 can be at least one spatial coordinate, for example, an elevation angle, a distance, a velocity, in particular a Doppler velocity, a radar cross-sectional area, a signal intensity, and / or other information. This heterogeneous assignment 36 contrasts with a homogeneous assignment, in which all points 26 have the same number of features 34. It is particularly relevant when some points 26 have a feature 34 (represented here as circles around the points), in particular a Doppler velocity or an elevation angle, while other points 26 do not have such a feature 34.

[0056] Subsequently, a grid subdivision 38 of the point cloud 24 is performed, whereby the point cloud 24 is divided into a point cloud grid 40, which is spanned by grid cells 42 containing several points 26. For example, the two-dimensional point cloud 24 is divided here into a two-dimensional grid with, in particular, 25 grid cells 42.

[0057] The features 34 of the points 26 can subsequently be supplemented by further features 44 (here represented by a dashed circle), which depend, for example, on the arrangement and / or distribution of the points 26 relative to the grid cell 42 in which the points 26 are located. For example, the further feature 44 could be the distance of the respective point 26 to the center of the grid cell 42 in which the point lies.

[0058] Subsequently, an output data calculation 46 is performed, with which, for at least several of the grid cells 42, in particular for all grid cells 42 of the point cloud grid 40, and for the relevant grid cell 42, a statistically aggregated representation 49 is carried out over selected points 48 with respect to the relevant grid cell 42 for at least several of the features 34, in particular for all features 34 of the points 26, and for the relevant feature 34, with which the output data 20 are calculated. The representation 49 is, for example, for the relevant grid cell 42' and the relevant feature 34, a histogram 52 divided into intervals 50.

[0059] The representations 49 can be formed, for example, with a total of two features: a first feature 54 corresponding to feature 34 (illustrated here by the circles around the points) and a second feature 56 (which can correspond to the further feature 44 and is illustrated by the dashed circles around the points) of the points 26.

[0060] For example, for the relevant grid cell 42', where the representation is currently being performed, a first histogram 64 is created as the first representation 66 for the first feature 54, and a second histogram 68 is created as the second representation 70 for the second feature 56. The first histogram 64 can have a first number 72 of intervals 50, and the second histogram 68 a second number 74 of intervals 50, which may be equal to or different from the first number 72. For example, the first histogram 64 comprises four intervals 50, and the second histogram 68 comprises five intervals 50. The respective histogram 52 then indicates a frequency distribution 76 of the relevant feature 54, 56 over the intervals 50.

[0061] The selected points 48 of the relevant grid cell 42' include, in addition to the points 26 of grid cell 42, further points 58 from surrounding grid cells 42''. The selected points 48 can be limited to the points 26 lying within the relevant grid cell 42' or, as shown here by the circle spanning the grid cells 42, be encompassed by a selection area 60.

[0062] The output data 20 comprise an output matrix 84 assigned to the relevant grid cell 42'. The output data 20 are calculated from the representations 49 by transferring the values ​​of the intervals of the first and second histograms 64 and 68 into the output matrix 84. The output data 20 correspond specifically to the output matrix 84. The output matrix 84 has a first dimension 78 with a first dimension size 80. The first dimension size 80 corresponds to the number of features 34, here the first and second features 54 and 56, for which the representations 49 were performed, thus two in this case. Furthermore, the output matrix 84 has a second dimension size 82 that depends on the number of intervals 50 of the histogram 52. The second dimension size 82 can be formed by the maximum number of intervals 50 of the first and second histogram 64, 68, here by the second number 74, corresponding to five.The values ​​for the first characteristic 56, which are missing for the second dimension 82, can be filled with 0 (indicated here by the dashed area).

[0063] The output matrix 84 is thus spanned over the first and second dimension sizes 80, 82 and composed of the first and second representations 66, 67. The output data 20 comprise, as total output data 86, further output matrices 84' for other grid cells 42''' of the point cloud grid 40, in particular all output matrices 84 of the several grid cells 42 of the point cloud grid 40, in each of which the representation is carried out, and form the input data 18 for the object acquisition model 16 for the object acquisition 10 of the at least one environment object.

[0064] The object detection model 16 can comprise a neural network 92 and, in particular, be trained through deep learning. The object detection model 16 can be a multilayer perceptron that outputs an encoded feature vector or, as a holistic model, object parameters 14 of the at least one environment object.

Claims

[1] Point cloud processing (22) for processing point clouds (24), comprising the steps Providing a point cloud (24) which has several points (26) which in turn are calculated from sensor data (28) of an environment sensor for an environment containing at least one environment object and which have several features (34), Grid subdivision (38) of the point cloud (24) into a point cloud grid (40) spanned by several grid cells (42) containing the points (26), output data calculation (46) with which, for at least several of the grid cells (42), a statistically aggregated representation (49, 66, 70) is carried out over selected points (48) that include at least the points (26) of the grid cell (42) in a cell-related manner at the relevant grid cell (42') and for at least several of the features (34, 54, 56) in a feature-related manner at the relevant feature (34, 54, 56), from which output data (20) are calculated for the formation of input data (18) for an object acquisition model (16) for object acquisition (10) of the at least one environment object. [2] Point cloud processing (22) according to claim 1, characterized by, that the output data (20) have a first dimension (78) with a first dimension size (80) that depends on the number of features (34, 54, 56) in each of which a representation (49, 66, 70) is carried out. [3] Point cloud processing (22) according to claim 1 or 2, characterized by , that the representation (49, 66, 70) is a vector with statistical features and a vector length of . [4] Point cloud processing (22) according to claim 1 or 2, characterized by , that the representation (49, 66, 70) is a histogram (52) divided into intervals (50). [5] Point cloud processing (22) according to claim 4, characterized by , that the histogram (52) indicates a frequency distribution (76) of the relevant feature (34, 54, 56) of the selected points (48) over the intervals (50). [6] Point cloud processing (22) according to claim 4 or 5, characterized by, that the output data (20) have a second dimension size (82) that depends on the number of intervals (50) of the histogram (52). [7] Point cloud processing (22) according to claim 2 and 6, characterized by , that the output data (20) comprise an output matrix (84) spanning the first and second dimension sizes (80, 82). [8] Point cloud processing (22) according to any one of the preceding claims, characterized by , that the output data (20) as total output data (86) comprise all output matrices (84, 84') of the several grid cells (42, 42', 42'', 42''') of the point cloud grid (40), in each of which the representation (49, 66, 70) is carried out. [9] Point cloud processing (22) according to any one of the preceding claims, characterized by , that the selected points (48) include, in addition to the points (26) of the grid cell (42), further points (58) from at least one surrounding grid cell (42''). [10] Object detection (10) for detecting at least one environment object in an environment, exhibiting a point cloud processing (22) according to one of the preceding claims for calculating input data (18), a calculation (12) of object parameters (14) of the at least one environment object by an object acquisition model (16) at least dependent on the input data (18). [11] environmental sensor system at least one environmental sensor for providing sensor data (28) about an environment containing at least one environmental object, an object detection unit for object detection (10) of the at least one surrounding object according to claim 10.

Images