Method for sensor data processing and processing device

Abstract

The invention relates to a method for processing sensor data (10) from sensor data (18) of an environmental sensor (12), comprising providing the sensor data (18) of the environmental sensor (12) which detects at least one target object in a sensor environment, calculating a point cloud (32) comprising a spatial distribution of several points (30) from the sensor data (18), each assigned to a reflection of the target object, inputting the points (30) and the information segments (34) of the sensor data (18) assigned to them to a trained processing model (36), calculating feature vectors (40) assigned to the points (30) by the processing model (36) depending on the input, and outputting the points (30) and the assigned feature vectors (40) for further processing by another processing model (42). The invention further relates to a processing device.

Description

[0001] The invention relates to a method for sensor data processing according to claim 1. Furthermore, the invention relates to a processing device. State of the art

[0002] In vehicles, driver assistance systems and autonomous driving functions require precise detection of the vehicle's surroundings. For this purpose, environmental sensors, such as cameras and radar sensors, are used. These environmental sensors provide sensor measurement data, which is then processed and calculated into sensor data in the form of spectra. A point cloud comprising a spatial distribution of several points, each assigned to reflections from target objects in the vehicle's surroundings, is calculated from the sensor data. From these point clouds, the position, pose, class, and, if necessary, other object properties of the target objects are determined using environmental detection algorithms.

[0003] With the advent of deep learning, traditional algorithms are increasingly being replaced by object detection networks and similar methods. Here, a deep neural network determines the probability of existence, size, pose, and class of the target objects, i.e., oriented bounding boxes (OBBs). Only then are the detected OBBs tracked temporally.

[0004] Object detection networks are still relatively unpopular with radar sensors due to the smaller number of points. When they are used, point cloud points are usually used as input for object detection networks. The points are calculated from the sensor data and represent a compressed form of the original sensor data, with many details omitted. Disclosure of the invention

[0005] According to the present invention, a method for sensor data processing with the features of claim 1 is proposed. As a result, the data output for further processing by the further processing model can have a smaller data size while still containing as much information as possible from the sensor data. The storage space required for the data stored for further processing can be reduced. The data rate for output for further processing by the further processing model can be reduced. The input data provided for further processing can have a higher information content. Furthermore, sensor-independent input data can be generated for further processing.

[0006] A data interface that transmits the output of the processing model as input to the further processing model can have a lower bandwidth and transmit a smaller amount of data than if the sensor data as a whole were used as input for further processing. Nevertheless, the input data of the further processing model can contain more information from the sensor data than if only the points form the input data.

[0007] The environment sensor can be a radar sensor, a lidar sensor, a camera, in particular a stereo camera or a mono camera, a time-of-flight camera, a microphone, or an ultrasonic sensor. The environment sensor can be arranged on a vehicle, in particular a motor vehicle, a two-wheeled vehicle, a bicycle, or on a mobile or stationary device, in particular a robot, a building, a traffic infrastructure, or an industrial component.

[0008] The target object can be a living being, for example a person, an object, in particular a traffic sign, a facility, in particular a building and / or a plant, for example a tree, in the sensor environment.

[0009] The sensor data may be the result of processing sensor measurement data from the environmental sensor. The sensor data may be in a form that allows certain attributes or features of the sensor measurement data to be understood, quantified, and / or interpreted. The sensor data may specify a frequency, an amplitude, a phase, a signal intensity, a target object speed, a target object distance, an angle of the target object, such as an azimuth angle or an elevation angle, a target object position, and / or an target object orientation. The sensor data may form the input data for obtaining the points of the point cloud.

[0010] In addition to the points and information segments, the input to the processing model may include further measurement parameters of the environmental sensor, in particular modulation parameters, antenna parameters, calibration data, transmission parameters, reception parameters, filter parameters and / or processing parameters.

[0011] The points of the point cloud can be derived from the sensor data. A point in the point cloud represents at least one measured spatial coordinate of the reflection. A point can contain additional parameters in addition to the spatial coordinate. The additional parameters can be an angle, in particular an azimuth angle or an elevation angle, a radar cross section (RCS), a speed, and / or a reflection intensity. The point cloud provides a three-dimensional representation of the detected target object and can be used to determine the shape, size, and orientation of the target object. Each point can be assigned a Cartesian coordinate or polar coordinate, in particular a distance and an azimuth angle.

[0012] An information section can contain information from a sub-area of ​​the sensor data and / or a sub-area of ​​the sensor data.

[0013] The processing model can be a model trained using training data, in particular a deep learning model. The processing model can include a neural network.

[0014] The processing model can be a convolutional neural network (CNN). This uses convolutional layers, particularly two-dimensional convolutional layers, with convolutions to filter and extract features from the input data, particularly to detect spatial structures and patterns, such as textures or edges, in the input data. The convolutional layers can be sparse convolutions, performing convolution only at those locations in the input data where information is actually present. For sparse input data, submanifold convolutions can be used, which restrict processing to the information-containing data and omit the information-free data while maintaining the original dimension of the input data.

[0015] The processing model may comprise at least one pooling layer. The pooling layer may form the last layer of the multi-layer processing model, in particular to obtain a single feature vector for each point.

[0016] The processing model can be a recurrent neural network (RNN). This uses loops to store information over time and recognize patterns across time sequences. The loops can transfer information from one time step to the next, particularly to model sequential and temporal dependencies.

[0017] The processing model can be a Long Short Term Memory (LSTM). This uses gating mechanisms to address the vanishing and exploding gradient problem in RNNs, specifically input, forget, and output gates to selectively add, store, and retrieve information.

[0018] The processing model can be an autoencoder. This uses an encoder that converts the input data into a latent space and a decoder that converts the converted data back to the original space.

[0019] The processing model can be a support vector machine (SVM), which uses a multi-dimensional dividing line or hyperplane between classes of input data.

[0020] The processing model can be a transformer model. This model uses the self-attention mechanism to consider relationships between values ​​of the input data and, in particular, uses an encoder that transforms the input data into a latent space.

[0021] The feature vector can be sensor-independent. Simultaneous and spatially consistent detection of the target object with different environmental sensors of the same or different sensor categories can result in consistent feature vectors after applying the individual processing models assigned to each environmental sensor. Sensor-specific properties or measurement parameters can be incorporated into the features of the feature vector. This allows further processing to be performed independently of the environmental sensor.

[0022] The output data of the processing model can be processed or directly form the input data of the subsequent processing model. The data can be transmitted via a data interface. The data interface can be a CAN bus.

[0023] The method for sensor data processing may be a computer-implemented method.

[0024] In a preferred embodiment of the invention, the sensor data is divided into data bins as discretization elements, and at least one data bin is assigned to each point as a point bin. The discretization element is a sub-area of ​​the sensor data. The data bins can be tiles of the sensor data divided into a grid structure.

[0025] In a specific embodiment of the invention, it is advantageous if the information segments assigned to the points comprise several data bins adjacent to the respective point bins. The information segments can be formed by the data bins. The information segments can include the point bin. The partial area of ​​the sensor data covered by the respective information segment is preferably larger than the partial area of ​​the sensor data delimited by a data bin.

[0026] A preferred embodiment of the invention is advantageous in which the information segments assigned to the points comprise several data bins surrounding the respective point bins. The data bins can partially or completely surround the assigned point bin. For example, an information segment can have 7x7 data bins. The point bin can be arranged in the center of the data bins. The information segments can be rectangular, square, or any other shape.

[0027] In a preferred embodiment of the invention, it is advantageous if the calculation of the feature vectors by the processing model and the further processing by the additional processing model take place on different calculation units. The calculation by the processing model can take place on the environment sensor itself or on a calculation unit individually, in particular exclusively, assigned to the environment sensor. The calculation unit can process exclusively the points and feature vectors of a single environment sensor.

[0028] The processing by the additional processing model can run on a central control unit. The processing can be performed by a calculation unit that processes feature vectors and points from multiple environmental sensors. For this purpose, the sensor data from each environmental sensor can be processed by a separate processing model to obtain the points and associated feature vectors for each environmental sensor. The data output by the processing models can be aggregated and passed on to a single additional processing model or to several additional processing models for further processing.

[0029] If one of several environmental sensors fails, is replaced, or is modified, the processing model assigned to it can also be modified or eliminated. Regardless of this, the other environmental sensors and their associated processing models can still provide the outputs to the additional processing model. The failure of a single environmental sensor can thus be compensated for. Retraining the additional processing model and the other processing models of the environmental sensors is not necessary.

[0030] If an environmental sensor is replaced with its associated processing model, the new processing model can be re-trained in the context of the existing additional processing model and the existing other processing models of the other environmental sensors. Preferably, the parameters of the additional processing model and / or the other processing models of the other environmental sensors are left unchanged when training the new processing model.

[0031] This allows the new processing model to be inserted without any adjustments or without complex adjustments to the further processing model and the other processing models.

[0032] In a specific embodiment of the invention, it is advantageous if the additional processing model processes the points and feature vectors and calculates at least one object property of the target object from them. The additional processing model can implement environment detection, in particular object detection, object recognition, object classification, semantic segmentation, and / or localization. The additional processing model can calculate a planning or functional specification.

[0033] The further processing model can be a model learned from training data, in particular a deep learning model. The further processing model can include a neural network.

[0034] The further processing model can be a convolutional neural network (CNN). This uses convolutional layers with convolutions to filter and extract features from the input data, particularly to detect spatial structures and patterns, such as textures or edges, in the input data. The convolutional layers can be sparse convolutions, performing the convolution only at those locations in the input data where information is actually present. For sparse input data, submanifold convolutions can be used, which restrict processing to the data containing information and omit the information-free data while maintaining the original dimension of the input data.

[0035] The further processing model can be a recurrent neural network (RNN). This uses loops to store information over time and recognize patterns across time sequences. The loops can transfer information from one time step to the next, especially to model sequential and temporal dependencies.

[0036] The further processing model can be a Long Short Term Memory (LSTM). This uses gating mechanisms to address the problem of vanishing and exploding gradients in RNNs, specifically input, forget, and output gates to selectively add, store, and retrieve information.

[0037] The further processing model can be an autoencoder. This uses an encoder that converts the input data into a latent space and a decoder that converts the converted data back to the original space.

[0038] The further processing model can be a support vector machine (SVM). This uses a dividing line or hyperplane in multiple dimensions between classes of the input data.

[0039] The further processing model can be a transformer model. This uses the self-attention mechanism to consider relationships between values ​​of the input data and, in particular, uses an encoder that transforms the input data into a latent space.

[0040] The additional processing model can be trained by having other processing models from additional environmental sensors provide the input data for the processing model. The additional processing model can be trained by masking out individual other processing models when compiling the input data, in particular to better compensate for a later failure of individual environmental sensors and / or to better balance the weighting of the environmental sensors, for example, to mitigate an excessive influence of individual environmental sensors.

[0041] The processing model can be trained jointly with the further processing model. The training can use the gradient descent method. Backpropagation in the further processing model can calculate the gradient of the cost function for each weight and bias of the neural network of the further processing model. The calculated gradient at the input of the neural network can be used for backpropagation when training the processing model during joint training.

[0042] The processing model can be specifically trained for one task. An additional processing model for the same environment sensor can be trained to perform a different task. The processing model can be task-specific. The processing model and the additional processing model can each output the points and feature vectors to the additional processing model simultaneously or selectively, depending on the desired task.

[0043] The object property of the target object can be an object distance, an angle, for example an azimuth angle or an elevation angle, a position, a pose and / or an orientation of the target object.

[0044] In a preferred embodiment of the invention, the sensor data is provided as a spectrum. The spectrum can be a distance-velocity spectrum, a frequency spectrum, a phase spectrum, a polarization spectrum, a power spectrum, and / or a time spectrum.

[0045] The sensor data can alternatively be a time signal from a time-of-flight sensor, in particular a lidar sensor.

[0046] In a preferred embodiment of the invention, it is advantageous if the spectrum is at least two-dimensional. The spectrum can also be one-dimensional or higher-dimensional, in particular three-dimensional.

[0047] In a preferred embodiment of the invention, it is advantageous if the feature vectors each contain information derived from the associated information segments by processing in the processing model. The features contained in the dimensions of the feature vector can be compressed and / or aggregated information from the associated information segments. The features contained in the feature vector can be encoded information from the information segments. The feature vector can contain information about a variance, a signal strength, and / or a peak extension of the sensor data, in particular in encoded form.

[0048] The feature vector can contain multiple features, for example, 32 features. Each feature can have a predetermined bit size, for example, 16 bits. The feature vector can be one-dimensional.

[0049] According to the present invention, a processing device is further proposed, which is configured to carry out the method for sensor data processing with at least one of the previously described features, comprising an environment sensor for detecting at least one target object in a sensor environment, a calculation unit for executing the processing model, and a calculation unit for executing the further processing model. The calculation unit for executing the processing model can be formed by the environment sensor itself or by a calculation unit individually, in particular exclusively, assigned to the environment sensor. The calculation unit can process exclusively the points and feature vectors of a single environment sensor.

[0050] The calculation unit for executing the further processing model can be formed by a central control unit. This calculation unit can process the feature vectors and points from multiple environmental sensors.

[0051] Further advantages and advantageous embodiments of the invention emerge from the description of the figures and the illustration. Character description

[0052] The invention is described in detail below with reference to the figure.

[0053] Fig.1 shows a method for sensor data processing in a specific embodiment of the invention. The method for sensor data processing 10 processes the information emitted by at least one environmental sensor 12, for example, a radar sensor 14. The environmental sensor 12 detects at least one target object in a sensor environment and thus provides sensor measurement data 16. From the sensor measurement data 16, sensor data 18 is calculated by raw signal processing, which is present as a two-dimensional spectrum 22 having two dimensions A, B. The spectrum 22 can be a distance-velocity spectrum. The sensor data 18 is preferably divided into data bins 24 as discretization elements 26. The data bins 24 can be tiles of the spectrum 22 divided into a grid structure 28.

[0054] From the sensor data 18, points 30 assigned to reflections of the target object are derived, which form a point cloud 32 in a spatial distribution corresponding to the reflections. The points 30, together with the information sections 34 assigned to them from the sensor data 18, are input to a trained processing model 36. The information sections 34 are, for example, each a range of data bins 24 comprising several tiles of the two-dimensional spectrum 22. The information section 34 assigned to a respective point 30 is, for example, a sub-range of the spectrum 22 comprising 3x3 data bins 24, in the center of which lies the point bin 38 assigned to the respective point 30.

[0055] The processing model 36 can be a model trained using training data, in particular a deep learning model, and preferably comprise a neural network. The processing model 36 calculates feature vectors 40 associated with the points 30 for further processing by another processing model 42. The feature vectors 40 each contain information derived from the associated information segments 34 by processing in the processing model 36. This information can be compressed and aggregated information from the associated information segments 34, in particular as encoded information.

[0056] The calculation of the feature vectors 40 by the processing model 36 and the further processing by the further processing model 42 take place on different calculation units 44. The calculation by the processing model 36 takes place in particular on the environment sensor 12 itself as the calculation unit 44. The processing by the further processing model 42 preferably takes place on a central control unit 46 as the calculation unit, which receives and processes the data output by the processing model 36.

[0057] The further processing model 42 has, in particular, further input data in the form of further points 48 and associated further feature vectors 50 of a further environmental sensor 52, for example, a lidar sensor 54. The further feature vectors 50 assigned to the further points 48 are calculated, in particular, by another processing model 56 assigned to the further environmental sensor 52, depending on the further points 48 and associated further information segments 58 of further sensor data 60 of the further environmental sensor 52. The further sensor data 60 are calculated from further sensor measurement data 62 of the further environmental sensor 52 and are specified as a further spectrum with the dimensions C, D.

[0058] The points 30 and associated feature vectors 40, as well as the further points 48 and further feature vectors 50, can be a form of data representation independent of the concrete form of the sensor data 18 and further sensor data 60, whereby the specific measurements of the respective environmental sensors 12, 52 are encoded in a uniform form. The further processing model 42 processes the points 30 and feature vectors 40 of the environmental sensor 12, the further points 48 and further feature vectors 50 of the further environmental sensor 52, and further outputs of possible other environmental sensors, and calculates at least one object property 64 of the target object therefrom. The further processing model 42 can implement environmental sensing, in particular object detection. The further processing model 42 comprises, in particular, an object detection network and is preferably trained, in particular by deep learning.The object property 64 of the target object may be an object distance, an angle, for example an azimuth angle or an elevation angle, a position, a pose and / or an orientation of the detected target object.

Claims

[1] Method for sensor data processing (10) of sensor data (18) of an environmental sensor (12), comprising Providing the sensor data (18) of the environmental sensor (12) detecting at least one target object in a sensor environment, Calculating a point cloud (32) comprising a spatial distribution of a plurality of points (30) each associated with reflections of the target object from the sensor data (18), Input of the points (30) and the information sections (34) of the sensor data (18) assigned to them to a trained processing model (36), calculation of feature vectors (40) assigned to the points (30) by the processing model (36) depending on the input, Output of the points (30) and the associated feature vectors (40) for further processing by a further processing model (42). [2] Method for sensor data processing (10) according to claim 1, characterized bythat the sensor data (18) are divided into data bins (24) as discretization elements (26) and at least one data bin (24) is assigned to each of the points (30) as a point bin (38). [3] Method for sensor data processing (10) according to claim 2, characterized by that the information sections (34) assigned to the points (30) comprise a plurality of data bins (24) adjacent to the respective point bins (38). [4] Method for sensor data processing (10) according to claim 2 or 3, characterized by that the information sections (34) assigned to the points (30) comprise several data bins (24) surrounding the respective point bins (38). [5] Method for sensor data processing (10) according to one of the preceding claims, characterized by that the calculation of the feature vectors (40) by the processing model (36) and the further processing by the further processing model (42) take place on mutually different calculation units (44, 46). [6] Method for sensor data processing (10) according to one of the preceding claims, characterized by that the further processing model (42) processes the points (30) and feature vectors (40) and calculates therefrom at least one object property (64) of the target object. [7] Method for sensor data processing (10) according to one of the preceding claims, characterized by that the sensor data (18) are available as a spectrum (22). [8] Method for sensor data processing (10) according to claim 7, characterized by that the spectrum (22) is at least two-dimensional. [9] Method for sensor data processing (10) according to one of the preceding claims, characterized by that the feature vectors (40) each contain information derived from the associated information extracts (34) by processing in the processing model (36). [10] Processing device configured to carry out the method for sensor data processing (10) according to one of the preceding claims, comprising an environment sensor (12) for detecting at least one target object in a sensor environment, a calculation unit (44) for executing the processing model (36) and a calculation unit (46) for executing the further processing model (42).

Images