Method and processing device for sensor data processing

Abstract

The invention relates to a method for sensor data processing (10) of sensor data (18) of a surroundings sensor (12), comprising: providing sensor data (18) of at least one target object in a sensor surroundings of the surroundings sensor (12) which performs detection, calculating a point cloud (32), the point cloud comprises a spatial distribution of a plurality of points (30) from the sensor data (18) which are each assigned to a reflection of the target object, inputting the points (30) and information segments (34) of the sensor data (18) which are each assigned to the points into a trained processing model (36), calculating feature vectors (40) assigned to the points (30) from the input by means of the processing model (36), the output points (30) and the associated feature vectors (40) are used for further processing by a further processing model (42). The invention further relates to a treatment device.

Description

Technical Field

[0001] The invention relates to a method for sensor data processing of sensor data of an environmental sensor. The invention also relates to a processing device. Background Art

[0002] In vehicles, driver assistance systems and autonomous driving functions require precise detection of the vehicle environment. For this purpose, environmental sensors are used, such as radar sensors in addition to cameras. These environmental sensors provide sensor measurement data, from which sensor data in the form of a spectrum is calculated by preprocessing or processing. A point cloud is calculated from the sensor data, which includes a spatial distribution of a plurality of points, each corresponding to a reflection of a target object in the vehicle environment. The position, pose, class and, if necessary, other object properties of the target object are determined from these point clouds using environmental perception algorithms.

[0003] With the introduction of deep learning, traditional algorithms are gradually being replaced by object detection networks and similar methods. Here, deep neural networks determine the existence probability, size, pose, and class of the target object, i.e., oriented bounding boxes (OBBs). The detected OBBs are then tracked in time.

[0004] In radar sensors, object detection networks have not yet been extended due to the small number of points. If so, points of a point cloud are usually used as input for the object detection network. These points are calculated from the sensor data and reproduce the original information of the sensor data in a compressed form, wherein a large amount of information is not taken over. Summary of the invention

[0005] According to the invention, a method for sensor data processing of sensor data of an environmental sensor is proposed. As a result, the data output for further processing by a further processing model can have a smaller data size and nevertheless contain information from the sensor data as comprehensively as possible. The memory space required for the data stored for further processing can be reduced. The data rate when outputting for further processing by a further processing model can be reduced. The input data provided for further processing can have a higher information content. In addition, sensor-independent input data can be generated for further processing.

[0006] Compared to using all sensor data as input data for further processing, the data interface that uses the output of the processing model as input data for transmission to the further processing model has a lower bandwidth and transmits a smaller amount of data. Therefore, the input data of the further processing model can have more information from the sensor data than only point-forming input data.

[0007] The environmental sensor can be a radar sensor, a lidar sensor, a camera, in particular a stereo camera or a mono camera, a time-of-flight camera, a microphone or an ultrasonic sensor. The environmental sensor can be arranged on a vehicle, in particular a motor vehicle, a two-wheeled vehicle, a bicycle, or a mobile or stationary device, in particular a robot, a building, a traffic infrastructure or an industrial component.

[0008] The target object may be a living being, such as a person, an object, in particular a traffic sign, a device, in particular a building, and / or a plant, such as a tree, in the sensor environment.

[0009] The sensor data may be the result of processing sensor measurement data of an environmental sensor. The sensor data may be in a form that allows understanding, quantification and / or interpretation of specific properties or features of the sensor measurement data. The sensor data may specify frequency, amplitude, phase, signal strength, object velocity of a target object, object distance of a target object, angle of a target object, such as azimuth or elevation, position of a target object and / or orientation of a target object. The sensor data may form initial data for obtaining points of a point cloud.

[0010] In addition to points and information sections, the input to the processing model may also include further measurement parameters of the environmental sensors, in particular modulation parameters, antenna parameters, calibration data, transmission parameters, reception parameters, filter parameters and / or processing parameters.

[0011] The points of the point cloud can be derived from sensor data. The points of the point cloud represent at least the spatial coordinates of the reflection measurement. In addition to the spatial coordinates, the points can also contain further parameters. Further parameters can be angles - in particular azimuth or elevation, radar cross section (RCS), speed and / or reflection intensity. The point cloud provides a three-dimensional representation of the detected target object and can be used to detect the shape, size and orientation of the target object. The points can be assigned rectangular or polar coordinates, in particular distance and azimuth.

[0012] The information section may contain information about a sub-region of the sensor data and / or a sub-region of the sensor data.

[0013] The processing model may be a model trained by training data, in particular a deep learning model. The processing model may include a neural network.

[0014] The processing model may be a convolutional neural network (CNN). It uses convolutional layers, especially two-dimensional convolutional layers, to filter and extract input data features by means of convolution, especially to identify spatial structures and patterns in the input data, such as textures or edges. The convolutional layers may be sparse convolutions, and convolutions are performed only at locations of the input data where information actually exists. In the case of sparse input data, submanifold convolutions may be available, which restrict processing to data containing information and remove uninformative data while retaining the original dimensionality of the input data.

[0015] The processing model may have at least one pooling layer. The pooling layer may form the last layer of a multi-layer processing model, in particular in order to obtain a single feature vector for each point.

[0016] The processing model can be a recurrent neural network (RNN). It uses loops to store information over time and recognize patterns across time series. The loop can transfer information from one time step to the next time step, especially to model sequential and temporal correlations.

[0017] The processing model may be a Long Short Term Memory (LSTM), which uses a gating mechanism to solve the gradient vanishing and exploding problems in RNN, especially input, forgetting, and output gates to selectively add, store, and call information.

[0018] The processing model may be an autoencoder, which uses an encoder that transforms the input data into a latent space and a decoder that transforms the transformed data back into the original space.

[0019] The processing model may be a Support Vector Machine (SVM), which uses dividing lines or hyperplanes in multiple dimensions between classes of the input data.

[0020] The processing model may be a transformer model that uses a self-attention mechanism to take into account the relationship between the values ​​of the input data, and in particular uses an encoder that transforms the input data into a latent space.

[0021] The feature vector can be sensor-independent. The simultaneous and spatially consistent detection of a target object by means of different environmental sensors of the same or different sensor types can result in a consistent feature vector after the application of a single processing model assigned to each environmental sensor. Sensor-specific properties or measurement parameters can also be taken into account in the features of the feature vector. This allows further processing to be performed independently of the environmental sensors.

[0022] The output data of the process model can form the input data of another process model after processing or directly. The data can be transmittable via a data interface. The data interface can be a CAN bus.

[0023] The method for sensor data processing may be a computer-implemented method.

[0024] In a preferred embodiment of the present invention, the sensor data is divided into data partitions as discretization elements, and each point is assigned at least one data partition as a point partition. The discretization elements are partial areas of the sensor data. The data partitions can be patches of the sensor data divided into a grid structure.

[0025] In a special embodiment of the invention, it is advantageous if the information segment assigned to a point comprises a plurality of data partitions adjacent to the corresponding point partition. The information segment can be formed by the data partitions. The information segment can contain the point partitions. The partial area of ​​the sensor data covered by the corresponding information segment is preferably larger than the partial area of ​​the sensor data delimited by the data partitions.

[0026] A preferred configuration of the present invention is advantageous, wherein the information segment assigned to a point comprises a plurality of data partitions around the corresponding point partition. The data partition may partially or completely surround the assigned point partition. For example, the information segment may have 7×7 data partitions. The point partition may be arranged in the center of the data partition. The information segment may be rectangular, square or have any other shape.

[0027] In a preferred embodiment of the invention, it is advantageous if the calculation of the characteristic vectors by means of a processing model and the further processing by means of a further processing model are performed on different computing units. The calculation by means of the processing model can be performed on the environment sensor itself or on a separate computing unit, in particular only assigned to the environment sensor. A computing unit can only process points and characteristic vectors of a single environment sensor.

[0028] The processing by the additional processing model can be run on the central control device. The processing can be carried out by a calculation unit, which processes the feature vectors and points of a plurality of environmental sensors. For this purpose, the sensor data of the environmental sensors can be processed by their own processing model in each case, so that a point and an associated feature vector are obtained for each environmental sensor. The data output by the processing model can be combined and transmitted to a single additional processing model or to a plurality of additional processing models for further processing.

[0029] If, in the case of a plurality of environmental sensors, one environmental sensor fails, is replaced or changed, the processing model assigned to this environmental sensor may also be changed or removed. Independently of this, the other environmental sensors and the processing models assigned to them can nonetheless still provide outputs to the further processing models. Thus, the failure of a single environmental sensor can be compensated. There is no need to retrain the further processing models and other processing models of the environmental sensors.

[0030] If an environmental sensor with an associated processing model is replaced, a new processing model can be retrained against the background of existing further processing models and existing further processing models of other sensors. Preferably, the parameters of the further processing model and / or the further processing model of the further environmental sensor remain unchanged when the new processing model is trained. This allows the addition of new processing models without adapting the further processing model or without adapting the further processing model in a complex manner.

[0031] In a special embodiment of the invention, it is advantageous if the further processing model processes points and feature vectors and calculates therefrom at least one object property of the target object. The further processing model can implement environment detection, in particular object detection, object recognition, object classification, semantic segmentation and / or localization. The further processing model can calculate a plan or a functional specification.

[0032] Another processing model may be a model trained by training data, in particular a deep learning model. Another processing model may include a neural network.

[0033] Another processing model may be a convolutional neural network (CNN). It uses convolutional layers to filter and extract input data features by means of convolution, in particular to identify spatial structures and patterns in the input data, such as textures or edges. The convolutional layers may be sparse convolutions, and the convolutions are performed only at locations of the input data where information actually exists. In the case of sparse input data, submanifold convolutions may be used, which restrict processing to data containing information and remove uninformative data while retaining the original dimensionality of the input data.

[0034] Another processing model can be a recurrent neural network (RNN). It uses loops to store information over time and recognize patterns across time series. The loop can transfer information from one time step to the next time step, especially to model sequential and temporal correlations.

[0035] Another processing model can be Long Short-Term Memory (LSTM), which uses a gating mechanism to solve the problem of vanishing and exploding gradients in RNNs, especially input, forget, and output gates to selectively add, store, and call information.

[0036] Another processing model can be an autoencoder, which uses an encoder that transforms the input data into a latent space, and a decoder that transforms the transformed data back into the original space.

[0037] Another processing model may be a support vector machine (SVM), which uses dividing lines or hyperplanes in multiple dimensions between classes of the input data.

[0038] Another processing model may be a transformer model, which uses a self-attention mechanism to take into account the relationship between the values ​​of the input data, and in particular uses an encoder that transforms the input data into a latent space.

[0039] Further processing models can be trained in that further processing models of further environmental sensors provide input data for the processing model. Further processing models can be trained in that the individual further processing models are hidden when the input data are combined, in particular in order to be able to better intercept later failures of individual environmental sensors and / or to better balance the weights of the environmental sensors, for example in order to mitigate the strong influence of individual environmental sensors.

[0040] The processing model can be trained together with another processing model. The training can apply a gradient descent method. Back propagation in the other processing model can calculate the gradient of the cost function of each weight and bias of the neural network of the other processing model. In the case of joint training, the calculated gradient at the input of the neural network can be used for back propagation when training the processing model.

[0041] The processing model can be purposefully trained for one task. Additional processing models for the same environmental sensor can be trained to meet other tasks. The processing model can be task-specifically trained. The processing model and the additional processing model can output points and feature vectors to another processing model simultaneously or selectively, in particular depending on the desired task.

[0042] The object property of the target object may be an object distance, an angle, such as an azimuth or elevation angle, a position, a posture and / or an orientation of the target object.

[0043] In a preferred embodiment of the present invention, the sensor data is present as a spectrum, which may be a range-velocity spectrum, a frequency spectrum, a phase spectrum, a polarization spectrum, a power spectrum and / or a time spectrum.

[0044] Alternatively, the sensor data may be a time signal of a time-of-flight sensor, in particular a lidar sensor.

[0045] In a preferred embodiment of the invention it is advantageous if the spectrum is at least two-dimensional. The spectrum can also be one-dimensional or higher-dimensional, in particular three-dimensional.

[0046] In a preferred embodiment of the invention, it is advantageous that the feature vectors each contain information derived from the associated information segment by processing in the processing model. The features contained in the feature vector dimension can be compressed and / or summarized information of the associated information segment. The features contained in the feature vector can be coded information of the information segment. The feature vector can contain information about the variance, signal strength and / or peak spread of the sensor data, in particular in coded form.

[0047] The feature vector may contain a plurality of features, for example 32 features. Each feature may have a predetermined bit size, for example 16 bits. The feature vector may be one-dimensional.

[0048] According to the invention, a processing device is also proposed, which is configured to implement a method for sensor data processing having at least one of the above-mentioned features, the processing device having an environment sensor for detecting at least one target object in the sensor environment, a computing unit for implementing a processing model, and a computing unit for implementing a further processing model. The computing unit for implementing the processing model can be formed by the environment sensor itself or by a separate computing unit, in particular only assigned to the environment sensor. The computing unit can only process points and feature vectors of a single environment sensor.

[0049] The central control device can form a computing unit for implementing the further processing model. The computing unit can process feature vectors and points of a plurality of environmental sensors.

[0050] Further advantages and advantageous embodiments of the invention are apparent from the graphic description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The present invention will be described in detail below with reference to the accompanying drawings.

[0052] Figure 1A method for sensor data processing is shown in a special embodiment of the present invention. DETAILED DESCRIPTION

[0053] Figure 1 A method for sensor data processing in a special embodiment of the present invention is shown. The method 10 for sensor data processing processes information output from at least one environmental sensor 12, such as a radar sensor 14. The environmental sensor 12 detects at least one target object in the sensor environment and thus provides sensor measurement data 16. Sensor data 18 are calculated from the sensor measurement data 16 by raw signal processing, which sensor data is present as a two-dimensional spectrum 22 with two dimensions A, B. The spectrum 22 can be a distance-velocity spectrum. The sensor data 18 is preferably divided into data partitions 24 as discretization elements 26. The data partitions 24 can be patches of the spectrum 22 divided into a grid structure 28.

[0054] Points 30, each assigned to a reflection of the target object, are derived from the sensor data 18 and form a point cloud 32 in a spatial distribution corresponding to the reflection. The points 30 are input into a trained processing model 36 together with an assigned information segment 34 of the sensor data 18. For example, the information segment 34 is a region of a data partition 24 that includes a plurality of slices of a two-dimensional spectrum 22. The information segment 34 assigned to the corresponding point 30 is, for example, a partial region of the spectrum 22 that includes a 3x3 data partition 24, at the center of which the point partition 38 assigned to the point 30 is located.

[0055] The processing model 36 can be a model trained by training data, in particular a deep learning model, and preferably has a neural network. The processing model 36 calculates a feature vector 40 respectively assigned to the point 30 for further processing by a further processing model 42. The feature vector 40 respectively contains information derived from the associated information segment 34 by processing in the processing model 36. This information can be compressed and summarized information of the associated information segment 34, in particular as coded information.

[0056] The calculation of the characteristic vector 40 by the process model 36 and the further processing by the further process model 42 are performed on different computing units 44. In particular, the calculation by the process model 36 is performed on the environment sensor 12 itself as computing unit 44. The processing by the further process model 42 is preferably performed on a central control device 46 as computing unit, which receives and processes the data output by the process model 36.

[0057] In particular, the further processing model 42 has further input data in the form of further points 48 and associated further feature vectors 50 of a further environmental sensor 52, for example a lidar sensor 54. The further feature vectors 50 associated with the further points 48 are calculated, in particular, by a further processing model 56 associated with the further environmental sensor 52 from the further points 48 and associated further information segments 58 of further sensor data 60 of the further environmental sensor 52. The further sensor data 60 are calculated from further sensor measurement data 62 of the further environmental sensor 52 and are presented as further spectra with dimensions C, D.

[0058] The points 30 and the associated feature vectors 40 as well as the further points 48 and the further feature vectors 50 can be data representations that are independent of the specific form of the sensor data 18 and the further sensor data 60, whereby the specific measurements of the corresponding environmental sensors 12, 52 are encoded in a unified form. The further processing model 42 processes the points 30 and the feature vectors 40 of the environmental sensor 12, the further points 48 and the further feature vectors 50 of the further environmental sensor 52 and the further outputs of possible other environmental sensors, and calculates at least one object characteristic 64 of the target object from this. The further processing model 42 can realize environmental detection, in particular object detection. The further processing model 42 in particular comprises an object detection network and is preferably trained, in particular by deep learning. The object characteristic 64 of the target object can be an object distance, an angle, for example an azimuth or elevation angle, a position, a posture and / or an orientation of the detected target object.

Claims

1. A method for sensor data processing (10) of sensor data (18) of an environmental sensor (12), the method comprising: providing sensor data (18) of at least one target object in a sensor environment of a detecting environmental sensor (12), calculating a point cloud (32) comprising a spatial distribution of a plurality of points (30) from the sensor data (18) which are each assigned to a reflection of the target object, inputting points (30) and information sections (34) of sensor data (18) respectively associated with the points into a trained processing model (36), calculating a feature vector (40) associated with the point (30) based on the input by means of a processing model (36), The output points (30) and the associated feature vectors (40) are used for further processing by a further processing model (42).

2. The method (10) for sensor data processing according to claim 1, characterized in that The sensor data (18) are divided into data partitions (24) as discretization elements (26), and at least one data partition (24) is respectively assigned to the points (30) as point partitions (38).

3. The method (10) for sensor data processing according to claim 2, characterized in that The information section (34) assigned to the point (30) includes a plurality of data partitions (24) adjacent to corresponding point partitions (38).

4. The method (10) for sensor data processing according to claim 2 or 3, characterized in that The information section (34) assigned to the point (30) includes a plurality of data partitions (24) surrounding a corresponding point partition (38).

5. The method (10) for sensor data processing according to any one of the preceding claims, characterized in that The calculation of the characteristic vector (40) by means of the processing model (36) and the further processing by means of the further processing model (42) are carried out on different computing units (44, 46).

6. The method (10) for sensor data processing according to any one of the preceding claims, characterized in that The further processing model (42) processes the points (30) and the feature vectors (40) and calculates therefrom at least one object property (64) of the target object.

7. The method (10) for sensor data processing according to any one of the preceding claims, characterized in that The sensor data (18) are present as a spectrum (22).

8. The method (10) for sensor data processing according to claim 7, characterized in that The spectrum (22) is at least two-dimensional.

9. The method (10) for sensor data processing according to any one of the preceding claims, characterized in that The feature vectors (40) each contain information from the associated information section (34) derived by processing in the processing model (36).

10. A processing device, which is configured to implement the method (10) for sensor data processing according to any one of the preceding claims, the processing device having an environmental sensor (12) for detecting at least one target object in a sensor environment, a computing unit (44) for implementing the processing model (36), and A computing unit (46) for implementing a further processing model (42).

Images