Apparatus and method for processing sensor data, sensor system

Abstract

The present invention relates to processing sensor data from one or more sensors, where the sensor data is provided as a chaotic point cloud. This chaotic set of points is then transformed into a regular structure by a point processing neural network and provided for further processing. A transformation mechanism (11) is designed to receive a set of input data elements from sensors (2-1, 2-2, 2-3), such as radar sensors, lidar sensors, ultrasonic sensors, and / or cameras. Each input data element of this set of input data elements includes a point that specifies at least one location. In some cases, one or more additional properties may also be specified for this point. The transformation mechanism (11) further includes a point processing neural network. This point processing neural network is designed to map the points of the set of input data elements to a regular output data structure. In particular, the regular output structure may include a lattice structure of any dimension. A processing mechanism (12) is designed to use the regular output data structure to detect objects and / or determine their properties. Optionally, a pre-processing mechanism (13) is provided for pre-processing the received sensor data, for example filtering. This concept for converting sensor data from a chaotic point cloud into a regular data structure is suitable for further processing efficiently with as little data loss as possible. For this, a point processing neural network is provided for converting the points of the chaotic point cloud into a regular structure.

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for processing sensor data and a sensor device comprising such an apparatus. [Background technology]

[0002] The operation of driver assistance systems, especially those used in fully or at least partially autonomous vehicles, requires as accurate an understanding of the vehicle's surroundings as possible. Various sensor systems can be used to capture the surroundings. For example, radar sensors can be used to detect objects, their positions, and possibly their relative movements. The sensors provide, for example, data on individual points and possibly additional characteristics, such as backscatter coefficients or the like, for these points. These point data can then be further processed to identify and possibly classify the associated objects.

[0003] Document DE 10 2018 214 959 A1 describes, for example, a method for evaluating sensor data, which are supplied by at least one sensor, on the basis of which objects can be detected and the surface properties of the objects can be determined. Summary of the Invention

[0004] The present invention provides an apparatus and method for processing sensor data and a sensor system having the features of the independent patent claims. Further advantageous embodiments are the subject of the dependent claims.

[0005] Based on this: It is contemplated that an apparatus for processing sensor data is provided that includes a transformation mechanism. The transformation mechanism is designed to receive a set of input data elements, each of which includes a point that specifies at least one location. In some cases, one or more additional properties may also be specified for the point. The transformation mechanism further includes a point-processing neural network that is designed to map the points of the set of input data elements to a regular output data structure. In particular, the regular output structure may include a lattice structure of any dimension.

[0006] moreover: It is contemplated that a sensor system is provided, comprising a sensor arrangement and an apparatus according to the invention for processing sensor data, the sensor arrangement being designed for capturing the surrounding environment of the sensor by means of a sensor, the sensor arrangement being further designed for providing sensor data, the sensor data being provided as points that define at least one position.

[0007] lastly: It is contemplated that a method for processing sensor data is provided that includes receiving a set of input data elements, where each input data element includes a point having a location. Optionally, the point may also include at least one additional characteristic. The method further includes mapping the points of the set of input data elements to a regular output data structure. The mapping is performed using a point processing neural network. Finally, the method includes outputting the regular output data structure. The regular output structure may be output to any additional device, for example, for further processing.

[0008] Advantages of the Invention In many application areas, it is desirable to derive information about objects from sensor data. Various approaches can be pursued for this purpose. One possibility is, for example, to feed the sensor data information to a neural network that is trained to determine objects and / or object properties from the sensor data. In this regard, the sensor data can be provided in various ways.

[0009] For example, if the sensor data originates from a radar sensor, the sensor data may include a position within the sensor's field of view and at least one additional characteristic, such as a backscattering coefficient or the like. Thus, for a single, extended object within the sensor's field of view, the sensor data may include a point cloud with numerous points. These disordered point clouds must be transformed into a regular structure, such as a raster or grid, for further processing, for example, using a neural network or the like. Based on the discrete spacing between the individual grid points, the sensor data must be transformed from the original point cloud into discrete positions of the configured grid. This may result in errors or a reduction in the information content due to the discretization.

[0010] Therefore, the idea of ​​the present invention is to take this knowledge into account and provide a concept for converting sensor data from a chaotic point cloud into a regular data structure suitable for further processing efficiently with as little data loss as possible.For this purpose, a point processing neural network is provided to convert the points of the chaotic point cloud into a regular structure.Such a point processing neural network can convert the information of any number of points or vectors into a regular structure, for example, into individual grid cells.By using this point processing neural network, the aggregation of the point information into individual grid cells can be performed particularly efficiently and with very little information flow.

[0011] The resulting regular data structure in the form of a grid is highly advantageous or even necessary for further processing, for example by a convolutional neural network (CNN), since the data structure required for this purpose can be generated efficiently and with as little data loss as possible by correspondingly converting the point data from the initially unstructured point cloud into a regular structure.

[0012] According to one embodiment, the points further comprise data with at least one further characteristic of the respective point. Essentially, the further characteristic of the points can be any specific characteristic or even an unspecific characteristic. In particular, it can be a specific characteristic, such as the radar backscatter coefficient, but also any unspecific characteristic that is not specifically "explainable". In such a case, the subsequent neural network can also be trained on such an unspecific characteristic. Furthermore, different characteristics can be provided for different points.

[0013] According to one embodiment, the point processing neural network comprises a kernel point convolution, KPConv, or a graph neural network, GNN, although essentially any other suitable point processing neural network is possible that can transform the unstructured characteristics of the product cloud data into a regular structure in an efficient and low-loss manner.

[0014] According to one embodiment, the device for processing sensor data includes a processing mechanism designed to detect objects and / or determine object properties using information contained in the output data. According to one embodiment, the object properties may be the object's position, dimensions, and / or orientation. However, any other properties for characterizing the object are also possible. For example, surface properties, movement direction, or the like may be determined for the detected object.

[0015] According to one embodiment, the processing mechanism includes a neural network that can be designed to detect objects and / or determine object characteristics using the information specified in the output data structure. Any suitable neural network can be used for this purpose, depending on the application. Such a neural network can be previously trained with suitable training data, depending on the application.

[0016] According to one embodiment, the processing mechanism includes a filter mechanism, which may be designed to filter the received input data elements and / or process them based on other predetermined rules. This filtering or processing may be performed, among other things, before the points are mapped to a regular output data structure.

[0017] The sensor mechanism processed by the device according to the invention may be provided for example by a radar sensor, a LiDAR sensor, an ultrasonic sensor and / or a camera, which may in particular be a 3D camera, for example a stereo camera or a ToF camera.

[0018] The above-described embodiments and variations may be combined with one another in any meaningful way. Further embodiments, variations, and implementations of the invention include not-explicitly-listed combinations of the features of the invention described above or below in connection with the exemplary embodiments. In particular, those skilled in the art will also add individual aspects as improvements or supplements to each basic embodiment of the invention.

[0019] Further features and advantages of the present invention are described below with reference to the accompanying drawings. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a schematic block diagram of a sensor system with an apparatus for processing sensor data according to an embodiment; [Figure 2]FIG. 1 is a schematic principle diagram for illustrating the processing of sensor data according to an embodiment. [Figure 3] FIG. 1 is a flow diagram underlying a method for processing sensor data according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0021] FIG. 1 shows a schematic block diagram of a sensor system according to one embodiment. Within this sensor system, multiple sensors 2-i may be provided, capturing the surrounding environment and providing corresponding sensor data. The sensors 2-i may be, for example, radar sensors, lidar sensors, ultrasonic sensors, cameras, or any other suitable sensors suitable for detecting objects in the surrounding environment, particularly within a set field of view. The radar sensor may emit, for example, a radar signal, which is scattered and partially reflected by objects. Based on this, the radar sensor may receive and evaluate radar echoes reflected back toward the radar sensor to generate information about objects in the surrounding environment of the radar sensor. For example, such a radar sensor may provide information about individual points that at least partially reflected the radar signal back toward the radar sensor. The information about such points may include, for example, spatial positions in the form of azimuth and / or elevation angles, or even spatial positions in Cartesian coordinates. Furthermore, further information, such as relative velocities, backscatter coefficients, or the like, may also be specified for such points. In this regard, in the case of an object with a large area, a plurality of such points can be output. These points can be called, for example, a point cloud. Similarly, other sensors can provide corresponding information about points on objects in the surrounding environment. In particular, 3D cameras, such as stereo cameras or ToF cameras, can also provide spatial information about objects in the surrounding environment.

[0022] Basically, the data of each point includes at least one position. Furthermore, the data of a point may optionally include one or more further characteristics. This may be any specific characteristic or may be an unspecific characteristic. In particular, this may be a specific characteristic, such as a radar backscatter coefficient, but also any unspecific characteristic that is not specifically "explainable." In such a case, the subsequent neural network may also be trained on such unspecific characteristics. Furthermore, different characteristics may be provided at different points.

[0023] On this basis, the sensor data supplied by the sensor 2-i, for example in the form of the aforementioned points or point clouds, can be transmitted to the device 1 for processing the sensor data. This device 1 receives the sensor data, processes it and, based on this sensor data, can for example identify objects and / or determine object characteristics of objects in the field of view of the sensor 2-i. In particular, this evaluation of the sensor data can be performed using a neural network or the like, as will be explained further below.

[0024] The sensor data from the sensor 2-i may be provided in essentially any suitable manner to the device 1. The sensor data may, for example, be conveyed to the device 1 as an unordered list of individual points and one or more features thereof.

[0025] The sensor data may be received by the device for processing sensor data 1 and temporarily stored as needed. Optionally, a pre-processing mechanism 13 may be provided that performs pre-processing of the received sensor data, for example filtering, so as to reduce noise in the sensor data, or any other steps for pre-processing of the sensor data may be performed.

[0026] Furthermore, a conversion mechanism 11 is provided in the device 1 for processing sensor data. This conversion mechanism 11 can convert the sensor data, which is still present in a chaotic form in the form of points, into a regular output data structure. In such a regular structure, the sensor data can be provided, for example, as elements of a two- or more-dimensional grid. This desired regular data structure can be, in particular, a data structure that is suitable for a downstream processing mechanism 12 for further processing. For example, if the processing mechanism 12 is designed for further processing two- or three-dimensional grid data, the conversion mechanism 11 can provide the data structure required for this purpose.

[0027] To map or project the point-based data of the initially disordered data cloud of sensor data into a regular data structure, a point-processing neural network is provided in the conversion mechanism 11. This point-processing neural network can be, for example, a kernel point convolution (KPConv) or a graph neural network (GNN). Of course, any other suitable point-processing neural network is also possible. In this regard, the point-processing neural network can first be appropriately trained based on the requirements, in particular based on the desired regular output data structure and, if necessary, based on the data structure of the input data.

[0028] Based on this, output data from the conversion mechanism 11, i.e., data in a grid data structure based on the request of the processing mechanism 12, can be sent to the processing mechanism 12. The processing mechanism 12 then evaluates the provided data in the grid data structure. Through such evaluation, the processing mechanism 12 can, for example, detect one or more objects. Additionally or alternatively, the processing mechanism 12 can also determine characteristics of the detected objects. For example, for the detected objects, spatial position, spatial extent, material properties, particularly surface properties, or the like can be determined. In some cases, the movement direction and / or speed of the object can also be determined. Of course, any other suitable characteristics of the object can also be determined using the data in the grid data structure.

[0029] To process the data, the processing mechanism 12 may, for example, utilize a neural network. Such a neural network may be trained in any manner beforehand. FIG. 2 shows a schematic diagram to illustrate the concept of rewriting sensor data from sensors 2-i into regular data structures for further processing by processing mechanism 12.

[0030] As shown here, a point cloud 100 of a plurality of points can be provided, for example, by sensors 2-i, where each individual point represents sensor data. These points can represent not only the spatial and / or temporal position of the individual points, but possibly also at least one further characteristic, such as a backscattering coefficient in the case of a radar sensor. In principle, however, any suitable characteristic is possible. The individual data can be provided, for example, in the form of an unstructured list in the point cloud 100 or the like.

[0031] The conversion mechanism 11 can convert the unstructured data cloud 100 into a regular data structure 110. As already detailed above, a point processing neural network and KPConv or GNN can be used to map or convert points from the unstructured data cloud 100 to the regular data structure 110. The point processing neural network can be appropriately trained in advance based on respective requirements.

[0032] Based thereon, a regular data structure 110 can be provided to the processing mechanism 12, which can then detect objects and / or determine object properties based on this regular or grid-like data structure, which information can then be further processed in any manner.

[0033] For example, the results of the processing mechanism 12 may be used in a driver assistance system or a system for fully or at least partially autonomous driving of a motor vehicle, but any other suitable application is also possible, for example, the information may also be used for monitoring traffic spaces by stationary mechanisms.

[0034] Furthermore, information about detected objects can also be used for access control. For example, object detection can also include person recognition, especially facial recognition. For example, any area inside or outside a building can be monitored by sensors, and predetermined events can be triggered based on object detection or classification. For example, an alarm can be activated upon recognition of unauthorized entry, such as a home invasion or the like.

[0035] In alternative embodiments, the object detection or classification described above may also be used in industrial systems, for example in production facilities or the like, but any other application based on object detection or classification according to the invention using grid data of multiple grids with different scaling is of course also possible.

[0036] 3 shows a flow diagram on which a method for processing sensor data according to an embodiment can be based. This method can essentially include any steps as already described above in connection with the sensor system or the device for processing sensor data 1. Similarly, these components can include any elements as described below in connection with the method for processing sensor data.

[0037] In step S1, a set of input data elements is received, where each input data element comprises a point having a location, and optionally at least one further characteristic.

[0038] Based on this, in step S2 the points of the input data elements are mapped into a regular output data structure, this mapping being performed using a point processing neural network as already detailed above.

[0039] Finally, in step S3, the generated regular output data structure may be output. Based thereon, in a further step, the data of the regular output data structure can be processed to detect objects and / or to determine properties of the objects. The processing of the output data can be performed, inter alia, using a neural network.

[0040] In summary, the present invention relates to the processing of sensor data from one or more sensors, which is provided as a chaotic point cloud, which is then transformed into a regular structure by a point processing neural network and provided for further processing.

Claims

1. An apparatus (1) for processing sensor data, comprising: a transformation mechanism (11) designed to receive a set of input data elements, wherein each input data element comprises a point having a position; The apparatus (1), wherein the transformation mechanism (11) comprises a point processing neural network designed to map the points of the set of input data elements to a regular output data structure.

2. The apparatus (1) of claim 1, wherein the points further comprise data having at least one further characteristic of each of the points.

3. The device (1) according to claim 1 or 2, wherein the point processing neural network comprises a Kernel Point Convolution, KPConv, or a Graph Neural Network, GNN.

4. 4. The device (1) according to any one of claims 1 to 3, comprising a processing mechanism (12) designed to detect objects and / or determine properties of objects using said regular output data structure.

5. The apparatus (1) according to claim 4, wherein the properties of the object include the position, dimensions and / or orientation of the object.

6. 6. The apparatus (1) according to claim 4 or 5, wherein the processing mechanism (12) comprises a neural network designed to detect objects and / or determine properties of objects using the regular output data structure.

7. 7. The device (1) according to any one of claims 1 to 6, comprising a filter mechanism (13) designed to filter and / or process the received input data elements according to predetermined rules before the points are mapped into the regular output data structure.

8. a sensor mechanism (2-i) designed to capture the surrounding environment by a sensor and provide sensor data, said sensor data being provided as points; and Device (1) for processing sensor data according to any one of claims 1 to 7 A sensor system comprising:

9. The sensor system of claim 8 , wherein the sensor mechanism includes a radar sensor, a lidar sensor, an ultrasonic sensor, and / or a camera.

10. 1. A method for processing sensor data, comprising: receiving (S1) a set of input data elements, wherein each input data element comprises a point having a position; (S2) mapping the points of the set of input data elements to a regular output data structure using a point processing neural network; outputting the regular output data structure (S3); A method having the following.

Images