Device and method for processing sensor data, and sensor system

Abstract

The invention relates to the processing of sensor data of one or more sensors (2-1, 2-2, 2-n), wherein the sensor data can be provided as a collection of multiple points (100). The points (100) are projected (12) onto node elements of a plurality of grids (110, 120) with different scales. The sensor data can then be further processed (13, 200) on the basis of the plurality of grids (110, 120).

Description

[0001] Description

[0002] title

[0003] Device and method for processing sensor data, sensor system

[0004] Technical area

[0005] The present invention relates to a device and a method for processing sensor data and to a sensor device comprising such a device.

[0006] background

[0007] The operation of driver assistance systems, such as those used in fully or at least partially autonomous vehicles, requires the most precise possible knowledge of the vehicle's surroundings. Various sensor systems can be used to detect the surroundings. For example, radar sensors can be used to detect objects, their position, and, if necessary, their relative movement. The sensors provide data on individual points, for example, as well as additional properties such as backscatter coefficient or similar for these points. This point data can then be further processed to identify and, if necessary, classify related objects.

[0008] For example, the publication DE 10 2018 214 959 A1 describes a method for evaluating sensor data, wherein the sensor data is provided by at least one sensor. Based on the sensor data, objects can be detected and their surface properties determined.

[0009] Disclosure of the invention

[0010] The present invention provides a device and a method for processing sensor data, as well as a sensor system, with the features of the independent claims. Further advantageous embodiments are the subject of the dependent claims. Accordingly, it is provided:

[0011] A device for processing sensor data, comprising an input device and a projection device. The input device is configured to receive multiple sensor data items as points. Each point comprises at least data relating to a position of the respective point. Optionally, at least one further property can be specified for the points. The projection device is configured to project the points onto multiple grids. Each of the multiple grids has a different scale. In other words, the projection device generates multiple grids with different scales, with the points received by the input device being projected onto grid nodes in all grids according to the respective scale.

[0012] Furthermore, it is planned:

[0013] A sensor system comprising a sensor device and a device according to the invention for processing the sensor data. The sensor device is designed to sense the surroundings of the sensor. The sensor device is further designed to provide sensor data. The sensor data is provided as points with a spatial position and, optionally, at least one further corresponding property.

[0014] Finally, it is planned:

[0015] A method for processing sensor data, comprising a step of providing multiple sensor data items. The sensor data is provided as points. Each of the points includes at least data on one position of the respective point. If necessary, at least one further property can also be specified for one, several, or all points. The method further comprises a step of projecting the points onto multiple grids. The multiple grids have different scales. Thus, each point of the sensor data is projected onto a position in each of the grids according to the respective scale.

[0016] Advantages of the Invention In many application areas, it is desirable to derive information about objects from sensor data. Different approaches can be pursued for this purpose. One possibility, for example, is to feed information from the sensor data into a neural network trained to identify objects and / or object properties from the sensor data. The sensor data can be made available in various ways.

[0017] If the sensor data comes from a radar sensor, for example, the sensor data can specify a position in the sensor's field of view. Furthermore, at least one further property, such as a backscatter coefficient or something similar, can be specified for each point. In principle, however, the further properties of the points can be any specific or non-specific properties. In particular, in addition to specific properties such as a radar backscatter coefficient, any non-specific properties that cannot be specifically "explained" are also possible. In such cases, a downstream neural network can also be trained on such non-specific properties. Furthermore, it is also possible that different properties are provided for different points.

[0018] For an extended object within the sensor's field of view, the sensor data may, for example, comprise point clouds with numerous points. For further processing, e.g., using a neural network or similar, these disordered point clouds can be converted into a grid. Due to the discrete distances between the individual grid points, the sensor data must be converted from the original point cloud to the discrete positions of the specified grid. This may lead to inaccuracies or a reduction in information content due to discretization.

[0019] Therefore, one idea of ​​the present invention is to take this finding into account and not only transfer the sensor data in the form of an initially disordered point cloud to a single grid with a single scale, but to generate multiple grids with different scales, i.e., different grid spacings. For further processing, the multiple grids with the different scales can then be used simultaneously. Due to the different scales, greater accuracy can be achieved. The loss of information due to the assignment of discrete grid points is significantly lower when using multiple grids with different scales than when using only a single grid.

[0020] According to one embodiment, the multiple grids onto which the points are projected each refer to the same (spatial) area. In other words, the individual grids each represent the same spatial extent. The distances between neighboring grid points are preferably equidistant.

[0021] According to one embodiment, the sensor data is provided as an unordered list of point data. This enables particularly simple provision of the sensor data, as it can be processed without special sorting or similar.

[0022] According to one embodiment, the projection device is designed to summarize properties of points projected onto a common grid element. In other words, if two or more sensor data items relate to elements projected onto a common grid position in one of the multiple grids, the corresponding grid element can receive a value composed of the combination of the relevant points. The properties of the individual points can be individually weighted if necessary. For example, the properties of the individual points can be weighted according to their (spatial) distance from the respective grid position. In principle, any suitable approaches for summarizing the properties of multiple points for a common grid element are also possible.

[0023] According to one embodiment, the device for processing the sensor data comprises a filter device. This filter device is designed to filter the sensor data received from the input device or to process it according to a predetermined rule. The filtered or processed sensor data can then be projected onto the multiple grids. This enables appropriate preparation or preprocessing of the raw sensor data, if required. According to one embodiment, the device for processing the sensor data comprises a processing device. This processing device can comprise a neural network. In particular, the processing device is designed to process the multiple grids with the points of the sensor data projected onto them. Through this processing, objects can be detected and / or properties of objects can be determined.In particular, the processing of the grids with the points can be carried out by the neural network of the processing device.

[0024] According to one embodiment, the properties of the objects determined by the processing device include a position of an object, a dimension of an object, and / or an orientation of an object. Furthermore, the properties may also include a classification of objects or any other object-specific properties.

[0025] The sensor device processed by the device according to the invention can be provided, for example, by a radar sensor, a LiDAR sensor, an ultrasonic sensor, and / or a camera. In particular, the camera can be a 3D camera, for example, a stereo camera or a time-of-flight camera.

[0026] The above embodiments and further developments can be combined with one another as desired, where appropriate. Further embodiments, further developments, and implementations of the invention also include combinations of features of the invention not explicitly mentioned above or described below with respect to the exemplary embodiments. In particular, those skilled in the art will also add individual aspects as improvements or additions to the respective basic forms of the invention.

[0027] Short description of the drawings

[0028] Further features and advantages of the invention are explained below with reference to the figures. Figure 1 shows a schematic representation of a block diagram of a sensor system with a device for processing the sensor data according to one embodiment;

[0029] Fig. 2: a schematic representation of a principle diagram illustrating the processing of sensor data according to an embodiment; and

[0030] Fig. 3: a flowchart underlying a method for processing sensor data according to an embodiment.

[0031] Description of embodiments

[0032] Figure 1 shows a schematic representation of a block diagram of a sensor system 1 according to one embodiment. The sensor system 1 can contain a plurality of sensors 2-i, which sense an environment and provide corresponding sensor data. The sensors 2-i can be, for example, radar sensors, lidar sensors, ultrasonic sensors, cameras, or any other suitable sensors capable of detecting objects in their environment, in particular within a predetermined field of view. A radar sensor can, for example, emit radar signals that are scattered and partially reflected by objects. The radar sensor can then receive and evaluate the radar echoes reflected back toward the radar sensor in order to generate information about objects in the environment of the radar sensor.For example, such a radar sensor can provide information about individual points at which the radar signals have been at least partially reflected back to the radar sensor. The information about such points can, for example, include a spatial position in the form of an azimuth and / or elevation angle or also in Cartesian coordinates. In addition, further information such as a relative speed, a backscatter coefficient or similar can be specified for such points. For extended objects, several such points can be output. These points can, for example, be referred to as a point cloud. Analogously, other sensors can also provide corresponding information about points on objects in the surrounding area. In particular, 3D cameras such as stereo cameras or time-of-flight cameras can also provide spatial information about objects in the surrounding area.Generally, the data for individual points includes at least one position. Furthermore, the data for a point may include one or more additional properties. These can be any specific or non-specific properties. In particular, in addition to specific properties such as a radar backscatter coefficient or similar, any non-specific properties that cannot be specifically explained are also possible. In such cases, a downstream neural network can also be trained on such non-specific properties. Furthermore, it is also possible to provide different properties for different points.

[0033] The sensor data provided by the sensors 2-i can then be transmitted, for example in the form of the previously described points or point clouds, to a device 10 for processing the sensor data. This device 10 can receive and process the sensor data, and, based on this sensor data, for example, identify objects and / or determine object features of objects in the field of view of the sensors 2-i. In particular, this evaluation of the sensor data can be carried out using a neural network or the like, as explained below.

[0034] The sensor data from the sensors 2-i can, in principle, be provided to the device 10 in any suitable manner. For example, the sensor data can be transmitted to the device 10 as an unordered list of the individual points and their feature(s).

[0035] The sensor data can, for example, be received by an input device 11 and temporarily stored as needed. Optionally, a filter device 14 can be provided, which performs preprocessing or filtering of the received sensor data. This can, for example, reduce noise in the sensor data. However, any other processes for preprocessing the sensor data are also fundamentally possible.

[0036] Furthermore, a projection device 12 is provided in the device 10 for processing the sensor data. This projection device 12 can convert the sensor data, which has been received in the form of individual points, possibly with corresponding additional properties, into a data structure in the form of a grid. For example, the points or the properties of a point can each be assigned to a node in the grid whose position is closest to the position specified in the sensor data. An n-dimensional matrix, for example, is possible as a data structure for the sensor data in such a grid form. This can preferably be a two-dimensional or three-dimensional matrix.

[0037] If the sensor data contains multiple points that should be assigned to the same node in the grid structure, the corresponding data element at this position can be formed by combining the corresponding features of the individual points in the sensor data. For example, the individual points can be weighted according to their actual distance from the position represented by the respective node in the grid structure. In principle, however, any approach is possible to combine the features of the individual points in the sensor data into a common element of the grid structure.

[0038] In the projection device 12, however, not just a single data structure is formed for a single grid, but rather multiple data structures for multiple different grids. The different grids have different scaling. Different scaling of the individual grids means that the distances between adjacent nodes in the grids each have different gradations. Preferably, the multiple grids refer to the same spatial area. This spatial area can correspond at least approximately to the spatial area to be covered by the sensor data. This can, for example, correspond to a field of view of the sensors 2-i.

[0039] The projection device 12 thus generates a plurality of grids, i.e., two or more grids, each of which has a different scale. In other words, the individual grids have a different number of nodes to preferably represent the same area. The resulting plurality of grids can then be fed to a processing device 13. The processing device 13 evaluates the plurality of grids together. Through such an evaluation, the processing device 13 can detect objects and / or determine properties of the detected objects using the data provided by the plurality of grids. For example, a spatial position, spatial extent, material properties, or the like can be determined for a detected object. If necessary, it is also possible to determine a direction of movement and / or speed.Of course, any other suitable properties of objects can be determined using the data from the multiple grids.

[0040] For processing the data of the multiple grids from the projection device 12, a neural network, for example, can be provided in the processing device 13. This neural network can have been previously trained in any suitable manner.

[0041] Figure 2 shows a schematic representation to illustrate the sensor data according to the concept of multiple grids. As shown here, a point cloud 100 with multiple points, each representing sensor data, can be converted into multiple grids 110, 120. The number of only two grids shown here is for illustrative purposes only and is not intended to limit the present invention. Three or more grids are also possible. Each point of the point cloud 100 is mapped to a grid element or node in each of the grids 110, 120. The individual grids 110, 120 preferably present at least approximately the same area, but with different scaling. Depending on the different scaling, the individual grids 110, 120 thus comprise different target data elements or nodes.

[0042] After all points of the point cloud 100 have been projected onto the grids 110, 120, the grids can be further processed with the sensor data. For this purpose, the grids 110, 120 can be fed, for example, to a neural network 200 or another suitable processing device 13. The processing device 13 can thus output the detected objects and / or properties of the detected objects. This information can then be further processed in any suitable system.

[0043] For example, the results of processing device 13 can be used in a driver assistance system or a system for fully or at least partially autonomous driving of a motor vehicle. However, any other suitable applications are also possible. For example, the information can also be used to monitor a traffic area by a stationary device.

[0044] Furthermore, the information about detected objects can also be used for access control. For example, object detection can also include the recognition of people, particularly facial recognition. It is also possible, for example, to monitor any area inside or outside a building using sensors and trigger predetermined events based on the object detection or classification. For example, an alarm can be triggered upon detection of unauthorized access, such as a break-in or similar.

[0045] In alternative embodiments, the object detection or classification described above can also be used in industrial systems, for example, in production plants or the like. Furthermore, any other applications based on the object detection or classification according to the invention are of course also possible, using the grid data of multiple grids with different scaling.

[0046] Figure 3 shows a flowchart that may underlie a method for processing sensor data according to one embodiment. The method may, in principle, comprise any steps as previously described in connection with the sensor system or device 10 for processing the sensor data. Likewise, the previously described components may comprise any elements as described below in connection with the method for processing the sensor data.

[0047] In step S1, several sensor data items are provided. The sensor data can be provided, in particular, as points. Each point of the sensor data can comprise a position, in particular a spatial position, as well as, if appropriate, one or more other corresponding properties.

[0048] In step S2, the points are then projected onto nodes of multiple grids. Each of the multiple grids has a different scale.

[0049] In summary, the present invention relates to the processing of sensor data from one or more sensors, where the sensor data may be present as a collection of multiple points. These points are projected onto node elements of multiple grids with different scales. Subsequently, further processing of the

[0050] Sensor data is based on these multiple grids.

Claims

Claims 1. A device (10) for processing sensor data, comprising an input device (11) configured to receive a plurality of sensor data as points, each point comprising at least data on a position of the point; a projection device (12) configured to project the points onto a plurality of grids (110, 120), each of the plurality of grids (110, 120) having a different scaling.

2. Device (10) according to claim 1, wherein the points further comprise data on at least one further property of the respective point.

3. Device (10) according to claim 1 or 2, wherein the plurality of grids (110, 120) each refer to a same area.

4. Device (10) according to one of claims 1 to 3, wherein the sensor data comprises an unordered list of data of the points.

5. Device (10) according to one of claims 1 to 4, wherein the projection device (12) is designed to summarize properties of points projected onto a common grid element.

6. Device (10) according to one of claims 1 to 5, comprising a filter device (14) which is designed to filter the sensor data received from the input device (11) and / or to process them according to a predetermined rule before the points are projected onto the plurality of grids.

7. Device (10) according to one of claims 1 to 6, with a processing device (13) which comprises a neural network and which is designed to process the plurality of grids (110, 120) with the points of the sensor data using the neural network in order to detect objects and / or to determine properties of objects.

8. The device according to claim 7, wherein the properties of the objects include a position, dimension and / or orientation of objects 9. A sensor system (1), comprising: a sensor device (2-i) configured to sense an environment and provide sensor data, wherein the sensor data is provided as points with a spatial position; and a device (10) for processing the sensor data according to one of claims 1 to 8.

10. Sensor system (1) according to claim 9, wherein the sensor device (2 -Lenny) comprises a radar sensor, a lidar sensor, an ultrasonic sensor and / or a camera.

11. Method for processing sensor data, comprising the steps: Providing (S1) a plurality of sensor data, wherein the sensor data are provided as points, and wherein each point comprises at least data on a position of the point; Projecting (S2) the points onto a plurality of grids (110, 120), each of the plurality of grids (110, 120) having a different scaling.

Images