Device and method for processing sensor data, and sensor system
Patent Information
- Application Number
- EP2024701138
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-02-15
- Filing Date
- 2024-01-16
- Publication Date
- 2025-12-24
AI Technical Summary
Existing methods for processing sensor data from disordered point clouds, such as those from radar sensors, often result in inaccuracies and data loss due to discretization when converting data into regular structures like grids for further processing.
A point-processing neural network, such as KPConv or GNN, is used to efficiently map points from unordered point clouds into regular data structures like grids, allowing for efficient aggregation of information with minimal data loss, enabling effective processing for applications like object detection and classification.
This approach enables precise and efficient conversion of sensor data into regular structures, reducing information loss and facilitating further processing, such as object detection and property determination, in applications like autonomous driving and object recognition systems.
Smart Images
Figure EP2024050850_22082024_PF_FP
Abstract
Description
[0001] Description
[0002] title
[0003] Device and method for processing sensor data. Sensor system
[0004] Technical area
[0005] The present invention relates to a device and a method for processing sensor data and to a sensor device comprising such a device.
[0006] background
[0007] The operation of driver assistance systems, such as those used in fully or at least partially autonomous vehicles, requires the most precise possible knowledge of the vehicle's surroundings. Various sensor systems can be used to detect the surroundings. For example, radar sensors can be used to detect objects, their position, and, if necessary, their relative movement. The sensors provide data on individual points, for example, as well as additional properties such as backscatter coefficient or similar for these points. This point data can then be further processed to identify and, if necessary, classify related objects.
[0008] For example, the publication DE 10 2018 214 959 A1 describes a method for evaluating sensor data, wherein the sensor data is provided by at least one sensor. Based on the sensor data, objects can be detected and their surface properties determined.
[0009] Disclosure of the invention
[0010] The present invention provides a device and a method for processing sensor data, as well as a sensor system, with the features of the independent claims. Further advantageous embodiments are the subject of the dependent claims. Accordingly, it is provided:
[0011] A device for processing sensor data with a transfer device. The transfer device is designed to receive a group of input data elements. Each input data element of this group of input data elements comprises a point that specifies at least one position. If necessary, one or more further properties can also be specified for the points. The transfer device further comprises a point-processing neural network. This point-processing neural network is designed to map the points of the group of input data elements to a regular output data structure. In particular, the regular output structure can comprise a grid structure with any dimension.
[0012] Furthermore, it is planned:
[0013] A sensor system comprising a sensor device and a device according to the invention for processing the sensor data. The sensor device is designed to sense the surroundings of the sensor. The sensor device is further designed to provide sensor data. The sensor data is provided as points that specify at least one position.
[0014] Finally, it is planned:
[0015] A method for processing sensor data, comprising a step of receiving a group of input data elements. Each input data element comprises a point with a position. Optionally, the points may also comprise at least one further property. Furthermore, the method comprises a step of mapping the points of the group of input data elements to a regular output data structure. The mapping is performed using a point-processing neural network. Finally, the method comprises a step of outputting the regular output data structure. This regular
[0016] The output structure can, for example, be output to any other device for further processing. Advantages of the invention
[0017] In many application areas, it is desirable to derive information about objects from sensor data. Different approaches can be used for this. One possibility, for example, is to feed information from the sensor data into a neural network trained to identify objects and / or object properties from the sensor data. The sensor data can be made available in different ways.
[0018] If the sensor data originates from a radar sensor, for example, the sensor data can include a position in the sensor's field of view as well as at least one other property, such as a backscatter coefficient or similar. For an extended object in the sensor's field of view, the sensor data can therefore include point clouds with numerous points. For further processing, e.g. using a neural network or similar, these disordered point clouds must be converted into a regular structure such as a grid. Due to the discrete distances between the individual grid points, the sensor data must be converted from the original point cloud to the discrete positions of the specified grid. This can potentially lead to inaccuracies or a reduction in the information content due to discretization.
[0019] It is therefore an idea of the present invention to take this finding into account and to create a concept for the transition of sensor data from a disordered point cloud into a regular data structure that is suitable for further processing efficiently and with the least possible data loss. For this purpose, a point-processing neural network is provided for the transition of points from a disordered point cloud into a regular structure. Such point-processing neural networks are capable of transferring information from any number of points or vectors to a regular structure, for example, individual grid cells. By using a point-processing neural network, the aggregation of information from the points to the individual grid cells can be carried out particularly efficiently and with very little information flow.The resulting regular data structure in the form of a grid is highly advantageous or even essential for further processing, for example, using a convolutional neural network (CNN). Thus, by appropriately transforming the point data from an initially unstructured point cloud into the regular structure, the required data structure can be provided efficiently and with as little data loss as possible.
[0020] According to one embodiment, the points further comprise data with at least one further property of the respective point. In principle, the further properties of the points can be any specific or non-specific properties. In particular, in addition to specific properties such as a radar backscatter coefficient, any non-specific properties that cannot be specifically explained are also possible. In such cases, a downstream neural network can also be trained on such non-specific properties. Furthermore, it is also possible to provide different properties for different points.
[0021] According to one embodiment, the point-processing neural network comprises a kernel point convolution (KPConv) or a graph neural network (GNN). However, any other suitable point-processing neural network capable of efficiently and with minimal loss converting properties of the data of an unstructured product cloud into a regular structure is also possible.
[0022] According to one embodiment, the device for processing the sensor data comprises a processing device. This processing device is designed to detect an object and / or determine properties of an object using the information contained in the output data. According to one embodiment, the properties of the objects can be a position, a dimension, and / or orientation of the objects. Furthermore, any other properties for characterizing the objects are of course also possible. For example, surface properties, direction of movement, or the like can also be determined for a detected object.
[0023] According to one embodiment, the processing device comprises a neural
[0024] Network. This neural network can be designed to detect an object and / or determine properties of an object using the information specified in the output data structure. Depending on the application, any suitable neural network can be used for this purpose. Such a neural network can be previously trained for the respective application using suitable training data.
[0025] According to one embodiment, the processing device comprises a filter device. The filter device can be configured to filter the received input data elements and / or process them according to another predetermined rule. The filtering or processing can, in particular, take place before the points are mapped to the regular output data structure.
[0026] The sensor device processed by the device according to the invention can be provided, for example, by a radar sensor, a LiDAR sensor, an ultrasonic sensor, and / or a camera. In particular, the camera can be a 3D camera, for example, a stereo camera or a time-of-flight camera.
[0027] The above embodiments and developments can be combined with each other as appropriate. Further embodiments, developments, and implementations of the invention also include combinations of features of the invention not explicitly mentioned above or described below with regard to the exemplary embodiments. In particular, the person skilled in the art will also add individual aspects as improvements or additions to the respective basic forms of the invention.
[0028] Short description of the drawings
[0029] Further features and advantages of the invention are explained below with reference to the figures. These show:
[0030] Fig. 1: a schematic representation of a block diagram of a sensor system with a device for processing the sensor data according to an embodiment; Fig. 2: a schematic representation of a principle diagram for illustrating the processing of sensor data according to an embodiment; and
[0031] Fig. 3: a flowchart underlying a method for processing sensor data according to an embodiment.
[0032] Description of embodiments
[0033] Figure 1 shows a schematic representation of a block diagram of a sensor system according to one embodiment. The sensor system can include a plurality of sensors 2-i, which sense an environment and provide corresponding sensor data. The sensors 2-i can be, for example, radar sensors, lidar sensors, ultrasonic sensors, cameras, or any other suitable sensors capable of detecting objects in their environment, in particular within a predetermined field of view. A radar sensor can, for example, emit radar signals that are scattered and partially reflected by objects. The radar sensor can then receive and evaluate the radar echoes reflected back toward the radar sensor in order to generate information about objects in the environment of the radar sensor.For example, such a radar sensor can provide information about individual points at which the radar signals have been at least partially reflected back to the radar sensor. The information about such points can, for example, include a spatial position in the form of an azimuth and / or elevation angle or also in Cartesian coordinates. In addition, further information such as a relative speed, a backscatter coefficient or similar can be specified for such points. For extended objects, several such points can be output. These points can, for example, be referred to as a point cloud. Analogously, other sensors can also provide corresponding information about points on objects in the surrounding area. In particular, 3D cameras such as stereo cameras or time-of-flight cameras can also provide spatial information about objects in the surrounding area.
[0034] Generally, the data for individual points includes at least one position. Furthermore, the data for a point may include one or more additional properties. These can be any specific or non-specific properties. In particular, in addition to specific properties such as a radar backscatter coefficient or similar, any non-specific properties that cannot be specifically explained are also possible. In such cases, a downstream neural network can also be trained on such non-specific properties. Furthermore, it is also possible to provide different properties for different points.
[0035] The sensor data provided by the sensors 2-i can then be transmitted, for example in the form of the previously described points or point clouds, to a device 1 for processing the sensor data. This device 1 can receive and process the sensor data, and, based on this sensor data, for example, identify objects and / or determine object features of objects in the field of view of the sensors 2-i. In particular, this evaluation of the sensor data can be carried out using a neural network or the like, as explained below.
[0036] The sensor data from the sensors 2-i can, in principle, be provided to the device 1 in any suitable manner. For example, the sensor data can be transmitted to the device 1 as an unordered list of the individual points and their feature(s).
[0037] The sensor data can be received by the sensor data processing device 1 and temporarily stored as needed. Optionally, a preprocessing device 13 can be provided, which performs preprocessing, for example, filtering of the received sensor data. This can, for example, reduce noise in the sensor data or perform any other operations for preprocessing the sensor data.
[0038] Furthermore, a transfer device 11 is provided in the device 1 for processing the sensor data. This transfer device 11 can convert the sensor data, which is still present in a disordered form in the form of points, into a regular output data structure. In such a regular structure, the sensor data can be provided, for example, as elements of a two- or multi-dimensional grid. This desired regular data structure can, in particular, be a data structure suitable for further processing by a downstream processing device 12. If, for example, the processing device 12 is designed to further process the data in a two-dimensional or three-dimensional grid, the data structure required for this can be provided by the transfer devices 11.
[0039] A point-processing neural network is provided in the transfer device 11 for mapping or projecting the data from the points of the initially disordered data cloud containing the sensor data into the regular data structure. For example, this point-processing neural network can be a kernel point convolution (KPConv) or a graph neural network (GNN). Furthermore, any other suitable point-processing neural networks are also possible. The point-processing neural network can initially be trained in a suitable manner according to the requirements, in particular the desired regular output data structure and, if applicable, the data structure of the input data.
[0040] The output data from the transfer device 11, i.e., the data in the grid-like data structure according to the requirements of the processing device 12, can then be fed to the processing device 12. The processing device 12 then evaluates the data in the provided grid-like data structure. Through such an evaluation, the processing device 12 can, for example, detect one or more objects. Additionally or alternatively, the processing device 12 can also determine properties of the detected objects. For example, a spatial position, a spatial extent, material properties, in particular surface properties, or the like can be determined for a detected object. If necessary, it is also possible to determine a direction of movement and / or a speed of an object.Of course, any other suitable properties of objects can be determined using the data in the grid-like data structure.
[0041] To process the data, the processing device 12 can, for example, use a neural network. Such a neural network can be trained in advance in any desired way. Figure 2 shows a schematic diagram illustrating the concept for transferring the sensor data from the sensors 2-i into the regular data structure for further processing by the processing device 12.
[0042] As shown here, for example, a point cloud 100 with multiple points can be provided by the sensors 2-i. The individual points each represent sensor data. In addition to a spatial and / or temporal position of the individual points, the points can each have at least one further property, for example, a backscatter coefficient in the case of a radar sensor. In principle, however, any suitable properties are possible. The individual data can be provided in the point cloud 100, for example, in the form of an unstructured list or similar.
[0043] The transfer device 11 can convert the unstructured data cloud 100 into a regular data structure 110. As previously explained, a point-processing neural network, such as a KPConv or GNN, can be used to map or transform the points from the unstructured data cloud 100 into the regular data structure 110. This point-processing neural network can be trained in advance in a suitable manner according to the respective requirements.
[0044] The regular data structure 110 can then be provided to the processing device 12, and the processing device 12 can then detect objects and / or determine object properties based on this regular or grid-like data structure. This information can then be further processed in any desired manner.
[0045] For example, the results of processing device 12 can be used in a driver assistance system or a system for fully or at least partially autonomous driving of a motor vehicle. However, any other suitable applications are also possible. For example, the information can also be used to monitor a traffic area by a stationary device.
[0046] Furthermore, the information about detected objects can also be used for access control. For example, object detection can also include the recognition of people, particularly facial recognition. It is also possible, for example, to monitor any area inside or outside a building using sensors and trigger predetermined events based on the object detection or classification. For example, an alarm can be triggered upon detection of unauthorized access, such as a break-in or similar.
[0047] In alternative embodiments, the object detection or classification described above can also be used in industrial systems, for example, in production plants or the like. Furthermore, any other applications based on the object detection or classification according to the invention are of course also possible, using the grid data of multiple grids with different scaling.
[0048] Figure 3 shows a flowchart that may underlie a method for processing sensor data according to one embodiment. The method may, in principle, comprise any steps as previously described in connection with the sensor system or device 1 for processing the sensor data. Likewise, the previously described components may comprise any elements as described below in connection with the method for processing the sensor data.
[0049] In step S1, a group of input data elements is received. Each input data element comprises a point with a position. Furthermore, the points may optionally also comprise at least one additional property.
[0050] In step S2, the points of the group of input data elements are then mapped to a regular output data structure. This mapping is performed, as previously described, using a point-processing neural network.
[0051] Finally, in step S3, the generated regular output data structure can be output.
[0052] In a further step, the data from the regular
[0053] The output data structure can be processed to detect objects and / or determine object properties. The output data can be processed, in particular, using a neural network.
[0054] In summary, the present invention relates to the processing of sensor data from one or more sensors, wherein the sensor data is provided as a disordered point cloud. The points of this disordered sequence are then converted into a regular structure using a point-processing neural network and provided for further processing.
Claims
Claims 1. Device (1) for processing sensor data, comprising: a transfer device (11) designed to receive a group of input data elements, each input data element comprising a point with a position, and wherein the transfer device (11) comprises a point-processing neural network designed to map the points of the group of input data elements to a regular output data structure.
2. Device (1) according to claim 1, wherein the points further comprise data with at least one further property of the respective point.
3. Device (1) according to one of claims 1 or 2, wherein the point-processing neural network comprises a kernel point convolution, KPConv, or a graph neural network, GNN.
4. Device (1) according to one of claims 1 to 3, comprising a processing device (12) which is designed to detect an object and / or to determine properties of an object using the regular output data structure.
5. Device (1) according to claim 4, wherein the properties of the objects include a position, dimension and / or orientation of objects 6. Device (1) according to claim 4 or 5, wherein the processing device (12) comprises a neural network designed to detect an object and / or determine properties of an object using the regular output data structure.
7. Device (1) according to one of claims 1 to 6, with a filter device (13) which is designed to filter the received input data elements and / or according to a predetermined rule before the points are mapped to the regular output data structure.
8. A sensor system comprising: a sensor device (2-i) configured to sense an environment and provide sensor data, the sensor data being provided as points; and a device (1) for processing the sensor data according to one of claims 1 to 7.
9. Sensor system according to claim 8, wherein the sensor device comprises a radar sensor, a lidar sensor, an ultrasonic sensor and / or a camera.
10. Method for processing sensor data, comprising the steps: Receiving (Sl) a group of input data elements, each input data element comprising a point with a position; Mapping (S2) the points of the group of input data elements to a regular output data structure using a point-processing neural network; and Output (S3) the regular output data structure.