Method for processing sensor data, object detection network and method for adjusting object detection network

Abstract

The invention relates to a method for respectively processing sensor data of a plurality of environmental sensors indicating at least one environmental object, comprising: providing at least first sensor data of a first environmental sensor; providing at least second sensor data of a second environmental sensor; providing at least one first adapter unit and a second adapter unit; providing at least one object detection model, the object detection model having at least one trained artificial neural network; inputting first sensor data to the first adapter unit, and calculating first interface data that satisfies a prescribed interface data specification; inputting second sensor data to the second adapter unit, and calculating second interface data satisfying the interface data specification; the first and second interface data are input as input data to an object detection model, and at least one object parameter of the at least one environmental object is calculated as an output. The invention further relates to an object detection network and to a method for adjusting an object detection network.

Description

Technical Field

[0001] This invention relates to a method for processing sensor data. Furthermore, the invention also relates to an object detection network and a method for adjusting the object detection network. Background Technology

[0002] Current Advanced Driver Assistance Systems (ADAS) and Autonomous Driving (AD) functions used in vehicles require accurate detection and display of the vehicle's environment. This is achieved through the use of environmental sensors, such as cameras, lidar sensors, and radar sensors. Radar sensors play a particularly important role here because they not only operate reliably in poor visibility conditions, such as fog or darkness, but also provide detailed information about the vehicle's environment.

[0003] A common application area for radar sensors is object recognition. For this purpose, radar sensors generate point clouds composed of individual reflections. Each reflection is described by polar coordinates (such as range and azimuth) and other features (such as signal strength, radar cross-section (RCS), and elevation angle). Object detection models, most of which are currently based on deep learning, are used to identify relevant objects, such as cars, trucks, or pedestrians, from these point clouds. For example, object detection models determine the location, orientation (pose), category, and possibly other characteristics of environmental objects, and can represent these environmental objects in the form of "Oriented Bounding Boxes" (OBBs).

[0004] The output of an object detection model depends on the input data from environmental sensors, which may vary due to the specific characteristics of each sensor. Therefore, the object detection model is tuned and trained based on each environmental sensor. If an environmental sensor is replaced by a new, different type of sensor, the object detection model must be retrained with the new sensor data. This means that training data for the new environmental sensors must be detected and prepared before repeating the training process of the object detection model. Summary of the Invention

[0005] According to the present invention, a method for processing sensor data is proposed. The method includes the following steps:

[0006] Provide at least the first sensor data from the first environmental sensor;

[0007] At least the second sensor data from the second environmental sensor should be provided;

[0008] Provide at least one first adapter unit and a second adapter unit;

[0009] Provide at least one object detection model, which has at least one trained artificial neural network;

[0010] The first sensor data is input to the first adapter unit, and the first interface data that meets the specified interface data requirements is calculated as the output of the first adapter unit.

[0011] The second sensor data is input to the second adapter unit, and the second interface data that satisfies the interface data specification is calculated as the output of the second adapter unit;

[0012] The first interface data and the second interface data are input into the object detection model, and at least one object parameter of the at least one environment object is calculated as the output.

[0013] Therefore, the object detection model can perform computations independent of the type of environmental sensor providing the input data. The computation of the object detection model can be decoupled from the type of environmental sensor and the specific data format of the sensor data. Adjustments to the object detection network based on new sensor data formats or new environmental sensors can be performed more quickly and easily.

[0014] Environmental sensors can be radar sensors, lidar sensors, cameras, ultrasonic sensors, and / or accelerometers. They can be deployed on vehicles, devices, or mobile robots.

[0015] Environmental objects can be organisms, buildings, plants, objects, devices, or vehicles.

[0016] Sensor data from environmental sensors can exist as point clouds, spectrum, or time signals.

[0017] Object detection models can be configured for object recognition, object classification, semantic segmentation, and / or free space recognition. An object detection model can be configured for a single task, such as object recognition, or for multiple tasks, such as object recognition and object classification.

[0018] In the case of neural networks, object detection models can have multiple layers, including at least one input layer, multiple intermediate layers, and at least one output layer.

[0019] Object detection models can be convolutional neural networks (CNNs). CNNs use convolutional layers, especially two-dimensional convolutional layers, to filter and extract features from interface data that serves as input data, particularly to identify structures and patterns in the input data. A CNN includes at least one convolutional layer, a pooling layer, and / or a dense output layer. The pooling layer can be an average pooling layer or a max pooling layer. Global pooling can be applied to the pooling layer. Convolutional layers can be used for feature extraction, pooling layers can be used to reduce the spatial size of features, and dense output layers can be used for classification. In cases where memory requirements are particularly high, a CNN can use only one convolutional layer and one pooling layer, which eliminates the need to store convolutional features.

[0020] Interface data can be used as input data in a sensor fusion model. The sensor fusion model can be part of an object detection model, or upstream of it.

[0021] Interface data specifications can be defined regarding the data structure, format, data type, and / or quality of interface data. By adhering to interface data specifications, uniform interface data can exist regardless of the structure of the input data.

[0022] Interface data can be mapped to the embedding space through their respective adapter units. Interface data specifications can define: the dimensions of the embedding space, the schema specifications within the embedding space, the structural specifications, and / or boundary values. Interface data specifications can form a standard for interface data, through which interface data can exist in a unified data format, with a unified data type, and / or with a unified data structure.

[0023] The adapter unit and the object detection model can form an object detection network that calculates at least one object parameter of at least one environmental object based on sensor data from environmental sensors as input data.

[0024] Training can be understood as an iterative process in which the neural network learns based on training data used to improve prediction accuracy. First, the neural network is defined as having input data along with its corresponding correct output (label) as the target. The neural network processes this data through its layers and outputs a prediction. Then, this prediction is compared to the actual label, and the error, or loss, is calculated. This loss indicates how far the neural network's prediction is from the actual answer. To minimize this error, the parameters of the neural network, especially the weights, need to be adjusted. This is achieved, for example, through gradient descent, where the gradient of the loss with respect to the weights is calculated. The weights are then changed in the direction that reduces the loss. This process is repeated iteratively, during which the neural network continuously adjusts its weights to reduce error and make more accurate predictions. The goal of training is to optimize the parameters in such a way that the neural network can perform accurate calculations even with new, unseen input data.

[0025] In a preferred embodiment of the invention, it is advantageous that the first and / or second adapter units include trained artificial neural networks. The training of the object detection model can be performed concurrently with the training of the first and second adapter units. Alternatively, the object detection model can be trained independently of the training of the first and second adapter units.

[0026] The model parameters of the object detection model can be trained together with the model parameters of the adapter unit at the beginning. Here, multiple sensor data from various environmental sensors can be used as training data. The model parameters of the adapter unit can be trained in the following ways: On one hand, the model parameters of the adapter unit can be trained only by using the training data generated by the environmental sensor to which the adapter unit belongs in a single run. Here, all training data can be formed by a random order of the sensor data from the environmental sensors. On the other hand, the training data can initially be generated using multiple sensor data from a single environmental sensor, and after multiple runs and training of the model parameters of the adapter unit assigned to that environmental sensor, the corresponding sensor data can be used as training data to train the next adapter unit.

[0027] The learning rates when training adapter units can be the same or different from each other. The learning rate when training the object detection model can be the same or different from the learning rate when training at least one adapter unit, or the same or different from the learning rate when training all adapter units.

[0028] If less training data exists for a particular environmental sensor and its associated adapter unit than for other environmental sensors and their associated adapter units, then this less training data can be reused when training the object detection network so that the same amount of training data is provided for each adapter unit during training. Alternatively, if the amount of training data used during training differs among the adapter units, different weights can be applied when calculating the loss function.

[0029] When the number of model parameters to be trained in the adapter units is different from each other, training data can be repeatedly applied to the adapter units with a larger number of model parameters, or different weights can be applied when calculating the loss function.

[0030] If a trained object detection model already exists, its earlier layers can be used as the basis for new adapter units to be trained. This allows for the initialization of the weights in these layers, which speeds up further training of the adapter units. This approach can be applied to one or more adapter units.

[0031] The adapter unit can also be pre-trained using unsupervised learning. The results of this unsupervised learning can serve as a starting point for subsequent supervised learning (weight initialization). This not only speeds up supervised learning but also reduces the need for labeled data. An example of unsupervised learning is training the adapter unit through an auxiliary task. Such an auxiliary task could be, for example, learning how to rotate sensor data, assumed to exist as points, as input data in 90-degree steps to conform to a specified point pattern defined as interface data.

[0032] The first and / or second adapter units may additionally or alternatively include deterministic algorithms.

[0033] A preferred configuration of the invention is advantageous in which the number of trained model parameters of the first and / or second adapter units is less than the number of trained model parameters of the object detection model. Thus, the object detection model can be quickly and easily used with new sensor data and / or with new environmental sensors that replace or supplement existing environmental sensors, without requiring adjustments to the object detection model itself or laborious modifications.

[0034] In an advantageous embodiment of the invention, the number of parameters for each adapter unit depends on the data mode of the corresponding sensor data. Here, the data mode indicates the data structure, data type, and / or data format of the sensor data. The larger the data structure, data type, and / or data format of the sensor data that serves as input data to the corresponding adapter unit, the larger the number of parameters that adapter unit may have. This allows for reliable processing of the amount of information contained in the sensor data.

[0035] In one particular embodiment of the invention, it is advantageous that at least one third adapter unit calculates third sensor data from a third environmental sensor into third interface data that satisfies the interface data specifications, and this third interface data forms additional input data for the object detection model. This can improve the amount and / or quality of the input data used for object detection. Other adapter units can also calculate other sensor data from other environmental sensors into other interface data that satisfies the interface data specifications, and this other interface data forms supplementary additional input data for the object detection model.

[0036] In a preferred embodiment of the invention, a third environmental sensor or at least one other environmental sensor is configured with a sensor mode different from that of the first and / or second environmental sensors, and / or the third sensor data or other sensor data is configured with a data mode different from that of the first and / or second sensor data. Through a unified interface data specification, the input data for the object detection model can exist uniformly and independently of the sensor mode and / or data mode. Here, the sensor mode indicates the sensor category of the environmental sensor. For example, a radar sensor is configured with a sensor mode different from that of a lidar sensor or a camera device.

[0037] In a preferred embodiment of the invention, the calculations for the first and second interface data can be performed in parallel, starting from their respective sensor data. The calculated first and second interface data can be processed by the object detection model sequentially, fused together, or in parallel.

[0038] According to the present invention, an object detection network for detecting environmental objects based on sensor data from environmental sensors is also proposed. The object detection network is configured to implement the processing method according to the present invention, and the object detection network has at least a first adapter unit, a second adapter unit, and an object detection model.

[0039] According to the present invention, a method for adjusting an object detection network is also proposed. The method includes the following steps:

[0040] Provides an object detection network according to the present invention;

[0041] A third environmental sensor is provided, which provides third sensor data, the third sensor data having a data mode different from the data modes of the first sensor data and the second sensor data;

[0042] The third adapter unit is trained using at least the third sensor data as input data and the third interface data that satisfies the interface data specification as the target specification.

[0043] The method for adjustment may further include: similar to the case of a third environmental sensor, adjusting based on at least one other environmental sensor.

[0044] In an advantageous embodiment of the invention, the model parameters of the object detection model are kept unchanged or trained at at most a lower learning rate than when training the third adapter unit during training. This reduces the workload required to adjust the object detection network based on an additional third environmental sensor.

[0045] According to the present invention, a computer program is also provided, having machine-readable instructions that can be implemented on at least one computer, which, when implemented, execute the above-described processing method or the above-described adjustment method.

[0046] According to the present invention, a memory cell is also provided, which is implemented in a machine-readable manner and can be accessed by at least one computer, and the aforementioned computer program is stored on the memory cell.

[0047] Other advantages and advantageous configurations of the invention will become apparent from the description and drawings. Attached Figure Description

[0048] The invention will now be described in detail with reference to the accompanying drawings. Specifically, the drawings show:

[0049] Figure 1 A method for processing sensor data in a specific embodiment of the present invention.

[0050] Figure 2 A method for adjusting an object detection network in one specific embodiment of the present invention. Detailed Implementation

[0051] Figure 1A method for processing sensor data is illustrated in a specific embodiment of the present invention. The method 10 for processing sensor data 14 indicating at least one environmental object 12 from a plurality of environmental sensors 16 includes: providing at least 18 first sensor data 20 from a first environmental sensor 22. The first environmental sensor 22 may be a radar sensor, and the first sensor data 20 may be associated with a first data mode 24, such as a point cloud. Furthermore, at least 26 second sensor data 28 from a second environmental sensor 30 is provided. The second environmental sensor 30 may be a camera device, and the second sensor data 28 may be associated with a second data mode 32 different from the first data mode 24, such as a camera device image with pixels.

[0052] Subsequently, at least one first adapter unit 36 ​​and a second adapter unit 38 are provided. The first and second adapter units 36 and 38 preferably each have a trained artificial neural network 40 with multiple layers.

[0053] Furthermore, at least one object detection model 44 is provided, which has at least one trained artificial neural network 46 with multiple layers. Here, the number of parameters of the trained model parameters of the first or second adapter units 36, 38 is particularly smaller than the number of parameters of the trained model parameters of the object detection model 44. The number of parameters of the corresponding adapter units 36, 38 depends, for example, on the data modality of the corresponding sensor data 14. This means that the larger the data structure and / or data format of the sensor data 14, the larger the number of parameters of the associated adapter units 36, 38. The object detection model 44, together with the first and second adapter units 36, 38, forms the object detection network 48.

[0054] Furthermore, the first sensor data 20 is input 50 to the first adapter unit 36, and the first adapter unit 36 ​​calculates first interface data 52 that satisfies the specified interface data specification 51. The interface data specification 51 specifies the interface data in terms of data structure, data type, and / or data format of the sensor data 14. The interface data specification 51 can form a standard for interface data, through which interface data can exist in a uniform data format, with a uniform data type, and / or with a uniform data structure. Here, the interface data specification 51 is fixedly specified.

[0055] In addition, in parallel with or after the calculation of the first interface data 52, the second sensor data 28 is input to the second adapter unit 38, and the second adapter unit 38 calculates the second interface data 56 that satisfies the interface data specification 51.

[0056] Subsequently, the first and second interface data 52 and 56 are input as input data 60 to the object detection model 44, and at least one object parameter 62 of at least one environment object 12 is calculated as the output of the object detection model 44. The object parameter 62 can be an object type, and the object detection model 44 can be used for object classification.

[0057] Figure 2 A method for adjusting an object detection network is illustrated in a specific embodiment of the invention. The method 64 for adjusting the object detection network 48 is, here, a method for extending the object detection network 48 66 to be able to process third sensor data from a third environmental sensor. The method includes: providing an object detection network 48 68 for detecting environmental objects based on first and second sensor data 20, 28 from first and second environmental sensors 22, 30. The object detection network 48 is configured to be implemented in… Figure 1 The method for processing described herein includes a first adapter unit 36, a second adapter unit 38, and an object detection model 44.

[0058] In addition, a third environmental sensor 72 is provided, which provides third sensor data 74. The third sensor data 74 has a different data mode than the data modes of the first and second sensor data 20, 28. For example, the third environmental sensor 72 is a lidar sensor, and the third sensor data 74 exists as a point cloud, but it differs in data structure, such as in the dimensions in which the point cloud is unfolded, which differs from the dimensions in which the point cloud of the first sensor data 20 is unfolded. Therefore, the third sensor data is associated with a different data mode.

[0059] Subsequently, using the third sensor data 74 as input data 80, and the third interface data 82, which is in the form of labeled data and satisfies the interface data specifications, as the target specification 84 during training 76, the third adapter unit 78 with a neural network having multiple layers 77 is trained. If the third adapter unit 78 has been trained, the object detection network 48 is extended to include the third adapter unit 78 and be able to process the third sensor data 74.

Claims

1. A method (10) for processing sensor data (14) of a plurality of environmental sensors (16) each indicating at least one environmental object (12), the method comprising the steps of: providing (18) at least first sensor data (20) of a first environmental sensor (22), providing (26) at least second sensor data (28) of a second environmental sensor (30), providing (34) at least one first adapter unit (36) and a second adapter unit (38), providing (42) at least one object detection model (44) having at least one trained artificial neural network (46), inputting (50) the first sensor data (20) into the first adapter unit (36) and computing first interface data (52) satisfying a prescribed interface data prescription (51) as an output of the first adapter unit (36), inputting (54) the second sensor data (28) into the second adapter unit (38) and computing second interface data (56) satisfying the interface data prescription (51) as an output of the second adapter unit (38), inputting (58) the first and second interface data (52, 56) as input data (60) into the object detection model (44) and computing at least one object parameter (62) of the at least one environmental object (12) as an output.

2. The method (10) for processing according to claim 1, characterized in that: The first and / or second adapter unit (36, 38) comprises a trained artificial neural network (40).

3. The method (10) for processing according to claim 2, characterized in that: The number of parameters of the trained model parameters of the first and / or second adapter unit (36, 38) is smaller than the number of parameters of the trained model parameters of the object detection model (44).

4. The method (10) for processing according to claim 3, characterized in that: The number of parameters of the respective adapter unit (36, 38) depends on the data modality of the respective sensor data (20, 28).

5. The method (10) for processing according to any of the preceding claims, characterized in that: At least one third adapter unit (78) computes third sensor data (74) of a third environmental sensor (72) as third interface data (82) satisfying the interface data prescription (51), the third interface data forming further input data for the object detection model (44).

6. The method (10) for processing according to claim 5, characterized in that: The third environmental sensor (72) is assigned a sensor modality different from the sensor modality of the first and / or second environmental sensor (22, 30), and / or the third sensor data (74) is assigned a data modality different from the data modality of the first and / or second sensor data (20, 28).

7. The method (10) for processing according to any of the preceding claims, characterized in that: The computation of the first and second interface data (52, 56) can be performed in parallel from the corresponding first and second sensor data (20, 28). The computation of the first and second interface data (52, 56) can be performed in parallel from the corresponding first and second sensor data (20, 28).

8. An object detection network (48) for detecting an environmental object (12) from sensor data (14) of an environmental sensor (16), the object detection network being configured to implement the method (10) for processing according to any one of the preceding claims, the object detection network having at least a first adapter unit (36) and a second adapter unit (38) and an object detection model (44).

9. A method (64) for adapting an object detection network (48) from at least one third environmental sensor (72), the method having the steps of: providing (68) an object detection network (48) according to claim 8, providing (70) a third environmental sensor (72) that provides third sensor data (74) having a different data modality than the data modality of the first and second sensor data (20, 28), training (76) a third adapter unit (78) with at least the third sensor data (74) as input data and with a third interface data (82) that meets the interface data specification (51) as a target specification (84). When training (76) the third adapter unit (78), the model parameters of the object detection model (44) remain unchanged or are trained at most with a lower learning rate than when training (76) the third adapter unit (78). ​ ​ 10. The method (64) for adjustment according to claim 9, characterized in that: ​

Images