Methods for processing sensor data, object detection network and methods for adapting an object detection network

Abstract

The invention relates to a method for processing (10) sensor data (14) from several environmental sensors (16) specifying at least one environmental object (12), providing (18) at least first sensor data (20) from a first environmental sensor (22), providing (26) at least second sensor data (28) from a second environmental sensor (30), providing (34) at least one first adapter unit (36) and a second adapter unit (38), providing (42) at least one object detection model (44) with at least one trained artificial neural network (46), inputting (50) the first sensor data (20) to the first adapter unit (36) and calculating first interface data (52) fulfilling a predetermined interface data specification (51) as output of the first adapter unit (36).Input (54) of the second sensor data (28) to the second adapter unit (38) and calculation of the second interface data (56) fulfilling the interface data specification (51) as output of the second adapter unit (38); input (58) of the first and second interface data (52, 56) as input data (60) to the object detection model (44) and calculation of at least one object parameter (62) of the at least one environment object (12) as output. Furthermore, the invention relates to an object detection network (48) and a method for adapting (64) an object detection network (48).

Description

[0001] The invention relates to a method for processing sensor data according to claim 1. Furthermore, the invention relates to an object detection network and a method for adapting an object detection network. State of the art

[0002] The advanced driver assistance systems (ADAS) and autonomous driving functions now used in vehicles require precise detection and representation of the vehicle's surroundings. For this purpose, environmental sensors such as cameras, LiDAR sensors, and radar sensors are used. Radar sensors play a particularly important role, as they not only operate reliably in poor visibility conditions such as fog or darkness, but also provide detailed information about the vehicle's environment.

[0003] A common application of radar sensors is object detection. For this purpose, radar sensors generate point clouds consisting of individual reflections. Each reflection is described by polar coordinates such as distance and azimuth angle, as well as other characteristics like signal strength, radar cross-section (RCS), and elevation angle. Using object detection models for capturing environmental objects in the vehicle's surroundings, which are now largely based on deep learning, relevant objects such as cars, trucks, or pedestrians are identified from these point clouds. The object detection models determine, for example, the position, orientation (pose), class, and potentially other properties of the environmental objects and can represent the environmental objects in the form of oriented bounding boxes (OBBs).

[0004] The output of the object detection models depends on the input data from the environmental sensors, which in turn can differ due to the specific characteristics of each sensor. Therefore, the object detection models are adapted and trained to each individual environmental sensor. If an environmental sensor is replaced with a new and different one, the object detection model must be retrained with the new sensor data. This means that the training data from the new environmental sensor must be acquired and processed before the object detection model's training process is repeated. Disclosure of the invention

[0005] According to the present invention, a method for processing sensor data with the features of claim 1 is proposed. This allows the object detection model to perform the calculation independently of the type of environmental sensors providing the input data. The calculation of the object detection model can be decoupled from the type of environmental sensors and from the specific data format of the sensor data. Adapting the object detection network to a new sensor data format or a new environmental sensor can be done more quickly and easily.

[0006] Environmental sensors can be radar sensors, LiDAR sensors, cameras, ultrasonic sensors, and / or accelerometers. These sensors can be mounted on a vehicle, device, or mobile robot.

[0007] The surrounding object can be a living being, a building, a plant, an object, a device, or a vehicle.

[0008] The sensor data from an environmental sensor can be in the form of a point cloud, frequency spectrum, or time signal.

[0009] The object detection model can be configured for object recognition, object classification, semantic segmentation, and / or free space detection. The object detection model can be configured for a single task, such as object recognition, or for multiple tasks, such as object recognition and object classification.

[0010] The object detection model can have multiple layers in the neural network, including at least one input layer, several intermediate layers, and at least one output layer.

[0011] The object detection model can be a Convolutional Neural Network (CNN). The CNN uses convolutional layers, particularly two-dimensional convolutional layers, with convolutions for filtering and extracting features from the interface data as input data, especially to recognize structures and patterns in the input data. The CNN comprises at least one convolutional layer, one pooling layer, and / or one dense output layer. The pooling layer can be a mean pooling layer or a max pooling layer. The pooling layer can apply global pooling. The convolutional layer can be used for feature extraction, the pooling layer for reducing the spatial size of the features, and the dense output layer for classification. For particularly high memory requirements, the CNN can function with only one convolutional layer and one pooling layer, thus eliminating the need to store the convolutional features.

[0012] The interface data can be used as input data in a sensor fusion model. The sensor fusion model can be part of the object acquisition model or placed before it.

[0013] The interface data specification can define a data structure, data format, data type, and / or data quality for the interface data. Adherence to the interface data specification ensures consistent interface data regardless of the structure of the input data.

[0014] The interface data can be mapped into an embedded space by the respective adapter unit. The interface data specification can define the dimensions of the embedded space, pattern templates, structural specifications, and / or limit values ​​within the embedded space. The interface data specification can establish a standard for the interface data, ensuring that the interface data is available in a uniform data format, a uniform data type, and / or a uniform data structure.

[0015] The adapter units and the object detection model can form an object detection network that, depending on the sensor data from the environmental sensors as input data, calculates at least one object parameter of the at least one environmental object.

[0016] Training refers to the iterative process by which a neural network learns from training data to improve its prediction accuracy. Initially, the neural network is given the input data along with the corresponding correct outputs (annotations) as a target. The neural network processes this data through its layers and outputs a prediction. This prediction is then compared to the actual annotations, and the error or loss is calculated. This loss indicates how far the neural network's prediction deviates from the actual response. To minimize this error, the neural network's parameters, particularly the weights, are adjusted. This is done, for example, using gradient descent, where the gradient of the loss is calculated with respect to the weights. The weights are then changed in the direction that reduces the loss.This process is repeated over many iterations, with the neural network continuously adjusting its weights to reduce errors and make more accurate predictions. The goal of this training is to optimize the parameters so that the neural network performs accurate calculations even with new, unseen input data.

[0017] In a preferred embodiment of the invention, it is advantageous if the first and / or second adapter unit comprises a trained artificial neural network. The training of the object detection model can be performed simultaneously with the training of the first and second adapter units. Alternatively, the object detection model can be trained independently of the training of the first and second adapter units.

[0018] The model parameters of the object detection model can initially be trained together with the model parameters of the adapter units. Multiple sensor data from various environmental sensors can be used as training data. The model parameters of the adapter units can be trained as follows: Firstly, only the model parameters of the adapter unit whose associated environmental sensor provides the training data for the current iteration can be trained. The total training data can be a random sequence of sensor data from the environmental sensors. Secondly, initially only multiple sensor data from a single environmental sensor can form the training data. After multiple iterations and training of the model parameters of the associated adapter unit, the next adapter unit can be trained with its corresponding sensor data.

[0019] The learning rate during the training of the adapter units can be the same or different from each other. The learning rate during the training of the object acquisition model can be the same or different from the learning rate during the training of at least one of the adapter units or during the training of all adapter units.

[0020] If less training data is available for one environmental sensor and thus for its associated adapter unit than for another environmental sensor and thus for its associated adapter unit, the less training data can be reused during the learning process of the object detection network. This ensures that each adapter unit receives the same amount of training data during the learning process. Alternatively, different weighting can be applied when calculating the loss function if the amount of training data used during learning differs between the adapter units.

[0021] Furthermore, if the adapter units have a different number of model parameters to be trained, repeated application of training data to the adapter unit with the higher number of model parameters or different weighting in the calculation of the loss function can be used.

[0022] If a trained object acquisition model already exists, the first layers of this model can be used as the basis for a newly trained adapter unit. This allows the weights of these layers to be initialized, which speeds up the further training of the adapter unit. This approach can be applied to one or more adapter units.

[0023] The adapter units can also be pre-trained using unsupervised learning. The result of this unsupervised learning can serve as a starting point (initialization of the weights) for the subsequent supervised learning. This not only accelerates supervised learning but also reduces the need for labeled data. An example of unsupervised learning is to train the adapter unit with an auxiliary task. Such an auxiliary task could, for example, involve learning how the sensor data, assumed to be in the form of points, must be rotated in 90-degree increments to match a predefined point pattern as the interface data specification.

[0024] The first and / or second adapter unit may additionally or alternatively include a deterministic algorithm.

[0025] A preferred embodiment of the invention is advantageous in which the number of parameters of the learned model parameters of the first and / or second adapter unit is smaller than the number of parameters of the learned model parameters of the object detection model. This allows the object detection model to be used quickly and easily with new sensor data and / or a new environmental sensor that replaces or adds to an existing one, without having to adapt the object detection model itself or having to adapt it in a complex manner.

[0026] In an advantageous embodiment of the invention, the number of parameters of the respective adapter unit depends on the data modality of the corresponding sensor data. The data modality specifies a data structure, a data type, and / or a data format of the sensor data. The more complex the data structure, data type, and / or data format of the sensor data as input data for the respective adapter unit, the greater the number of parameters this adapter unit can have. This allows the amount of information in the sensor data to be processed reliably.

[0027] In a specific embodiment of the invention, it is advantageous if at least a third adapter unit calculates third sensor data from a third environmental sensor to form third interface data that fulfills the interface data specification and constitutes further input data for the object detection model. This allows the amount and / or quality of the input data for object detection to be increased. Furthermore, additional adapter units can calculate further sensor data from additional environmental sensors to form further interface data that fulfills the interface data specification and constitutes supplementary input data for the object detection model.

[0028] In a preferred embodiment of the invention, the third or at least one further environmental sensor is assigned a sensor modality that differs from that of the first and / or second environmental sensor, and / or the third sensor data or further sensor data is assigned a data modality that differs from that of the first and / or second sensor data. The uniform interface data specification allows the input data for the object detection model to be uniform and independent of the sensor modality and / or the data modality. The sensor modality specifies a sensor class of the environmental sensor. For example, a radar sensor is assigned a different sensor modality than a LiDAR sensor or a camera.

[0029] In a preferred embodiment of the invention, the calculation of the first and second interface data from the respective sensor data can be performed in parallel. The calculated first and second interface data can be processed sequentially, fused, or in parallel by the object detection model.

[0030] According to the present invention, an object detection network with the features according to claim 8 is further proposed.

[0031] According to the present invention, a method for adapting an object detection network with the features of claim 9 is further proposed. The adaptation method can also include adaptation to at least one further environmental sensor analogous to that of the third environmental sensor.

[0032] In an advantageous embodiment of the invention, the model parameters of the object detection model remain unchanged during the training of the third adapter unit, or are trained at a learning rate that is lower than the learning rate during the training of the third adapter unit. This reduces the effort required to adapt the object detection network to the additional third environmental sensor.

[0033] According to the present invention, a computer program is further proposed which has machine-readable instructions executable on at least one computer, the execution of which involves the described processing method or the described adaptation method.

[0034] According to the present invention, a storage unit is further proposed which is machine-readable and accessible by at least one computer and on which the computer program described above is stored.

[0035] Further advantages and advantageous embodiments of the invention will become apparent from the description of the figures and the illustrations. Character description

[0036] The invention is described in detail below with reference to the illustrations. These show, in detail: Fig. 1: A method for processing sensor data in a specific embodiment of the invention. Fig. 2: A method for adapting an object detection network in a special embodiment of the invention.

[0037] Fig. Figure 1 shows a method for processing sensor data in a specific embodiment of the invention. The method for processing sensor data 10 from at least one environmental object 12, each from several environmental sensors 16, comprises providing at least one initial sensor data 20 from a first environmental sensor 22. The first environmental sensor 22 can be a radar sensor, and the initial sensor data 20 can be assigned to a first data modality 24, for example, as a point cloud. Furthermore, at least two sensor data 28 from a second environmental sensor 30 are provided. The second environmental sensor 30 can be a camera, and the second sensor data 28 can be assigned to a second data modality 32 that differs from the first data modality 24, for example, camera images containing pixels.

[0038] Subsequently, at least one first adapter unit 36 ​​and one second adapter unit 38 are provided. The first and second adapter units 36, 38 preferably each have a trained artificial neural network 40 with several layers.

[0039] Furthermore, at least one object detection model 44 with at least one trained artificial neural network 46 with multiple layers is provided. The number of parameters of the trained model parameters of the first or second adapter unit 36, 38 is, in particular, smaller than the number of parameters of the trained model parameters of the object detection model 44. The number of parameters of the respective adapter unit 36, 38 depends, for example, on the data modality of the corresponding sensor data 14. This means that the larger the data structure and / or the data format of the sensor data 14, the greater the number of parameters of the assigned adapter unit 36, 38. The object detection model 44 together with the first and second adapter units 36, 38 forms an object detection network 48.

[0040] Furthermore, the first sensor data 20 is input 50 to the first adapter unit 36, and the first adapter unit 36 ​​calculates the first interface data 52, which fulfills a predefined interface data specification 51. The interface data specification 51 is a specification for the interface data with regard to the data structure, data type, and / or data format of the sensor data 14. The interface data specification 51 can form a standard for the interface data, ensuring that the interface data is available in a uniform data format, a uniform data type, and / or a uniform data structure. The interface data specification 51 is fixed.

[0041] Furthermore, in parallel or after the calculation of the first interface data 52, an input 54 of the second sensor data 28 to the second adapter unit 38 takes place and a calculation of the second interface data 56 fulfilling the interface data specification 51 by the second adapter unit 38.

[0042] Subsequently, the first and second interface data 52, 56 are input 58 as input data 60 to the object acquisition model 44, and at least one object parameter 62 of the at least one environment object 12 is calculated as output of the object acquisition model 44. The object parameter 62 can be an object type, and the object acquisition model 44 can be used for object classification.

[0043] Fig. Figure 2 shows a method for adapting an object detection network in a specific embodiment of the invention. The method for adapting 64 an object detection network 48, here an extension 66 of the object detection network 48 by processing third sensor data from a third environmental sensor, comprises providing 68 the object detection network 48 for detecting environmental objects depending on the first and second sensor data 20, 28 of the first and second environmental sensors 22, 30. The object detection network 48 is configured, which in Fig. 1 described procedure for processing and includes the first adapter unit 36, the second adapter unit 38 and the object acquisition model 44.

[0044] Furthermore, a third environmental sensor 72 is provided, which provides third sensor data 74. The third sensor data 74 has a data modality that differs from that of the first and second sensor data 20, 28. For example, the third environmental sensor 72 is a LiDAR sensor, and the third sensor data 74 is in the form of a point cloud, but differs in a data structure, for example, in the dimensions spanning the point cloud, from the dimensions in which the point cloud of the first sensor data 20 is spanned, and is therefore assigned to a different data modality.

[0045] Subsequently, a third adapter unit 78, comprising a multi-layer neural network 77, is trained using the third sensor data 74 as input data 80 and the third interface data 82, fulfilling the interface data specification, as annotated data as target data 84 during training 76. Once the third adapter unit 78 has been trained, the object detection network 48 is extended to include the third adapter unit 78 and the ability to process the third sensor data 74.

Claims

[1] Method for processing (10) sensor data (14) specifying at least one environmental object (12) from several environmental sensors (16), comprising the steps Providing (18) at least first sensor data (20) from a first environmental sensor (22), Providing (26) at least two sensor data (28) from a second environmental sensor (30), Providing (34) at least one first adapter unit (36) and one second adapter unit (38), Providing (42) at least one object detection model (44) with at least one trained artificial neural network (46), inputting (50) the first sensor data (20) to the first adapter unit (36) and calculating first interface data (52) fulfilling a predefined interface data specification (51) as output of the first adapter unit (36), Input (54) of the second sensor data (28) to the second adapter unit (38) and calculation of the second interface data (56) fulfilling the interface data specification (51) as output of the second adapter unit (38), Input (58) of the first and second interface data (52, 56) as input data (60) to the object acquisition model (44) and calculation of at least one object parameter (62) of the at least one environment object (12) as output. [2] Processing method (10) according to claim 1, characterized by , that the first and / or second adapter unit (36, 38) comprises a trained artificial neural network (40). [3] Processing method (10) according to claim 2, characterized by , that the number of parameters of the learned model parameters of the first and / or second adapter unit (36, 38) is less than the number of parameters of the learned model parameters of the object acquisition model (44). [4] Processing method (10) according to claim 3, characterized by , that the number of parameters of the respective adapter unit (36, 38) depends on the data modality of the corresponding sensor data (20, 28). [5] Processing method (10) according to any of the preceding claims, characterized by , that at least a third adapter unit (78) calculates third sensor data (74) from a third environmental sensor (72) to third interface data (82) that fulfills the interface data specification (51) and forms further input data for the object detection model (44). [6] Processing method (10) according to claim 5, characterized by , that the third environmental sensor (72) is assigned a sensor modality different from a sensor modality of the first and / or second environmental sensor (22, 30) and / or the third sensor data (74) is assigned a data modality different from a data modality of the first and / or second sensor data (20, 28). [7] Processing method (10) according to any of the preceding claims, characterized by , that the calculation of the first and second interface data (52, 56) starting from the respective first and second sensor data (20, 28) can be carried out in parallel. [8] Object detection network (48) for detecting environmental objects (12) depending on sensor data (14) from environmental sensors (16), which is configured to carry out the processing method (10) according to one of the preceding claims and comprising at least the first adapter unit (36) and second adapter unit (38) and the object detection model (44). [9] Method for adapting (64) an object detection network (48) to at least one third environmental sensor (72), comprising the steps of providing (68) the object detection network (48) according to claim 8, providing (70) at least one third environmental sensor (72) providing a data modality different from that of the first and second sensor data (20, 28), Teaching (76) a third adapter unit (78) at least with the third sensor data (74) as input data and the third interface data (82) fulfilling the interface data specification (51) as target specification (84). [10] Method for adapting (64) according to claim 9, characterized by, that the model parameters of the object acquisition model (44) remain unchanged during the training (76) of the third adapter unit (78) or are trained at most at a learning rate that is lower than a learning rate during the training (76) of the third adapter unit (78).

Images