Training a machine learning model for object detection using environmental sensor measurement data, and creating an environmental sensor system

Abstract

Method for training a machine learning model to detect objects based on measurements from at least one environmental sensor; and method for producing an environmental sensor system. Provided training data comprises a data segment (D) containing multiple measurements (M) from an environmental sensor and a set of labels (L). Each measurement (M) and the label set (L) is associated with a time point (tmeas; tref). The label set (L) includes at least one label that characterizes an object. Modified data (D') are generated (S20) comprising at least one measurement (M; M') from at least one environmental sensor and a set of labels (L; L').The generation process includes: predicting at least one measurement (M') of the modified data from at least one measurement (M) of the data segment (D) based on a time difference (tmeas-label) and / or predicting the label set (L') of the modified data (D') from the label set (L) of the data segment (D) based on a time difference (tmeas-label); and training (S40) the machine learning model (110) based on at least the generated modified data (D').

Description

The invention relates to a method for training a machine learning model configured to detect objects based on measurements from at least one environmental sensor. The invention further relates to a method for manufacturing an environmental sensor system, comprising the method for training a machine learning model configured to detect objects based on measurements from at least one environmental sensor. background Advanced Driver Assistance Systems (ADAS) and autonomous driving (AD) technologies require precise and reliable perception of the vehicle's surroundings. These systems utilize various sensors for this purpose, including cameras and point-based sensors such as lidar (light detection and ranging) and radar (radio detection and ranging). These sensors provide measurements, for example, in the form of point clouds, which enable a detailed representation of the environment. A lidar sensor, for instance, represents each detected point by Cartesian coordinates (x, y, z) and intensity values ​​of the reflected signal (reflection intensity). A radar sensor, for example, provides polar coordinates such as distance and azimuth angle, supplemented by features such as signal strength, radar cross-section (RCS), and elevation angle.From these point clouds, environmental perception algorithms determine relevant object characteristics such as position, orientation, extent / size, and class (e.g., automobiles, trucks, or pedestrians). This information is essential for safe navigation and decision-making in ADAS and AD systems. Traditional approaches to environmental perception combine tracking algorithms, such as Kalman filters, with downstream object type classification. These tracking algorithms include models that describe the representation of objects in sensor data, such as reflection models for radar data or L-shaped vehicle models for lidar data. Deep learning enables a different approach. Models in the form of artificial deep neural networks, and especially object detection networks, can directly detect objects and output them as oriented bounding boxes (OBBs), also known as bounding rectangles. These OBBs contain, for example, estimates for an object's probability of existence, its position, orientation, extent / size, and class. The detected OBBs are then subjected to a time-tracking algorithm. Data augmentation is used to improve the performance of object detection models. Well-known augmentation techniques for point cloud data include random translation, random flipping, translation, scaling, and rotation. These techniques can effectively improve detection performance. Disclosure of the invention Applying machine learning methods to radar data for object detection presents a particular challenge. This is mainly due to the comparatively lower point density and other physical properties of radar measurements, for example, compared to lidar measurements. When increasing data volume, capturing and emulating the actual measurement noise of a sensor presents a particular challenge. Due to the significant effort required to create labels, it has been suggested that when compiling a dataset for training, labels should not be considered for all measurements, but rather assigned only at selected time points. For example, a dataset could contain labels at a rate of 2 Hz (corresponding to two sets of labels within a one-second period), while the reference sensors used for labeling operate at a rate of 10 Hz or output measurement data. A dataset for training and validating a neural network could contain pairs of a single set of labels and a corresponding single measurement. Since the sensors used for labeling and the sensors used for training are not always synchronized (e.g.,(A radar sensor with a cycle time of 66 ms and a lidar sensor with a cycle time of 100 ms can be used for labeling.) Multiple sensor cycles can be recorded before and after the frame used for labeling. The sensor measurements used during training are compared with the timestamp of the label set. The measurement with the smallest time difference is selected. The remaining measurements are discarded. The object of the invention is to provide an improved method for training a machine learning model to detect objects using environmental sensor measurement data. A further object of the invention is to provide a novel method for manufacturing an environmental sensor system in which a machine learning model is trained to detect objects according to a novel method. One or more of the problems are solved according to the invention by a method specified in claim 1 for training a machine learning model configured to detect objects based on measurements from at least one environmental sensor, and by a method specified in a dependent claim for manufacturing an environmental sensor system. Advantageous embodiments and further developments are specified below and in the dependent claims. According to one aspect of the invention, a method for training a machine learning model configured to detect objects based on measurements from at least one environmental sensor comprises: providing training data comprising at least one data segment comprising multiple measurements from at least one environmental sensor and a set of labels, wherein each of the multiple measurements and the set of labels is assigned a time point, and wherein the set of labels comprises at least one label that characterizes an object;Generating modified data comprising at least one measurement from at least one environmental sensor and a set of labels, based on the (provided) data segment, wherein generating the modified data comprises: predicting the at least one measurement of the modified data from at least one measurement of the (provided) data segment based on a time difference and / or predicting the set of labels of the modified data from the label set of the (provided) data segment based on a time difference; and training a machine learning model configured to detect objects based on measurement data from at least one environmental sensor, wherein the training is based on at least the generated modified data. The core idea of ​​the invention is therefore to improve the training of an object detection model, which may, for example, be configured to process point cloud-like data, by increasing the training data. This is achieved by generating further, alternative, or additional label assignments to a measurement from existing assignments of labels to environmental sensor measurements through data augmentation. This allows more measurements to be used for training while simultaneously avoiding the significant additional effort required to assign extra labels. The generation of the modified data can include prediction, for example, in the form of an adjustment (such as a temporal and / or spatial shift) of a label and / or at least one measurement. This prediction can involve a conversion taking into account associated ego (vehicle) motion data.Existing environmental sensor measurements can be recalculated to a different time point, and / or existing labels can be recalculated to a different time point. For example, modified data can be generated and used from the provided training data in a given training epoch. This creates a new data augmentation method that uses additional measurements taken close to the timestamp of the label(s), with a time difference. For example, in one step of generating a mini-batch of the training process, the measurements can be processed as follows: For a single set of labels, the following can be done: 1. Selecting a measurement from the associated measurements, e.g., random selection, optionally based on a predefined distribution (probability distribution); 2. Matching the measurement and the timestamp of the label set using prediction based on ego movement information; and 3. Adding / executing further augmentation (data augmentation) and using it as input for training along with the given set of labels. Object detection is not limited to detecting objects based on a single measurement; it is also common to aggregate point clouds measured within a given time interval. The new method can be applied accordingly. Instead of selecting different measurement times, different time intervals are chosen. Learning models, such as deep learning models, require large amounts of data to generalize effectively to unseen examples (samples). Recording and labeling data is time-consuming and expensive. Data amplification is beneficial because it increases the variability of examples within a dataset. The disclosed method can improve object detection performance and the performance of other downstream units. This is presumably because the added data is (modified) real-world data, eliminating the need for time-consuming and costly additional labels. In particular, this can improve the learning of sensor noise. According to a further aspect of the invention, a method for manufacturing an environmental sensor system comprises: providing at least one environmental sensor; generating a machine learning model for detecting objects based on measurements from the at least one environmental sensor, wherein generating the machine learning model comprises: providing a machine learning model, in particular a pre-trained or untrained machine learning model, and training the provided machine learning model according to the method described herein; and providing an evaluation unit connected to the environmental sensor, which comprises the generated machine learning model. The provided machine learning model can be a pre-trained or untrained machine learning model. One or more of the aforementioned aspects may include one or more of the features described below or features of one or more of the embodiments described below. The data section comprises multiple measurements from at least one environmental sensor and a set of labels. At least one of the multiple measurements can be assigned to the set of labels, for example, by the temporal proximity of their times, by the label set and the measurement occurring at the same time, or by the measurement time being located relative to the label set time. This measurement can be selected. For example, among the measurements with times that are at least one time offset less than or equal to the label set time, the measurement with the most recent (i.e., largest) time can be assigned to the label set. The time offset can, in particular, be zero. The data section can correspond to a time interval (data section time interval), and times assigned to the multiple measurements and the label set of the data section can be times from that time interval. When generating modified data comprising at least one measurement from at least one environmental sensor and a set of labels, the set of labels can be assigned to the at least one measurement. Each measurement and the set of labels can be assigned a time point in time. The modified data can correspond to a time interval (prediction time interval) or a time point in time (prediction time point), and the time points assigned to the at least one measurement and the set of labels can be time points from the prediction time interval or can be the same as the prediction time point. This involves predicting at least one measurement and / or a set of labels from the provided training data, each based on a time difference. This generates a modified measurement (predicted measurement) and / or a modified set of labels (predicted label set), allowing new training data to be used, but based on the provided training data. Predicting a measurement involves adjusting each measurement to (or by) a time difference (a prediction time interval), i.e., it is adjusted to a point in time that results from its own measurement time shifted by the time difference. Preferably, the prediction is performed in a direction of increasing time, i.e., in the direction of the flow of time. Each of several measurements can be adjusted by the same time difference. In embodiments, the generated modified data can comprise a pair consisting of a measurement from at least one environmental sensor and a set of labels. The measurement can be a predicted measurement. For example, generating the modified data can involve selecting (e.g., randomly selecting) one of the measurements from the data segment and determining the time difference for prediction based on the selected measurement. For instance, the time difference can be determined based on the time of the selected measurement and a time point in the label set of the modified data, e.g., as the difference between the time of the selected measurement and the time point in the label set of the modified data. In embodiments, the generated modified data can comprise several predicted measurements of at least one environmental sensor, wherein generating the modified data comprises: predicting the multiple measurements of the modified data section from multiple measurements of the data section based on a time difference. Thus, multiple predicted measurements are predicted based on the same time difference. The time difference for predicting the measurement(s) and the time difference for predicting the label set can be the same or different. In embodiments, in the case of predicting at least one measurement, the at least one respective measurement of the modified data is assigned to a respective time point in time that is earlier than or equal to the time of the label set of the modified data. Thus, the label set of the modified data refers exclusively to predicted measurements whose timestamps are less than or equal to the time of the label set of the modified data. This takes into account the fact that, in the later application of the trained machine learning model, for example in the operation of an environmental sensor system, the model can only access measurements that are already available at the time of object detection. In embodiments where the label set is predicted, at least one measurement of the modified data is assigned to a specific time point that is earlier than or equal to the time point of the label set of the modified data. Thus, the predicted label set of the modified data refers exclusively to measurements whose times are earlier than or equal to the time point of the predicted label set of the modified data. In embodiments (particularly in the case of predicting at least one measurement and / or in the case of predicting the label set), the at least one measurement of the modified data is assigned to a respective time point that is earlier than or equal to the time point of the label set of the modified data. Thus, the label set of the modified data refers exclusively to measurements whose times are earlier than or equal to the time point of the label set of the modified data. In particular, if at least one measurement is predicted and the label set is predicted, the predicted label set of the modified data can refer exclusively to predicted measurements whose time points are less than or equal to the time point of the predicted label set of the modified data. In embodiments, in the case of predicting at least one measurement, the at least one measurement of the data segment used to predict the at least one measurement of the modified data is assigned to a respective time point that is earlier than or equal to the time point of the label set of the modified data. In other words, a time point of a measurement used for prediction is earlier than or equal to a time point of the label set of the modified data. Thus, prediction relies exclusively on measurements whose time points are less than or equal to the time point of the label set of the modified data. In other words, prediction relies solely on measurements that could have already been measured at the time point of the label set of the modified data (according to their assigned time points). However, it is also conceivable to use future measurements for prediction. That is, at least one measurement of the data segment used for prediction (back-projection) can be assigned to a time point that is later than the time point of the label set of the modified data. The time of the label set of the modified data can be smaller (earlier) than the time of the label set of the data segment, equal to the time of the label set of the data segment, or larger (later) than the time of the label set of the data segment. In the case of predicting a label set, the time of the predicted label set can be earlier (earlier) or later (greater) than the time of the label set of the data segment. In embodiments, the time difference when predicting the at least one measurement, or the time difference when predicting the label quantity, or the respective time difference is less than or equal to 500 ms, preferably less than or equal to 400 ms, more preferably less than or equal to 300 ms, particularly preferably less than or equal to 250 ms. This leads to particularly realistic predictions of measurements and / or label sets, since the development of a traffic event can be predicted with a high degree of probability and near-perfect accuracy within such a timeframe. For stationary targets, prediction, especially adjustment to a forecast time based on the time difference, can be performed without error. Experiments have shown that with a sufficiently small time difference to the nearest measurement time (e.g., 150 ms to 250 ms), the error of a predicted measurement (obtained through adjustment) is still acceptable, and a machine learning model for object detection can benefit from the increased training data, even for moving targets. The training data can comprise a large number of such data segments, for each of which the steps of generating modified data and training are performed. In embodiments, the training data comprises several data sections, each comprising several measurements from at least one environmental sensor and a set of labels, wherein each of the several measurements and the set of labels is assigned a time point, and wherein the set of labels comprises at least one label that characterizes an object, wherein the step of generating modified data is performed for each of the several data sections, wherein the generation of modified data is based on the respective data section, and wherein the training of the machine learning model is based on at least the respective generated modified data. For example, modified data from several data sections can be combined into a batch, and this batch can then be used to train the model. Different or, for example, randomly selected time differences can be used for each data segment for the respective prediction. A random selection of a time difference can be achieved, in particular, by randomly selecting one measurement from the multiple measurements of the data segment. Specifically, the time difference can then be determined by the difference between the time of the measurement and the time of the label set. The step of generating modified data for the multiple data sections can be carried out in each case of predicting at least one measurement with a different respective time difference (or a randomly chosen respective time difference) for predicting the at least one measurement, and / or the step of generating modified data for the multiple data sections can be carried out in each case of predicting the label set with a different respective time difference (or a randomly chosen respective time difference) for predicting the label set. In embodiments, the respective time difference is randomly selected from values ​​within an interval. The time interval preferably has an absolute value of less than or equal to 500 ms, more preferably less than or equal to 400 ms, further preferably less than or equal to 300 ms, and particularly preferably less than or equal to 250 ms. After performing training based on the multiple data segments (corresponding, for example, to a training epoch), the generation of the modified data and the training can be repeated (corresponding, for example, to another training epoch), whereby a different or, for example, randomly selected time difference can be used for each data segment. Thus, the steps of generating modified data and training can be repeated, whereby in each iteration of the steps of generating modified data and training for the same data segment, different respective time differences (or randomly chosen respective time differences) are used for predicting the at least one measurement in the case of predicting the at least one measurement and / or different respective time differences (or randomly chosen respective time differences) are used for predicting the label set in the case of predicting the label set. During each execution of the training step, training for individual data segments (or for modified data based on individual data segments) can be performed in different sequences of the data segments (assigned to the modified data). Thus, in different training epochs, the modified data based on the data segments can be trained in different sequences. The procedure can include: splitting the training data into the respective data sections. For training purposes, multiple measurements of predicted data can be aggregated. In some embodiments, the generated modified data comprises multiple (e.g., predicted) measurements from at least one environmental sensor. The method further comprises aggregating the multiple (e.g., predicted) measurements of the generated modified data into a single aggregated measurement from at least one environmental sensor, and training is performed based on at least the generated modified data, comprising the aggregated measurement. This allows the model to be provided with modified data containing a higher number of measurements. During aggregation, the respective times of the predicted measurements can be preserved. Aggregating multiple measurements can be done before predicting or generating the modified data. In embodiments, the data section comprises several measurements of at least one environmental sensor, the method further comprising: aggregating the several measurements of the data section into an aggregated measurement of at least one environmental sensor, and wherein the generation of modified data is based on the aggregated measurement and the label set of the data section, wherein the modified data comprise the aggregated measurement of at least one environmental sensor and a label set, wherein the generation of the modified data comprises: predicting the at least one measurement of the modified data from the aggregated measurement of the data section based on a time difference and / or predicting the label set of the modified data from the label set of the data section based on a time difference. In embodiments, the machine learning model is an artificial neural network, in particular an object detection network for detecting objects based on measurements from at least one environmental sensor. The machine learning model can be configured to receive environmental sensor measurements as input, process them, and calculate an output. In embodiments, the machine learning model is configured to output at least one boundary box, where each boundary box represents a detected object. The output can comprise a set of boundary boxes. The set can include at least one boundary box. In the case of each detected extended object, the output can specify the respective boundary box of the detected object. In the case of multiple detected extended objects, the output can comprise multiple boundary boxes. The measurements may include, in particular, radar measurements and / or lidar measurements. The environmental sensor measurements can include, in particular, point cloud-like measurements, for example, points in the form of reflections, detections, or locations. In the case of multiple detected points, the point cloud-like measurements can be structured point by point; for example, they can comprise a list of points or a point cloud. In embodiments, the measurements of at least one environmental sensor, in the case of several points detected by one environmental sensor, comprise point clouds or point lists whose points correspond to the respective detected points. In embodiments, the machine learning model is configured to detect objects based on measurements from at least one radar sensor, wherein the at least one data section comprises several measurements from at least one radar sensor. The respective set of labels for a data segment can correspond to an object situation (e.g., traffic situation) embodied by at least one of the measurements of the data segment. A label can, in particular, identify characteristics of an object, including its probability of existence, size or extent, an angle such as an elevation angle, position or coordinates, orientation, and / or class. A label can also, in particular, include a description of an object in the form of an oriented bounding box (OBB). In the case of multiple objects (i.e., when multiple objects are identified by the label set), the label set comprises multiple labels, with each label identifying characteristics of the respective object. A label set is associated with a label set point in time (a timestamp). The training data can describe at least one coherent scene (of a traffic event), in particular a scene of a traffic event in the vicinity of an ego-vehicle. The scene can comprise several data segments. The measurements of at least one environmental sensor can originate from at least one environmental sensor of an Ego vehicle or be assigned to at least one environmental sensor of an Ego vehicle. The training data section can also include ego motion data, specifically ego vehicle motion data (motion data of the ego vehicle). Each piece of ego motion data can be associated with a corresponding ego time point. An ego time point can correspond to a measurement time point (the point at which a measurement is taken) or a label (set) time point (the point at which a set of labels is taken). The ego motion data can include: velocity, acceleration, and / or yaw rate of the ego vehicle. The modified data may include ego movement data, in particular ego movement data (or the ego movement data) of the data section. The procedure can further include: modifying the measurements and the label set of a data segment or the modified data generated based on a data segment, wherein the modification includes at least one of the following: truncating the at least one measurement and the label set, (spatially) translating the at least one measurement and the label set, (spatially) inverting or mirroring the at least one measurement and the label set, e.g., with respect to a coordinate axis or a coordinate (e.g., a Cartesian coordinate), scaling the at least one measurement and the label set, and rotating the at least one measurement and the label set. This can further improve the detection performance of the trained model. The modification can be performed before generating the modified data, in which case the modified data is generated based on the modified data.The modification can be performed after the modified data has been generated, at which point the modified data is changed. The original training data can be retained and reused for subsequent training periods. The following are exemplary embodiments explained in more detail with reference to the drawings. They show: Fig. 1 a schematic representation of a training method for a machine learning model; Fig. 2 a schematic representation of a method for manufacturing an environmental sensor system; Fig. 3 a schematic representation of the environmental sensor system with an evaluation unit comprising the machine learning model; Fig. 4 a schematic representation of a data section of training data as well as modified data with a predicted set of labels or a predicted measurement; Fig. 5 a schematic representation of a data section of training data as well as modified data with a predicted set of labels; Fig. 6 a schematic representation of a data section of training data as well as modified data with a predicted measurement; Fig.Figure 7 shows a schematic representation of a data section of training data as well as modified data with a predicted set of labels and a predicted measurement; and Figure 8 shows a schematic representation of a data section of training data as well as modified data with multiple predicted measurements. Fig. 1 schematically shows an example of a training procedure for a machine learning model for object detection based on measurements from an environmental sensor. Fig. 2 shows a schematic representation of a method for creating an environmental sensor system, and Fig. 3 shows a schematic representation of the environmental sensor system with an environmental sensor 200 in the form of a radar sensor and an evaluation unit 100, which includes the machine learning model 110. To create the environmental sensor system, the environmental sensor 200 is provided (step S210), the machine learning model 110 is provided in a pre-trained or untrained state and trained according to the training procedure (step S220), and the evaluation unit 100 with the machine learning model 110 implemented therein is provided and connected to the environmental sensor 200 (step S230). The training procedure can be structured as follows: Training data is provided (step S10) which (as shown, for example, in Fig. 4) contains measurements M from a radar sensor, which is, for example, identical in construction to the environment sensor 200. The training data also contains sets of labels L, each of which can comprise a number of labels. The training data can also be shuffled in step S10, i.e., the order of data sections of the training data can be changed, for example, randomly. The training data is then divided into mini-batches of equal size in step S12. Each mini-batch contains, for example, 16 to 256 data sections D (samples). Each data section D contains multiple measurements M. The measurements M have respective time points tmeas i. Each data section D also contains a label set L, which has a time point tref i. Optionally, the respective data sections are subjected to data multiplication (step S14), for example random cropping, flipping or rotating. Modified data D' is then generated from the respective data segment (step S20). This is explained in more detail below. Optionally, and as an alternative to step S14, the respective modified data are subjected to data multiplication (step S30), for example random cropping, flipping or rotating. Then, the training (step S40) of the machine learning model 110 takes place. For this purpose, the corresponding measurements for the respective modified data D' are entered as input 112 into the model 110, and error gradients are determined based on the corresponding label sets from the outputs 114. The gradients are averaged over the mini-batch, and the model parameters are updated using an optimization algorithm such as SGD (stochastic gradient descent). Steps S20, possibly S14 or S30, and S40 are repeated for all mini-batches of training data, and the model parameters are updated after each mini-batch. Once all mini-batches have been processed, a training epoch is complete, and the procedure is repeated starting from step S10 for the next epoch until the model converges. Fig. 4 illustrates examples of generating (S20) the modified data D'. tref denotes the times of the labels and can correspond to the timestamps of a reference sensor among the sensors used for labeling; tmeas denotes the times of the measurements M. A data section D, also called a label frame, is defined according to a time interval tmeas - Δt ≤ tLabel ≤ tmeas + Δt around the timestamp tLabel of the label set L. Thus, for each label set L, there are several associated measurements M. In a first variant, the following steps are performed for a single label set L or a single data segment D: 1. A measurement is randomly selected from the interval tmeas - Δt ≤ tLabel ≤ tmeas + Δt (step S22). The selection can optionally be based on a probability distribution, for example, a uniform distribution or a categorical distribution with a maximum at the label time tLabel. The parameter Δt, the type of distribution, and its parameters are then hyperparameters of the training and can be set together with the other parameters. 2. The measurement M and the timestamp of the label set L are aligned (step S24) using ego movement data E to account for changes in the measurement or the labels due to movement in the environment. This can be done: a) by predicting (PL) the label set L (to account for the time difference Δtmeas-label) to the timestamp of the measurement M (e.g.,a) using the time difference, ego motion state, and speed of a respective label; this is illustrated separately in Fig. 5; or b) by predicting (PM) the measurement M (around the time difference Δtmeas-label) onto the timestamp of the label set L (e.g., using the time difference and the ego motion state); this is illustrated separately in Fig. 6. 3. Further data augmentation procedures can optionally be performed (step S30). The environmental sensor system can access the Ego motion data E via an interface 120, for example by accessing data from an ESP system. Fig. 7 illustrates another example of generating (S20) the modified data D'. In a second variant, the following steps are performed for a single label set L or a single data segment D: 1. A data multiplication timestamp (time point) taug is selected from the interval tmeas - Δt ≤ tLabel ≤ tmeas + Δt (step S22), based on a probability distribution (e.g., a uniform distribution or a normal distribution with a mean = tLabel and e.g., sigma = 150 ms). The parameter Δt, the type of distribution, and its parameters are then hyperparameters of the training and can be set together with the other parameters. 2. The measurement M and the timestamp of the label set L are aligned (step S24) using ego movement data E to account for changes in the measurement or the labels due to movement in the environment. Unlike the first variant, both the measurement M and the label set L are predicted. This can be done by: a) Selecting, e.g.,a) the measurement M that is closest in time to the prediction time (data augmentation timestamp) taug, and b) predict the measurement M at the prediction time taug (e.g., using the relevant time difference and the ego movement state); and b) predict the measurement M at the prediction time taug (e.g., using the relevant time difference and the ego movement state). 3. Further data augmentations (augmentation procedures) can optionally be performed (step S30). The first variant allows a fixed number of possible data-enhanced states (modified data) per label set. The second variant has the advantage of further increased variability in the training data, since the selected value of the prediction time is tautly continuous and the number of possible data-enhanced states is not limited. Predicting measurements is well feasible for point cloud data such as radar data (reflections or locations) and lidar data, since the positions of the individual measurement points can be shifted according to the ego movement of the ego vehicle. Fig. 8 illustrates an embodiment in which multiple measurements M are predicted. The modified data D' comprise several predicted measurements and a label set L. The training procedure can include aggregating the multiple measurements M (step S32). Aggregation can also be included in step S20 of generating the modified data. In the case of aggregation, mixing does not occur in step S10, but rather after augmentation, for example, in step S32 after aggregation. Experiments on a training dataset (Bosch LH5-2019) have shown that the training method disclosed herein is advantageous in addition to existing data multiplication methods. In these experiments, a sampling (selection) time Δt of 250 ms and a uniform distribution for sample selection were used. Ten trainings were evaluated for each method. The results given in Table (1) show the mean AP in percent and the corresponding standard deviations (SD): (Table (1)) (Table (1)) Conventional data amplification 24.66 (0.33) 17.82 (0.57) 11.65 (1.04) 5.43 (0.47) Additional prediction with Δt selection: 25.43 (0.23) 18.58 (0.66) 13.11 (1.54) 5.81 (0.38)

Claims

Method for training a machine learning model configured to detect objects based on measurements from at least one environmental sensor, the method comprising: providing (S10) training data comprising at least one data segment (D) containing multiple measurements (M) of at least one environmental sensor and a set of labels (L), wherein each of the multiple measurements (M) and the set of labels (L) is assigned a time point (tmeas; tref), and wherein the set of labels (L) includes at least one label that characterizes an object; generating (S20) modified data (D') comprising at least one measurement (M; M') of at least one environmental sensor and a set of labels (L;L') comprise, based on the data section (D), wherein generating the modified data (D') comprises: predicting at least one measurement (M') of the modified data from at least one measurement (M) of the data section (D) based on a time difference (tmeas-label) and / or predicting the label set (L') of the modified data (D') from the label set (L) of the data section (D) based on a time difference (tmeas-label); and training (S40) a machine learning model (110) configured to detect objects based on measurement data from at least one environmental sensor (200), wherein the training is based on at least the generated modified data (D'). Method according to claim 1, wherein the generated modified data (D') comprises a pair of a measurement (M; M') of at least one environmental sensor and a set of labels (L; L'). Method according to claim 1 or 2, wherein the generated modified data (D') comprise several predicted measurements (M') of at least one environmental sensor, wherein generating the modified data (D') comprises: predicting the several measurements (M') of the modified data section (D') from several measurements (M) of the data section (D) based on a time difference. Method according to one of the preceding claims, wherein the at least one respective measurement (M; M') of the modified data (D') is assigned to a respective time point (tmeas) which is earlier in time than or equal to the time point of the label set (L; L') of the modified data (D'). Method according to one of the preceding claims, wherein in the case of predicting at least one measurement (M') the at least one respective measurement (M) of the data section (D') used to predict the at least one measurement (M') of the modified data (D') is assigned to a respective time point (tmeas) which is earlier in time than or equal to the time point of the label set (L; L') of the modified data (D'). Method according to one of the preceding claims, wherein the respective time difference (tmeas-label) is less than or equal to 500 ms. A method according to one of the preceding claims, wherein the training data comprises several data sections (D), each comprising several measurements (M) of at least one environmental sensor and a set of labels (L), wherein each of the several measurements (M) and the set of labels (L) is assigned a time point (tmeas; tref), and wherein the set of labels (L) comprises at least one label that characterizes an object, wherein the step of generating (S20) modified data is performed for each of the several data sections (D), wherein the generation of modified data (D') is based on the respective data section, and wherein the training (S40) of the machine learning model (110) is based at least on the respective generated modified data (D').wherein the step of generating (S20) modified data (D') for the multiple data sections (D) is performed in each case of predicting the at least one measurement (M') with a different respective time difference or with a randomly selected respective time difference for predicting the at least one measurement (M') and / or the step of generating (S20) modified data (M') for the multiple data sections (D) is performed in each case of predicting the label set (L') with a different respective time difference or with a randomly selected respective time difference for predicting the label set (L'). A method according to one of the preceding claims, wherein the steps of generating (S20) modified data and training (S40) are repeated, wherein in each iteration of the steps of generating (S20) modified data and training (S40) for the same data segment (D) different respective time differences or randomly selected respective time differences are used for predicting the at least one measurement (M') in the case of predicting the at least one measurement (M') and / or different respective time differences or randomly selected respective time differences are used for predicting the label set (L) in the case of predicting the label set (L). Method according to one of the preceding claims, wherein the generated modified data (D') comprise several measurements (M') of at least one environmental sensor, wherein the method further comprises: aggregating the several measurements (M') of the generated modified data (D') into an aggregated measurement of at least one environmental sensor, and wherein the training (S40) is performed based at least on the generated modified data (D'), comprising the aggregated measurement. A method according to any of the preceding claims, wherein the data section (D) comprises multiple measurements (M) of at least one environmental sensor, wherein the method further comprises: aggregating (S32) the multiple measurements (M) of the data section (D) into an aggregated measurement of at least one environmental sensor, and wherein the generation (S20) of modified data is based on the aggregated measurement and the label set (L) of the data section (D), wherein the modified data (D') comprises the aggregated measurement of at least one environmental sensor and a label set (L), wherein the generation of the modified data (D') comprises: predicting the at least one measurement (M') of the modified data (D') from the aggregated measurement of the data section (D) based on a time difference and / or predicting the label set (L') of the modified data (D') from the label set (L) of the data section (D) based on a time difference. Method according to one of the preceding claims, wherein the machine learning model (110) is an artificial neural network, in particular an object detection network for detecting objects based on measurements from at least one environmental sensor. Method according to one of the preceding claims, wherein the machine learning model (110) is configured to output a set of bounding boxes, the bounding boxes each representing a detected object. Method according to one of the preceding claims, wherein the machine learning model (110) is configured to detect objects based on measurements from at least one radar sensor, wherein the at least one data section (D) comprises several measurements (M) from at least one radar sensor. Method according to one of the preceding claims, wherein the measurements (M) of at least one environmental sensor in the case of several points detected by one environmental sensor comprise point clouds or point lists, the points of which correspond to the respective detected points. Method for manufacturing an environmental sensor system, the method comprising: providing (S210) at least one environmental sensor; generating (S220) a machine learning model (110) for detecting objects based on measurements (M) of the at least one environmental sensor, wherein generating the machine learning model (110) comprises: providing a machine learning model (110), in particular a pre-trained or untrained machine learning model (110), and training the provided machine learning model (110) according to the method of any one of claims 1 to 14; providing an evaluation unit (100) connected to the environmental sensor (200), which comprises the generated machine learning model (110).

Images