Method and device for training a neural network

Abstract

Method for training a neural network (30), wherein sensor data (20) marked with a ground truth (24) and unmarked sensor data (21) of a target data domain (10) are obtained, wherein sensor data (22) of at least one starting data domain (11) marked with a ground truth (24) are obtained, and wherein a database (40) is generated which forms a basis for a piecewise replacement of sensor data (20, 21, 22) by means of quilting, wherein for this purpose sensor data patches (60) are generated from the received sensor data (20, 21, 22) and stored in the database (40), wherein the database (40) is generated such that a reconstruction data domain defined by means of quilting piecewise replaced sensor data (20-e, 22-e) has a smaller data domain distance to the target data domain (10) than the at least one start data domain (11), wherein the neural network (30) is trained using training data (25) which is composed at least partially of the sensor data (20, 22) of the target data domain (10) and the at least one starting data domain (11) marked with a ground truth (24) and piecewise replaced sensor data (20-e, 22-e), wherein the piecewise replaced sensor data (20-e, 22-e) are generated by quilting using the generated database (40) from the sensor data (20, 22) of the target data domain (10) and the at least one starting data domain (11) marked with the ground truth (24), and wherein the trained neural network (31) is provided.

Description

[0001] The invention relates to a method and a device for training a neural network. Furthermore, the invention relates to a computer program and a data carrier signal.

[0002] Machine learning, for example based on neural networks, has great potential for application in modern driver assistance systems and automated vehicles. Functions based on deep neural networks process sensor data (e.g., from cameras, radar, or lidar sensors) to derive relevant information. This information includes, for example, the type and position of objects in the vehicle's surroundings, the behavior of the objects, or the road geometry or topology.

[0003] Among neural networks, convolutional neural networks (CNNs) have proven particularly suitable for applications in image processing. Convolutional networks gradually extract various high-quality features from input data (e.g., image data) in an unsupervised manner. During a training phase, the convolutional network independently develops feature maps based on filter channels that locally process the input data to derive local properties. These feature maps are then processed again by additional filter channels, which derive higher-quality feature maps. Based on this information condensed from the input data, the deep neural network ultimately derives its decision and provides this as output data.

[0004] While convolutional networks outperform traditional approaches in terms of functional accuracy, they also have drawbacks. For example, attacks based on adversarial perturbations in the sensor data / input data can lead to misclassification or incorrect semantic segmentation despite the semantically unchanged content of the acquired sensor data.

[0005] Chuan Guo et al., Countering Adversarial Images Using Input Transformations, axXiv:1711.00117v3 [cs.CV], 25 Jan 2018, https: / / arxiv.org / pdf / 1711.00117.pdf, describes a quilting method for removing adversarial noise in image data.

[0006] A key feature in the development of deep neural networks (training) is purely data-driven parameter fitting without expert intervention: This involves determining the deviation of a neural network's output (for a given parameterization) from a ground truth (the so-called loss). The loss function used here is chosen such that the neural network's parameters depend on this loss function in a differentiable manner. In the gradient descent method, the neural network's parameters are adjusted in each training step depending on the derivative of the deviation (determined from several examples). These training steps are repeated many times until the loss no longer decreases.

[0007] In this approach, the parameters of the neural network are determined without expert assessment or semantically motivated modeling. However, the parameters depend significantly on the data used for training. While machine-learned models can generally generalize very well (successfully apply what they have learned to unknown data), they can only do so within the data domains they were exposed to during the training process. Applying the trained model to data from other data domains, on the other hand, usually results in a significantly reduced output accuracy. This means that the AI functions provided by the neural network can, for example, only be used in regions or contexts from which the data used for training originates, or which are very similar to it.

[0008] US Pat. No. 9,864,931 B2 discloses a method and system for augmenting training data. A source domain and a target domain are provided, and then an operation is performed to augment data in the source domain with transformations that utilize features learned from the target domain. The augmented data is then used to improve the accuracy of image classification in a new domain.

[0009] The invention is based on the object of creating a method and a device for training a neural network, with which the neural network can be trained on a desired target data domain, for which in particular only a few sensor data are available that are marked with a ground truth ("marked data").

[0010] The object is achieved according to the invention by a method having the features of patent claim 1 and a device having the features of patent claim 9. Advantageous embodiments of the invention emerge from the subclaims.

[0011] In particular, a method for training a neural network is provided, wherein sensor data marked with a ground truth and unmarked sensor data of a target data domain are obtained, wherein sensor data marked with a ground truth of at least one starting data domain are obtained, and wherein a database is generated which forms a basis for a piecewise replacement of sensor data by means of quilting, wherein for this purpose sensor data patches are generated from the obtained sensor data and stored in the database, wherein the database is generated such that a reconstruction data domain defined by means of sensor data replaced piecewise by quilting has a smaller data domain distance to the target data domain than the at least one starting data domain, wherein the neural network is trained using training data,which are composed at least partially of the sensor data of the target data domain and the at least one starting data domain marked with a ground truth and piecewise replaced sensor data, wherein the piecewise replaced sensor data are generated by quilting using the generated database from the sensor data of the target data domain and the at least one starting data domain marked with the ground truth, and wherein the trained neural network is provided.

[0012] Furthermore, in particular, a device for training a neural network is provided, comprising a data processing device, wherein the data processing device is configured to receive sensor data marked with a ground truth and unmarked sensor data of a target data domain, to receive sensor data marked with a ground truth of at least one start data domain, and to generate a database that forms a basis for a piecewise replacement of sensor data by means of quilting, and for this purpose to generate sensor data patches from the obtained sensor data and to store them in the database, wherein the database is generated such that a reconstruction data domain defined by sensor data replaced piecewise by quilting has a smaller data domain distance to the target data domain than the at least one start data domain, to train the neural network using training data,which are composed at least partially of the acquired sensor data of the target data domain and the at least one starting data domain marked with a ground truth and piecewise replaced sensor data, and for this purpose, to generate the piecewise replaced sensor data by means of quilting using the generated database from the sensor data of the target data domain and the at least one starting data domain marked with the ground truth, and to provide the trained neural network.

[0013] The method and device make it possible to provide training data for a target data domain for which only a small amount of marked sensor data is available. Marked sensor data refers to sensor data for which a ground truth is known. For this purpose, the training data is partially generated by quilting from the sensor data of at least one starting data domain marked with a ground truth and sensor data of the target data domain marked with a ground truth. During quilting, the sensor data of the at least one starting data domain and the target data domain marked with a ground truth are replaced piece by piece using the sensor data patches stored in the database. In this process, a ground truth known for the marked sensor data is assigned to the replaced sensor data, so that the replaced sensor data is also marked with the respective ground truth.

[0014] To create the database, sensor data patches, which can also be referred to as data blocks, are generated from marked sensor data of the at least one start data domain and marked and unmarked sensor data of the target data domain. In particular, partial sections of the respective sensor data are generated and stored in the database. The sensor data patches can also be referred to as data blocks. The sensor data patches form, in particular, subsymbolic subsets of the sensor data. The database with the sensor data patches then forms the basis for the piecewise replacement of the sensor data performed during quilting. If the sensor data is, for example, a camera image, the sensor data patches are image sections of this camera image. The image sections or sensor data patches can, for example, have a size of 8x8 pixels.Quilting then involves replacing image sections from acquired sensor data with the sensor data patches that are most similar to the image section in terms of a distance measure.

[0015] The database is created in such a way that a reconstruction data domain, which is defined by sensor data replaced piecemeal by quilting, has a smaller data domain distance to the target data domain than the at least one starting data domain. This allows sensor data from starting data domains to be projected into the target data domain and made usable as training data for the target data domain. In particular, with such a projection, a known ground truth of sensor data from the at least one starting domain can be adopted or retained during projection. Cost-intensive acquisition and marking of sensor data in the target data domain can therefore be eliminated or at least reduced in scope. The target data domain can, for example, be sensor data from the city of New York, whereby only a few sensor data are marked with a ground truth and a large part of the sensor data is unmarked.A starting data domain can, for example, comprise sensor data from the city of Munich labeled with a ground truth. Using the method described in this disclosure, the labeled sensor data from Munich can be utilized for training on the target domain ("New York"). The reconstruction data domain defined by the method and device via the sensor data replaced by quilting using the generated database then has a smaller data domain distance to the target data domain than the starting data domain. By specifically generating the database from sensor data patches, in which unlabeled sensor data from the target data domain is also used to generate the sensor data patches, the replaced or reconstructed sensor data, and thus the reconstruction data domain formed thereby, can be shifted toward the target data domain.In this way, the respective ground truth of the supplied labeled sensor data can be retained during subsequent training. This allows training data labeled with a ground truth to be generated that has a smaller data domain distance to the target data domain ("New York") than the at least one starting data domain ("Munich").

[0016] It can be provided that the received sensor data is additionally marked with property information, which includes, for example, a type of sensor and / or a context in which the sensor data was recorded (e.g. weather, time, location, traffic scenario, etc.). This property information can each be assigned to a sensor data patch generated from it in the database. This can accelerate a search for a sensor data patch during piecewise replacement using quilting, since a set of sensor data patches can be restricted depending on predetermined property information. Furthermore, it can be provided that efficient hashing is performed for the database so that a search in the database can be accelerated during quilting.

[0017] Quilting specifically refers to the piecewise replacement of sensor data. Quilting can also be referred to as the piecewise reconstruction of the sensor data. In connection with image data, the term "image quilting" is also used. A set of replaced sensor data forms a reconstruction data domain or is encompassed by a reconstruction data domain. For example, if the images are from a camera, the camera image is divided into several subsections. For this purpose, small, rectangular image sections (also referred to as patches) can be defined. The individual sections or image sections are compared with subsections, referred to in this disclosure as sensor data patches, which are stored in the database. The comparison is based on a distance measure, which is defined, for example, via a Euclidean distance on image element vectors. For this purpose, a section or image section is linearized as a vector.The distance is then determined using a vector norm, for example the L2 norm. The partial or image sections from the acquired sensor data are each replaced by the nearest or most similar sensor data patch from the database. It can be stipulated that a minimum distance must be maintained or that at least there must be no identity between the partial section of the sensor data and the sensor data patch. If the sensor data has a different form (e.g. lidar data) or a different format, the piecewise replacement is carried out in a similar way. The piecewise replacement is carried out for all partial sections of the sensor data, so that replaced or reconstructed sensor data is subsequently available. After the piecewise replacement, i.e. after quilting, the effect of adversarial interference in the (replaced or reconstructed) sensor data is eliminated or at least reduced.

[0018] It can be provided that labeled and / or unlabeled sensor data from at least one additional target data domain are taken into account. This allows the neural network to be trained on multiple target data domains simultaneously.

[0019] In particular, the database generated by the method described in this disclosure can also be used in a method for robustifying sensor data against adversarial interference, wherein sensor data is obtained from at least two sensors, wherein the obtained sensor data of the at least two sensors are each replaced piecewise by quilting based on the sensor data patches stored in the generated database, wherein the piecewise replacement is carried out in particular such that each replaced sensor data from different sensors is plausible to one another, and wherein the piecewise replaced sensor data is output. The robustification method is carried out in particular by means of an associated robustification device.In particular, a robustification device for robustifying sensor data against adversarial disturbances is used for this purpose, comprising a computing device, wherein the computing device is configured to receive sensor data from at least two sensors, to replace the received sensor data of the at least two sensors piecewise by quilting on the basis of the sensor data patches stored in the generated database, and to carry out the piecewise replacement in particular in such a way that respectively replaced sensor data of different sensors are plausible to one another, and to output the piecewise replaced sensor data.

[0020] The "plausibility" of replaced sensor data should specifically mean that the replaced sensor data are physically plausible with each other. In particular, the probability that the replaced sensor data in the respective cross-sensor combination would also occur under real conditions, i.e., in the real world, should be as high as possible (e.g., in the sense of a maximum likelihood). In other words, the replaced sensor data of at least two sensors should be selected such that the probability that these sensor data would also actually occur in this combination is maximized.For example, if the at least two sensors are a camera and a lidar sensor, plausibility between the respectively replaced sensor data means that a viewed image section in the replaced camera data and a spatially and temporally corresponding partial section from the replaced lidar data are selected such that the sensor data are consistent with each other, i.e., physically consistent with each other. In the aforementioned example, in which the at least two sensors are a camera and a lidar sensor, the partial sections of the sensor data are each replaced such that each replaced image section corresponds to a replaced partial section of the lidar data, as would most likely also result from simultaneous acquisition of sensor data from the camera and the lidar sensor.

[0021] In particular, it is therefore provided that the generated database is provided. In particular, the generated database is output, for example, in the form of a data packet. The generated database can, for example, be loaded into a memory of a control device or a robustification device, as described above.

[0022] The sensor data, or the sensors with which the sensor data was acquired within a data domain, are calibrated to each other spatially and temporally, so that the sensor data from the sensors correspond spatially and temporally or have common temporal and spatial reference points. The sensor data originates, in particular, from different types of sensors, for example, from a camera and a lidar or radar sensor. In particular, the sensor data are physically plausible to each other, meaning that the sensor data do not physically contradict each other.

[0023] In principle, the sensor data can be one-dimensional or multi-dimensional, in particular two-dimensional. For example, the sensor data can be two-dimensional camera images from a camera and two-dimensional lidar data from a lidar sensor. However, it can also be sensor data from other sensors, such as radar sensors and / or ultrasonic sensors.

[0024] It may be provided that the sensor data for the target data domain and / or the source data domain are acquired within the scope of the method and at least partially marked with a respective ground truth. This is done in particular using appropriate sensors (e.g., one or more cameras, a lidar sensor and / or a radar sensor, etc.).

[0025] In particular, a data domain should refer to a set of sensor data that corresponds to a specific context or whose data is similar in at least one characteristic with regard to its origin. Such a context can be, for example, a geographical context; for example, one data domain can comprise sensor data from one city, while a different data domain can comprise sensor data from another city, etc.

[0026] The data domains are, in particular, data domains in the context of automated driving. These can include, in particular, recorded sensor data (measurement data) from various application scenarios of an automated vehicle with and without ground truth, homologation data and / or simulation data, as well as sensor data recorded in a vehicle fleet (which, for example, also include or depict rare events and / or typical error situations). In particular, it is envisaged that the sensor data recorded or stored in the data domains are sensor data that were recorded for a function for the automated driving of a vehicle and / or for a driver assistance system of the vehicle and / or for environmental detection or for a perception function. A vehicle is, in particular, a motor vehicle. In principle, however, the vehicle can also be another land, air, water, rail, or space vehicle.In principle, the method can also be used for other types of sensor data, for example in connection with robots, e.g. industrial robots or medical robots.

[0027] An adversarial perturbation is, in particular, a deliberate perturbation of the input data of a neural network, for example provided in the form of sensor data, in which a semantic content in the input data is not changed, but the perturbation leads to the neural network inferring an incorrect result, i.e., for example, a misclassification or an incorrect semantic segmentation of the input data.

[0028] A neural network is, in particular, a deep neural network, especially a convolutional neural network (CNN). The neural network is trained for a specific perceptual function, such as the perception of pedestrians or other objects in captured camera images.

[0029] As a data domain distance measure for determining a data domain distance, for example, statistical properties or statistical characteristics of data sets formed from sensor data of a known data domain and a data set of the reconstruction data domain formed from replaced sensor data can be determined and compared with each other. In the case of camera images, for example, color value histograms across the camera images and replaced camera images or the respective partial sections or sensor data patches can be compared with each other. A data domain and the reconstruction data domain can then be compared with each other based on the respectively determined statistical properties or characteristics of the associated data sets, and the data domain distance measure can be determined from this. The data domain distance measure between data sets of two data domains can, for example, be calculated from a difference between the respectively associated statistical characteristics.

[0030] Furthermore, it can be provided to determine the data domain distance measure based on a respective distribution of features in the data sets of the data domains formed from the original or replaced sensor data. For this purpose, features are extracted from the respective sensor data or from the partial sections of the sensor data, whose statistical distributions are then compared pairwise for a data set of a data domain and a data set of the reconstruction data domain. The data domain distance measure is then determined from the comparison result, for example, by calculating the difference between statistical parameters of the respective statistical distributions.

[0031] The features can, for example, originate from dimensionality reduction methods and / or be provided by kernel functions and / or feature maps of a (deep) neural network. Classification results from a (deep) neural network can also be used as features. The statistical distributions of the features are then compared using statistical parameters, such as expected values, for the data sets of the data domains and defined as the data domain distance between pairs of data sets of the data domains.

[0032] More complex methods are also possible to determine the data domain distance from the extracted features, such as sampled k-NN confusion (cross-domain retrieval). This involves creating three sets of features: one set each from the original and replaced data sets of the two data domains to be compared, and a mixed set with features from both data domains. All three sets are randomly drawn from the respective data sets (e.g., 1,000 samples per set). For each sample of the data domain-specific sets, the k-nearest neighbors in the mixed set are then searched for, and the number of these coming from the other data domain is counted. The determined numbers can then be used as a data domain distance measure.

[0033] The method can be implemented as a computer-implemented method. In particular, the method can be implemented using a data processing device. The data processing device comprises, in particular, at least one computing device and at least one memory device.

[0034] In particular, a computer program is also provided, comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method steps of the disclosed method according to any of the described embodiments.

[0035] In addition, in particular, a data carrier signal is created which transmits the aforementioned computer program.

[0036] Parts of the device, in particular the data processing device, can be designed individually or collectively as a combination of hardware and software, for example as program code that is executed on a microcontroller or microprocessor.

[0037] Selecting a partial section from the sensor data in order to generate a sensor data patch from the selected partial section can be done in different ways. In a simple embodiment, the sensor data patches are generated randomly, for example using a Monte Carlo method, from the received sensor data. However, it can also be provided that the sensor data patches are selected or generated based on property information with which the received sensor data is marked ("tagged"). The property information can, for example, contain a location, a time, and / or a context that characterizes the acquired sensor data. By generating the sensor data patches depending on the property information, it can be ensured, for example, that certain locations, certain times, and / or certain contexts are mapped to the database via the sensor data patches.In particular, regulatory requirements can also be taken into account here.

[0038] In one embodiment, simulated sensor data marked with a ground truth is obtained, wherein the training data additionally comprises at least partially the simulated sensor data and piecewise replaced sensor data generated from the simulated sensor data by quilting using the generated database. This makes it possible to also use simulated sensor data as training data when training the neural network. This makes it possible, in particular, to generate and provide sensor data marked with a ground truth in a cost-effective manner. For example, situations and scenarios that only occur very rarely in the real world can be simulated and taken into account during training. This can increase the reliability when using the neural network, since it then exhibits a high level of quality even in rare situations and scenarios.The simulated sensor data is replaced piecewise using quilting with the generated database and thus projected into the target data domain. A respective ground truth with which the simulated sensor data is marked can be adopted for the replaced sensor data.

[0039] In one embodiment, a target domain reconstruction database is generated, which forms a basis for piecewise replacement of sensor data by quilting. For this purpose, sensor data patches are generated from the obtained sensor data of the target data domain and stored in the target domain reconstruction database. This can provide a basis for comparison of generated reconstruction data domains with the target data domain.

[0040] In a further embodiment, it is provided that the proportion of the respective sensor data of a data domain (start data domain(s) and / or target data domain) and the respective piecewise replaced sensor data of the data domains in the training data is selected to be larger, the smaller a data domain distance of the data domain to a reconstruction data domain is, which is formed from replaced sensor data generated by quilting the sensor data using the target domain reconstruction database. This allows the sensor data comprised by the training data to be shifted in a targeted manner toward the target data domain.

[0041] In a further further development, it is provided that the neural network is retrained, wherein for this purpose at least one increment of the database is generated by changing the database, the reconstruction data domain of which has a predetermined data domain distance to a reconstruction data domain generated using the target domain reconstruction database, wherein the retraining is carried out using training data which comprise sensor data replaced by quilting using the respective increment of the database, wherein the retraining is carried out until a performance of the neural network reaches a predetermined value when applied to replaced sensor data which was generated using the target domain reconstruction database by quilting sensor data of the target data domain.In other words, the database is gradually modified by modification such that a reconstruction data domain associated with the respective increments of the database gradually has a smaller data domain distance from the respective reconstruction data domain formed using the target domain reconstruction database. This allows the training data generated using the database increments to be gradually shifted further and further toward the target data domain while maintaining a respective ground truth. Furthermore, this allows for continuous retraining of the (trained) neural network for slight changes in the target data domain, in which newly obtained sensor data patches from the (slightly) modified target data domain are continuously taken into account when generating an increment of the database.Changing the database can be done, for example, by changing a number of sensor data patches, a size of the sensor data patches and / or by choosing a different selection method when selecting the sensor data patches during the creation of the database and / or during the quilting step.

[0042] In one embodiment, it is provided that sensor data included in the training data are replaced piece by piece by means of quilting, wherein this is repeated for a predetermined number of steps, and wherein with each step a proportion of sensor data patches that were generated starting from sensor data of the target data domain is increased in the database, and wherein the neural network is subsequently trained with the training data that was replaced piece by piece in this way. From the replaced sensor data generated in this way, training data can be generated that are closer to the target data domain with each iteration. This makes it possible, in particular, to start from only very few recorded sensor data from a target data domain (e.g., New York) that are marked with a ground truth, but many sensor data from at least one starting data domain (e.g.,Munich) and many simulated sensor data labeled with a ground truth to create training data that lies in the target data domain. This is achieved, in particular, by quilting projection using the database. Since the database is also generated using acquired sensor data that is unlabeled, a projection of the piecewise replaced sensor data into the target data domain can be improved by increasing the proportion of sensor data patches generated from the target data domain. As this proportion is gradually increased, a gradual projection of the training data towards the target data domain can be achieved without losing the original ground truth.

[0043] It can be provided that during an application phase within the framework of the method for robustifying sensor data against adversarial interference or by the robustification device during piecewise replacement by quilting, a distance between the acquired sensor data and the sensor data patches stored in the database used for piecewise replacement is checked. If this distance exceeds a predetermined limit (e.g., on average), the method can be carried out, particularly in the embodiments described above, to adapt the database. This can create an "out-of-sample" detector, i.e., a detector that determines that the reconstruction data domain mapped via the database has deviated from the (updated) target data domain.It can also be provided that, once the specified limit is exceeded, the acquisition of additional sensor data from the target data domain is initiated, for example, using vehicles in a fleet. Safety measures (e.g., a more defensive driving style) can also be initiated, for example, because the currently detected environment no longer corresponds, or only partially corresponds, to the context anticipated at the time of development and rollout, and thus functional quality may only be partially ensured.

[0044] In one embodiment, it is provided that the neural network provides a function for the automated driving of a vehicle and / or for driver assistance of the vehicle and / or for environmental detection and / or a perception function.

[0045] Further features of the device design will become apparent from the description of embodiments of the method. The advantages of the device are the same as those of the embodiments of the method.

[0046] In an application phase, the provided trained neural network is applied to piecewise sensor data replaced by quilting. The piecewise replacement is performed using the same database used to train the neural network. This also projects the sensor data acquired in the application phase into the same reconstruction data domain on which the neural network was trained. At the same time, the acquired sensor data is robustified against adversarial perturbations through piecewise replacement.

[0047] Therefore, in particular, a method for providing a neural network is also provided, wherein the neural network is or has been trained in a training phase by means of the method described in this disclosure in one of the described embodiments, and wherein during an application phase, sensor data supplied to the neural network as input data are previously replaced piecewise by means of quilting using the database generated in the training phase.

[0048] The invention will be explained in more detail below using preferred embodiments with reference to the figures. Fig. 1 a schematic representation of an embodiment of the device for training a neural network; Fig. 2 a schematic representation of an embodiment of the method for training a neural network and its embedding in a method for operating the neural network.

[0049] In Fig. Figure 1 shows a schematic representation of an embodiment of the device 1 for training a neural network 30. The device 1 comprises a data processing device 2, which includes a computing device 3 and a storage device 4. The device 1 executes the method for training a neural network 30 described in this disclosure.

[0050] Parts of the device 1, in particular the data processing device 2, can be designed individually or collectively as a combination of hardware and software, for example as program code that is executed on a microcontroller or microprocessor.

[0051] The computing device 3 receives, in particular, sensor data 20 of a target data domain 10 marked with a ground truth and unmarked sensor data 21 of the target data domain 10 as well as sensor data 22 of at least one start data domain 11 marked with a ground truth. This is done, for example, via an input interface 5. The sensor data 20, 21, 22 are stored in the storage device 4.

[0052] A structure and initial parameters of the neural network 30 to be trained are also stored in the storage device 4. The structure and the initial parameters can also have been obtained or received via the input interface 5.

[0053] The computing device 3 generates a database 40, which forms the basis for a piecewise replacement of sensor data by means of quilting. For this purpose, the computing device 3 generates a predetermined number (e.g., 10,000, 100,000, etc.) of sensor data patches 60 from the received sensor data 20, 21, 22 and stores them in the database 40. The sensor data patches 60 each correspond to partial sections of the sensor data 20, 21, 22 with a predetermined size (for camera images, e.g., 8x8 image elements each).

[0054] The database 40 is generated by the computing device 3 in such a way that a reconstruction data domain 13 defined by means of sensor data replaced piece by piece by means of quilting (cf. Fig. 2) has a smaller data domain distance to the target data domain 10 than the at least one start data domain 11. For this purpose, the computing device 3 can, for example, replace a plurality of sensor data 20, 21, 22 piecewise using the generated database 40 by means of quilting and, via a predetermined data domain distance measure, determine a data domain distance between the reconstruction data domain 13 defined by the replaced or reconstructed sensor data 20-e, 21-e, 22-e ( Fig. 2) and the starting data domain 11 or the target data domain 10. In addition, the data domain distance between the starting data domain 11 and the target data domain 10 is also determined. The database 40 can then be adapted or optimized step by step such that the determined data domain distance fulfills the specified condition. The adaptation can be performed, for example, by changing a number of sensor data patches 60, a size of the sensor data patches 60, and / or by choosing a different selection method when selecting the sensor data patches 60 during the quilting step.

[0055] The computing device 3 then trains the neural network 30 using training data 25, which is composed at least partially of the acquired sensor data 20 of the target data domain 10, marked with a ground truth, and the marked sensor data 22 of the starting data domain 11, as well as piecewise replaced sensor data 20-e, 22-e. To this end, the computing device 2 generates the piecewise replaced sensor data 20-e, 22-e by quilting using the generated database 40 from the sensor data 20 of the target data domain 10, marked with the ground truth, and the marked sensor data 22 of the starting data domain 11.

[0056] After training, the trained neural network 31 is provided by the computing device 3. In particular, the trained neural network 31 is output via an output interface 6 and loaded, for example, into a memory of a control device 50 of a vehicle.

[0057] The generated database 40 is also provided and, in particular, output via the output interface 6. The generated database 40 is, for example, also loaded into the memory of the control device 50. The control device 50 then operates the trained neural network 31 and, for this purpose, robustifies the sensor data 32 supplied to the trained neural network 31 against adversarial interference using a robustification device 70. For this purpose, the supplied sensor data 32 are replaced piecewise by the robustification device 70 by quilting using the database 40. The replaced sensor data 33 are supplied to the trained neural network 31.

[0058] The trained neural network 31 provides in particular a function for the automated driving of a vehicle and / or for driver assistance of the vehicle and / or for environmental detection and / or a perception function, wherein, based on detected sensor data 32, a result 34 is inferred and provided by means of the trained neural network 31, which serves, for example, as a basis for controlling and / or regulating longitudinal and / or lateral guidance of the vehicle.

[0059] It can be provided that simulated sensor data 23 marked with a ground truth are also obtained, wherein the training data 25 additionally at least partially comprise the simulated sensor data 23 and piecewise replaced sensor data 23-e generated from the simulated sensor data 23 by quilting using the generated database 40. This makes it possible to increase the range of situations and scenarios (e.g., critical or rare traffic situations) that are represented in the training data 25. In particular, this makes it possible to cost-effectively generate extensive training data 25 that lie within the target data domain 10.

[0060] It can be provided that a target domain reconstruction database 41 is generated, which forms a basis for a piecewise replacement of sensor data by means of quilting, wherein for this purpose sensor data patches 61 generated from the received sensor data 20, 21 of the target data domain 10 are generated and stored in the target domain reconstruction database 41.

[0061] In a further development, it can be provided that a proportion of the respective sensor data 20, 22, 23 of a data domain 10, 11 and of the respectively piecewise replaced sensor data 20-e, 22-e, 23-e of the data domains 10, 11 in the training data 25 is or is selected to be greater, the smaller a data domain distance of the data domain 10, 11 to a reconstruction data domain is, which is formed from replaced sensor data 20-e, 22-e, 23-e generated by quilting the sensor data 20, 22, 23 using the target domain reconstruction database 41.

[0062] Likewise, in a further development, it can be provided that the neural network 30 is retrained, wherein for this purpose at least one increment of the database 40 is generated by changing the database 40, the reconstruction data domains of which have a predetermined data domain distance to a reconstruction data domain generated using the target domain reconstruction database 41, wherein the retraining is carried out using training data which comprise sensor data 20-e, 22-e, 23-e replaced by quilting using the respective increment of the database 40, wherein the retraining is carried out until a performance of the neural network 30 when applied to replaced sensor data 20-e, 20-e, 23-e, which were generated using the target domain reconstruction database 41 by quilting sensor data 20, 21, 23 of the target data domain 10, reaches a predetermined value.The specified value is, for example, a specified quality when classifying the piecewise replaced sensor data 20-e, 22-e, 23-e, for example when detecting objects in camera images or in semantic segmentation.

[0063] It can be provided that sensor data 20, 22, 23, 20-e, 22-e, 23-e included in the training data 25 are replaced piecewise by quilting, wherein this is repeated for a predetermined number of steps, and wherein with each step, a proportion of sensor data patches 60, which were generated from sensor data 20, 21, 23 of the target data domain 10, is increased in the database 40, and wherein the neural network 30 is subsequently trained with the training data 25-e replaced piecewise in this way. As a result, a reconstruction data domain defined by the replaced training data 25-e can be gradually shifted further and further toward the target data domain 10 while retaining the respective associated markings with a ground truth. In particular, this enables a cost-effective generation and use of simulated sensor data 23 to provide the training data.

[0064] In Fig.2 is a schematic representation to illustrate a sequence in an embodiment of the method for training a neural network and its embedding in a method for operating a neural network.

[0065] Starting from sensor data 20 and unmarked sensor data 21 of a target data domain marked with a ground truth, sensor data 22 of at least one starting data domain marked with a ground truth, and marked simulated sensor data 23, a database 40 of sensor data patches is generated in a method step 100. This can be done taking into account regulatory specifications 15, which, for example, specify which contexts and / or application scenarios must be mapped via the sensor data patches in the database 40. The regulatory specifications 15 are taken into account when generating the sensor data patches and when generating the database 40. The database 40 is generated such that a reconstruction data domain 13 defined via sensor data 20-e, 22-e, 23-e piecewise replaced by quilting has a smaller data domain distance to the target data domain than the at least one starting data domain.

[0066] In a method step 200, sensor data 20, 22, 23, which are marked with a ground truth 24, are replaced piecewise using the generated database 40 by means of quilting, so that replaced sensor data 20-e, 22-e, 23-e, which can also be referred to as reconstructed sensor data, are generated. A ground truth 24 is adopted for the replaced sensor data 20-e, 22-e, 23-e from the sensor data 20, 22, 23, respectively.

[0067] The marked sensor data 20, 22, 23 and the replaced sensor data 20-e, 22-e, 23-e are used as training data 25 of the neural network. The neural network is trained using the training data 25 in a method step 300 in a manner known per se. The trained neural network 31 is subsequently provided.

[0068] In particular, the trained neural network 31 is output in the form of a data packet and transferred to a control device 50. There, the trained neural network 31 is loaded, for example, into a memory of the control device 50.

[0069] Furthermore, the generated database 40 is also provided. In particular, it is output in the form of a data packet and transferred to a control device 50. There, the database 40 is loaded, for example, into a memory of a robustification device 70.

[0070] In an application phase, sensor data 32 acquired by sensors 51 are replaced piecewise by quilting in a method step 400 using the robustification device 70, thereby making them robust against adversarial disturbances. In a method step 500, the trained neural network 31 is applied to the replaced sensor data 33. As a result, the trained neural network 31 provides, for example, a classification and / or recognized objects and / or a semantic segmentation.

[0071] The advantage of the method and device for training a neural network 30 is that, starting from a few labeled sensor data 20 of the target data domain, extensive training data 25 can be generated by projecting labeled sensor data 22 from starting domains and labeled simulated sensor data 23 into a reconstruction data domain 13 which has a smaller data domain distance to the target data domain. List of reference symbols 1 device 2 Data processing facility 3 Computing device 4 Storage device 5 Input interface 6 Output interface 10 Target data domain 11 Start data domain 20 sensor data (marked) 20-e replaced sensor data 21 Sensor data (unmarked) 21-e replaced sensor data 22 Sensor data (marked) 22-e replaced sensor data 23 simulated sensor data (marked) 23-e replaced sensor data 24 Basic Truth 25 training data 25-e replaced training data 30 Neural Network 31 trained neural networks 32 sensor data collected (application phase) 33 replaced sensor data (application phase) 34 results 40 database 41 Target domain reconstruction database 50 Control device 51 sensors 60 sensor data patch 70 Robustification device 100 process steps 200 process steps 300 process steps 400 process steps 500 process steps

Claims

[1] Method for training a neural network (30), wherein sensor data (20) marked with a ground truth (24) and unmarked sensor data (21) of a target data domain (10) are obtained, wherein sensor data (22) marked with a ground truth (24) of at least one start data domain (11) are obtained, and wherein a database (40) is generated which forms a basis for a piecewise replacement of sensor data (20, 21, 22) by means of quilting, wherein for this purpose sensor data patches (60) are generated from the received sensor data (20, 21, 22) and stored in the database (40), wherein the database (40) is generated such that a reconstruction data domain defined by means of quilting-replaced sensor data (20-e, 22-e) has a smaller data domain distance to the target data domain (10) than the at least one start data domain (11), wherein the neural network (30) is trained using training data (25) which is composed at least partially of the sensor data (20, 22) of the target data domain (10) and the at least one starting data domain (11) marked with a ground truth (24) and piecewise replaced sensor data (20-e, 22-e), wherein the piecewise replaced sensor data (20-e, 22-e) are generated by quilting using the generated database (40) from the sensor data (20, 22) of the target data domain (10) and the at least one starting data domain (11) marked with the ground truth (24), and wherein the trained neural network (31) is provided. [2] Method according to claim 1, characterized bythat simulated sensor data (23) marked with a ground truth (24) are obtained, wherein the training data (25) additionally at least partially comprise the simulated sensor data (23) and piecewise replaced sensor data (23-e) generated from the simulated sensor data (23) by means of quilting using the generated database (40). [3] Method according to claim 1 or 2, characterized by that a target domain reconstruction database (41) is generated which forms a basis for a piecewise replacement of sensor data (20, 22, 23) by means of quilting, wherein for this purpose sensor data patches (60) generated from the received sensor data (20, 21) of the target data domain (10) are generated and stored in the target domain reconstruction database (41). [4] Method according to claim 3, characterized bythat a proportion of the respective sensor data (20, 22, 23) of a data domain (10, 11) and of the respectively piecewise replaced sensor data (20-e, 22-e, 23-e) of the data domains (10, 11) in the training data (25) is or is selected to be greater, the smaller a data domain distance of the data domain (10, 11) is to a reconstruction data domain which is formed from replaced sensor data (20-e, 22-e, 23-e) generated by quilting the sensor data (20, 22, 23) using the target domain reconstruction database (41). [5] Method according to claim 3 or 4, characterized byin that the neural network (30, 31) is retrained, wherein for this purpose at least one increment of the database (40) is generated by changing the database (40), the reconstruction data domain of which has a predetermined data domain distance to a reconstruction data domain generated using the target domain reconstruction database (41), wherein the retraining is carried out using training data (25) which comprise sensor data (20-e, 22-e, 23-e) replaced by quilting using the respective increment of the database (40), wherein the retraining is carried out until a performance of the neural network (30, 31) reaches a predetermined value when applied to replaced sensor data (20-e, 21-e) which were generated using the target domain reconstruction database (41) by quilting sensor data (20, 21) of the target data domain (10). [6] Method according to one of the preceding claims, characterized bythat sensor data (20, 22, 23) comprised in the training data (25) are replaced piece by piece by means of quilting, this being repeated for a predetermined number of steps, and with each step a proportion of sensor data patches (60) generated from sensor data (20, 21) of the target data domain (10) is increased in the database (40), and the neural network (30) is subsequently trained with the training data (25-e) replaced piece by piece in this way. [7] Method according to one of the preceding claims, characterized by that the neural network provides a function for the automated driving of a vehicle and / or for driver assistance of the vehicle and / or for environmental detection and / or a perception function. [8] Method for providing a neural network (30, 31), wherein the neural network (30, 31) is or was trained in a training phase by means of a method according to one of claims 1 to 7, and wherein during an application phase, sensor data (32) supplied to the neural network (31) as input data are previously replaced piece by piece by means of quilting using the database (40) generated in the training phase. [9] Device (1) for training a neural network (30), comprising: a data processing device (2), wherein the data processing device (2) is designed to to obtain sensor data (20) marked with a ground truth (24) and unmarked sensor data (21) of a target data domain (10), to obtain sensor data (22) marked with a ground truth (24) of at least one start data domain (11), and to generate a database (40) which forms a basis for a piecewise replacement of sensor data (20, 22) by means of quilting, and for this purpose to generate sensor data patches (60) from the received sensor data (20, 21, 22) and to store them in the database (40), wherein the database (40) is generated such that a reconstruction data domain defined by means of quilting-replaced sensor data (20-e, 22-e) has a smaller data domain distance to the target data domain (10) than the at least one start data domain (11), to train the neural network (30) using training data (25) which are composed at least partially of the acquired sensor data (20) of the target data domain (10) and the at least one starting data domain (11) marked with a ground truth (25) and piecewise replaced sensor data (20-e, 22-e), and to generate the piecewise replaced sensor data (20-e, 22-e) by quilting using the generated database (40) from the sensor data (20, 22) of the target data domain (10) and the at least one starting data domain (11) marked with the ground truth (40), and to provide the trained neural network (31). [10] A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method steps of the method according to any one of claims 1 to 8, or a data carrier signal which transmits such a computer program.

Images