Device and method for training a model, and vehicle

Abstract

The present invention relates to a method for training a model in a computer-implemented manner, wherein the model has a first sub-model and a second sub-model: wherein digital data is scaled down to generate first output data; the digital data is divided into multiple data areas to generate second input data; the first sub-model generates first sub-model data for the first input data supplied to the first sub-model; the first sub-model data is scaled up to form first output data; the second sub-model generates an output data area corresponding to the data area for the second input data supplied to the second sub-model; the output data areas are combined into second output data; the first output data and the second output data are merged into third output data; the first sub-model is trained by comparing the provided target data assigned to the digital data with the first output data; and the second sub-model is trained by comparing the target data with the third output data.

Description

Technical Field

[0001] Various embodiments generally relate to an apparatus and method for training a model. Background Art

[0002] Imaging sensors, such as camera sensors and video sensors, capture digital image data with increasingly higher resolutions. For example, in the case of autonomous driving, it is necessary to be able to process these large data volumes in order to preserve the required detail information in the processed data so that objects, such as road signs, pedestrians, obstacles, etc., can be recognized and in order to control the vehicle, for example, based on the recognized objects. In this context, models such as neural networks are often used for object recognition.

[0003] However, the processing of data for object recognition using models requires a lot of memory during internal calculations. As a result, the amount of data that can be processed for network operation is limited. This leads to limitations such as data quality, image size, time scale, etc. Therefore, it may be necessary that the model for processing data requires less memory for internal calculations.

[0004] In Zhu et al., Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, arXiv:1703.10593, 2018, a method for unpaired image-to-image translation using two coupled GAN (generative adversarial network) neural networks is described.

[0005] In Liu et al., Unsupervised Image-to-Image Translation Networks, arXiv:1703.00858, 2018, a method for unpaired image-to-image translation using two coupled GANs is described, where the encoders of the first GAN and the second GAN are mapped into a common latent space.

[0006] A method for predicting a traffic situation is known from DE 10 2016 225 772 A1.

[0007] A method for operating a data analysis system for a motor vehicle function is known from DE 10 2017 116 016 A1.

[0008] A neural network for generating a representation of the surroundings is known from DE 10 2017 201 226 A1.

[0009] A method for developing a machine operation classifier is known from DE 11 2015 003 762 T5. Summary of the invention

[0010] Accordingly, a method and a device having the features of independent claims 1 (first example) and 8 (thirty-sixth example) have the object of training a model in such a way that the trained model has a reduced internal memory requirement when processing data.

[0011] The model may be any type of algorithm based on mathematical logic that provides an output based on an input using the algorithm. The model may have a matrix multiplication or a principal component analysis. The model may have a neural network. The neural network may be any type of neural network, such as an autoencoder network or a CNN (convolutional neural network). The neural network may have an encoder function and a decoder function. The neural network may have any number of layers, and the training of the neural network, that is to say the adaptation of the layers of the neural network, may be based on any type of training principle, such as back propagation, that is to say the back propagation algorithm.

[0012] The digital data may be data detected by a sensor. The features described in this paragraph form a second example in combination with the first example.

[0013] The first sub-model and / or the second sub-model may be computer-implemented.The features described in this paragraph in combination with the first example or the second example form a third example.

[0014] The first sub-model may generate first sub-model data for the supplied first input data, the first sub-model data having the same resolution as the first input data. The features described in this paragraph are combined with one of the first to third examples to form a fourth example.

[0015] The first sub-model and / or the second sub-model may have an autoencoder. The features described in this paragraph are combined with one of the first to fourth examples to form a fifth example.

[0016] The first sub-model and / or the second sub-model may have matrix multiplication, principal component analysis or a neural network. The features described in this paragraph are combined with one of the first to fifth examples to form a sixth example.

[0017] The calculation parameters of the first sub-model and / or the calculation parameters of the second sub-model may be stored in at least one memory. The calculation parameters of the first sub-model and the calculation parameters of the second sub-model may have weighting and / or activation. The features described in this paragraph are combined with one of the first to sixth examples to form a seventh example.

[0018] The digital data may include imaging sensor data, radar sensor data, LIDAR sensor data, driving dynamics-specific sensor data or engine-specific sensor data. Driving dynamics-specific sensor data may include speed, acceleration or angle. Engine-specific sensor data may include accelerator pedal position, injection quantity, temperature or rotational speed. The digital data may also include or may be ultrasonic sensor data or thermal sensor data. The features described in this paragraph are combined with one of the first to seventh examples to form an eighth example.

[0019] The digital data may have one of the following digital data types: image data, point cloud, or time series. The features described in this paragraph are combined with one of the first to eighth examples to form a ninth example.

[0020] The first input data may have global (large-scale) information of the digital data. The features described in this paragraph are combined with one of the first to ninth examples to form a tenth example.

[0021] The scaling down of the digital data may have a reduction in the scaling factor. The scaling down of the digital data may have a scaling factor that reduces the scaling factor by 4 to 8. The scaling down of the digital data has the effect that the model requires less memory for internal calculations on the one hand and requires less computational effort to process the digital data on the other hand. The memory requirement is increased by 1 / S d Proportionally reduced, where S describes a scaling factor and d describes a spatial dimension, where d is equal to 1 if the digital data has a time series, where d is equal to 2 if the digital data has image data, and where d is equal to 3 if the digital data has a three-dimensional spatial matrix. The features described in this paragraph are combined with one of the first to tenth examples to form an eleventh example.

[0022] The second input data may have local (small-scale) information of the digital data. The features described in this paragraph form a twelfth example in combination with one of the first to eleventh examples.

[0023] The second sub-model can generate second sub-model data for the supplied second input data, wherein for each of the plurality of data areas, a corresponding output data area in the plurality of output data areas can be generated. The features described in this paragraph are combined with one of the first to twelfth examples to form a thirteenth example.

[0024] The division of digital data into a plurality of data areas may include: the digital data is divided into a plurality of data areas according to a division key. The division of digital data into a plurality of data areas may include: the digital data is divided into 16 to 64 data areas. The division of digital data has the following effect: the model requires less memory for internal calculations on the one hand and requires lower computing technology expenditures on the other hand to process the digital data. The features described in this paragraph are combined with one of the first to thirteenth examples to form the fourteenth example.

[0025] Each of the plurality of data areas may have the same scaling ratio. The scaling ratio of each of the plurality of data areas may correspond to the scaling ratio of the first input data. In other words, the digital data may be divided into a plurality of data areas based on a uniform grid. The features described in this paragraph are combined with one of the first to fourteenth examples to form a fifteenth example.

[0026] Each of the first output data may be assigned with exactly one of the digital data, and each of the second output data may be assigned with exactly one of the digital data. The features described in this paragraph are combined with one of the first to fifteenth examples to form a sixteenth example.

[0027] The scaling ratio of the first sub-model data may correspond to the scaling ratio of the first input data.The features described in this paragraph are combined with one of the first to sixteenth examples to form a seventeenth example.

[0028] The scaling up of the first sub-model data may have an increase in a scaling ratio.The features described in this paragraph are combined with one of the first to seventeenth examples to form an eighteenth example.

[0029] The scaling ratio of the first output data may correspond to the scaling ratio of the target data.The features described in this paragraph are combined with one of the first to eighteenth examples to form a nineteenth example.

[0030] The scaling ratio of the target data may correspond to the scaling ratio of the digital data.The features described in this paragraph are combined with one of the first to nineteenth examples to form a twentieth example.

[0031] The output data area may have a plurality of output data areas. Each of the plurality of output data areas may be allocated exactly one of the plurality of data areas. The scaling ratio of each of the plurality of output data areas may correspond to the scaling ratio of the allocated data area of ​​the plurality of data areas. Combining the output data areas into the second output data may include: combining the plurality of output data areas based on a partitioning key. The features described in this paragraph are combined with one of the first to twentieth examples to form the twenty-first example.

[0032] The scaling ratio of the second output data may correspond to the scaling ratio of the first output data.The features described in this paragraph are combined with one of the first to twenty-first examples to form a twenty-second example.

[0033] Combining the first output data and the second output data into the third output data may include: adding the first output data and the second output data. The features described in this paragraph are combined with one of the first to twenty-second examples to form a twenty-third example.

[0034] Combining the first output data and the second output data into the third output data may include: weighted addition of the first output data and the second output data. The features described in this paragraph are combined with the twenty-third example to form a twenty-fourth example.

[0035] Adding the first output data and the second output data may include: adding each of the first output data and the allocated second output data of the second output data.The features described in this paragraph form a twenty-fifth example in combination with the twenty-second example or the twenty-fourth example.

[0036] The scaling ratio of the third output data may correspond to the scaling ratio of the target data.The features described in this paragraph are combined with one of the first to twenty-fifth examples to form a twenty-sixth example.

[0037] Training the first sub-model on digital data may include determining a first loss value by comparing the first output data with the target data and may include adapting the first sub-model based on the first loss value. The first loss value may be determined using a first loss function. The adaptation of the first sub-model may include minimizing the first loss value. That is, the first sub-model is trained so that the first sub-model represents the target data as accurately as possible. This has the following effect: the first sub-model can, for example, learn to represent large areas, in particular uniform areas and / or main color distributions. The features described in this paragraph are combined with one of the first to twenty-sixth examples to form the twenty-seventh example.

[0038] Training the second sub-model on digital data may have the ability to determine a second loss value by comparing the third output data with the target data and may have the ability to adapt the second sub-model based on the second loss value. The second loss value may be determined using a second loss function. The adaptation of the second sub-model may have the ability to minimize the second loss value. That is, the second sub-model represents a difference mask between the first output data and the target data, so that third output data representing the target data can be generated by combining the first output data with the second output data. In other words, the second sub-model learns the difference between the first output data generated by the first sub-model and the target data. This has the effect that, unlike the first output data, the third output data reproduces the details of the target data. This also has the effect that the second sub-model can, for example, learn representations of edges and / or microstructures. The features described in this paragraph are combined with one of the first to twenty-seventh examples to form the twenty-eighth example.

[0039] The target data may correspond to digital data. The features described in this paragraph are combined with one of the first to twenty-eighth examples to form a twenty-ninth example.

[0040] The method may also have the function of generating a third input data. The third input data may have a plurality of second data areas of digital data. The method may have: generating a plurality of second output data areas for the plurality of second data areas by a third sub-model. The method may have: combining the plurality of second output data areas into a fourth output data. The method may have: merging the third output data and the fourth output data into a fifth output data. The method may have: training the third sub-model on the digital data using the target data and the fifth output data. The third sub-model may have an automatic encoder. The third sub-model may have a matrix multiplication, a principal component analysis, or a neural network. The features described in this paragraph are combined with one of the first to twenty-ninth examples to form the thirtieth example. It should be noted that the method may have any number of iterations (for example, in the case of a fourth sub-model, a fifth sub-model, a sixth sub-model, a seventh sub-model, etc.) and corresponding additional sub-models.

[0041] The calculation parameters of the third sub-model may be stored in at least one memory. The calculation parameters of the third sub-model may have weighting and / or activation. The features described in this paragraph are combined with the 30th example to form a 31st example.

[0042] A scaling ratio of a data area in the plurality of second data areas may be smaller than a scaling ratio of a data area in the plurality of data areas.The features described in this paragraph are combined with the thirtieth example or the thirty-first example to form a thirty-second example.

[0043] The third sub-model may generate third sub-model data for the delivered third input data, wherein a corresponding second output data area in the plurality of second output data areas is generated for each second data area in the plurality of second data areas. The features described in this paragraph are combined with one of the 30th to 32nd examples to form a 33rd example.

[0044] The model may be a generative model. The digital data may belong to the first domain, and the third output data may belong to the second domain. The method may also include: generating training data by the trained model. The method may include: training the second model using the generated training data. The features described in this paragraph are combined with one of the first to thirty-third examples to form a thirty-fourth example.

[0045] The method may also include: processing the second digital data by the second model in the driving assistance system. The second digital data may belong to the second domain. The method may also include: controlling the vehicle based on the processed second digital data. The features described in this paragraph are combined with the thirty-fourth example to form the thirty-fifth example.

[0046] The device may have a model. The model may have been trained according to a method according to one of the first to thirty-third examples. The trained model may be used, for example, to encode, convert or convert digital data (e.g., measurement data, e.g., simulation data) to another domain. That is, digital data can be encoded, converted or converted to another domain in a memory-efficient manner. A device having the features described in this paragraph forms a thirty-seventh example.

[0047] The device may also have a second model. The second model may have been trained according to the method according to the thirty-fourth example. The features described in this paragraph are combined with the thirty-seventh example to form the thirty-eighth example.

[0048] The device may have sub-models of the model, such as a first sub-model, a second sub-model and / or a third sub-model, wherein the model having the first sub-model and the second sub-model and optionally the third sub-model and optionally one or more additional sub-models may have been trained according to a method according to one of the first to thirty-second examples. The first output data generated based on the digital image data using the trained first sub-model can, for example, be processed by a segmentation network, wherein the segmentation network can output segmented digital image data. The second output data or the third output data generated using the trained second sub-model and / or the trained third sub-model can be used to identify edge structures and / or object structures. A device having the features described in this paragraph forms the thirty-ninth example.

[0049] The first sub-model, the second sub-model and / or the third sub-model may be a neural network. The features described in this paragraph are combined with one of the thirty-sixth to thirty-ninth examples to form a fortieth example.

[0050] The second model may be a neural network. The features described in this paragraph are combined with one of the thirty-eighth to fortieth examples to form a forty-first example.

[0051] The system may have a device according to one of the thirty-sixth to forty-first examples. The system may also have a sensor. The sensor may be set up to provide digital data to the device. A system having the features described in this paragraph forms a forty-second example.

[0052] The system may have a GAN (generative adversarial network). The GAN may have multiple generative networks. At least one of the multiple generative networks may have been trained according to one of the first to thirty-fifth examples. This has the following effect: the trained generative network can process digital data with greater memory requirements, such as higher data quality or larger image size, while the memory capacity remains unchanged, and thus can generate digital data with higher data quality or larger image size. The system having the features described in this paragraph forms the forty-third example.

[0053] The computer program may have program instructions, which are set up to implement the method according to one or more of the first to thirty-fifth examples when the program instructions are executed by one or more processors. The features described in this paragraph form a forty-fourth example.

[0054] The computer program may be stored in a machine-readable storage medium. The features described in this paragraph are combined with the forty-fourth example to form a forty-fifth example.

[0055] The vehicle may have at least one sensor that is configured to provide digital data. The vehicle may also have a driver assistance system. The driver assistance system may have a model trained according to one of the first to thirty-fifth examples. The model may be configured to process digital data. The driver assistance system may be configured to control the vehicle based on the processed digital data. A vehicle having the features described in this paragraph forms a forty-sixth example.

[0056] The vehicle may have at least one sensor that is configured to provide digital data. The vehicle may also have a driver assistance system. The driver assistance system may have a second model trained according to the thirty-fourth example. The second model may be configured to process digital data. The driver assistance system may be configured to control the vehicle based on the processed digital data. A vehicle having the features described in this paragraph forms the forty-seventh example. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Exemplary embodiments of the invention are illustrated in the drawings and are explained in more detail in the following description.

[0058] in:

[0059] Figure 1 shows devices according to different embodiments;

[0060] Figure 2 An imaging device according to various embodiments is shown;

[0061] Figure 3 A processing system according to various embodiments is shown;

[0062] Figure 4 A processing system for training a model according to various embodiments is shown;

[0063] Figure 5 Methods for training a model according to various embodiments are shown;

[0064] Fig. 6A and 6B A processing system for training a GAN network according to various embodiments is shown;

[0065] Figure 7 shows a generator portion of a GAN network according to various embodiments;

[0066] Fig. 8A and 8B shows a generator portion of a GAN network according to various embodiments;

[0067] Fig.9A and 9B shows a generator portion of a GAN network according to various embodiments;

[0068] Fig.10 A vehicle according to various specific embodiments is shown. DETAILED DESCRIPTION

[0069] In one embodiment, a "circuit" may be understood as any type of logic implementation entity, which may be hardware, software, firmware, or a combination thereof. Thus, in one embodiment, a "circuit" may be a hardwired logic circuit or a programmable logic circuit, such as a programmable processor, for example a microprocessor (e.g., a CISC (complex instruction set processor) or a RISC (reduced instruction set processor)). A "circuit" may also be software implemented or implemented by a processor, such as any type of computer program, such as a computer program using a virtual machine code, such as Java. Consistent with an alternative embodiment, any other type of implementation of the corresponding functions may be understood as a "circuit", which are described in more detail below.

[0070] Various embodiments relate to an apparatus and method for training a model such that the trained model requires less memory requirements for internal calculations when processing digital data. That is, the trained model can process digital data with higher storage requirements, such as higher data quality or larger image size, while maintaining the same internal memory capacity.

[0071] Figure 1 A system 100 is shown according to different embodiments. The system 100 may have one or more sensors 102. The sensor 102 may be configured to provide digital data 104, that is, the digital data 104 may have data detected by the sensor. The sensor 102 may be: an imaging sensor, such as a camera sensor or a video sensor; or a remote position sensor, such as a radar sensor, a LIDAR sensor or an ultrasonic sensor; or a thermal sensor. According to different embodiments, the digital data 104 has digital image data. The digital data 104 may have driving dynamics sensor data of the vehicle, wherein the sensor 102 may be a speed sensor, an acceleration sensor or an angle sensor. The digital data 104 may have engine-specific sensor data of the vehicle, wherein the sensor 102 may be a sensor for detecting an accelerator pedal position, an injection amount, a temperature or a rotational speed. According to different embodiments, the sensor 102 has other types of sensors. The sensors in the plurality of sensors may have sensors of the same type or sensors of different types.

[0072] The system 100 may also have a storage device 106. The storage device 106 may have at least one memory. The memory may be used, for example, in the processing performed by the processor. The memory used in these embodiments may be: a volatile memory, such as a DRAM (dynamic random access memory); or a non-volatile memory, such as a PROM (programmable read-only memory), an EPROM (erasable PROM), an EEPROM (electrically erasable PROM); or a flash memory, such as a floating gate memory device, a charge trap memory device, an MRAM (magnetoresistive random access memory) or a PCRAM (phase change random access memory). The storage device 106 may be set up to store digital data 104. The system 100 may also have at least one processor 108. As described above, the processor 108 may be any type of circuit, that is, any type of logic implementation entity. The processor 108 may be a graphics processor (GPU), and the graphics processor may use the allocated graphics memory (video RAM) when performing a data processor. In different embodiments, the processor 108 is set up to process the digital data 104.

[0073] In the following, the embodiments are described with reference to digital images as digital data 104. However, it should be pointed out that other data types of (digital) data may also be used, such as point clouds or time series.

[0074] Figure 2 An imaging system 200 is shown according to various embodiments, wherein the sensor 102 is implemented as an imaging sensor 202. The imaging sensor 202 may be a camera sensor or a video sensor. The imaging sensor 202 may be configured to provide digital image data 204. The digital image data 204 may have at least one digital image 206, for example a plurality of digital images. Each digital data of the digital image 206 may have one of a plurality of image points or be formed by such image points. According to various embodiments, the imaging system 200 may have a plurality of imaging sensors. The imaging system 200 may also have a storage device 106 for storing the digital image data 204. The imaging system 200 may have at least one processor 108, which is configured to process the digital image data 204.

[0075] Figure 3A processing system 300 according to various embodiments is shown. The processing system 300 may have a storage device for storing digital image data 204, such as a digital image 206. The processing system 300 may also have at least one processor 108. The processor 108 may be configured to process the digital image 206. The processor 108 may use the storage device 106 when performing data processing. The processor 108 may use at least one memory of the storage device 106 when performing data processing, wherein the at least one memory used may be different from the memory used to store the digital image 206. The processor 108 may be configured to generate first input data 302 based on the digital image 206. The first input data 302 may be stored in the storage device 106. The first input data 302 may be generated by scaling down the digital image 206. The scaling down of the digital image 206 may include: reducing the resolution (e.g. from 2048x2048 pixels to 512x512 pixels, e.g. from 1024x1024 pixels to 256x256 pixels). The scaling or reduction of the resolution of the digital image 206 may include a scaling factor in the range of 4 to 8. According to different embodiments, the scaling of the height and the scaling of the width are reduced differently. In other words, the height of the digital image 206 is reduced based on the height scaling factor, while the width of the digital image 206 is reduced based on the width scaling factor. When the digital image 206 is scaled down, all global, that is to say large-scale information of the digital image 206 is retained in the first input data 302; however, detail information (that is to say local, small-scale information) may be lost.

[0076] The processor 108 implements at least a portion of the model 330. The processor 108 may implement at least a portion of the first submodel 304. The first submodel 304 may include a matrix multiplication, a principal component analysis, or a neural network. The first submodel 304 may be an autoencoder network. The first submodel 304 may be configured to process the first input data 302, that is, the scaled-down digital image 206. The first submodel 304 may be configured to generate the first submodel data 306 for the first input data 302 supplied to the first submodel 304. The first submodel data 306 may be stored in the storage device 106. The resolution of the first submodel data 306 may correspond to the resolution of the first input data 302. The processor 108 may be configured to process the first submodel data 306 and may also be configured to generate the first output data 308 based on the first submodel data 306. Each of the plurality of image points of the first output data 308 may be assigned exactly one image point of the digital image 206. The first output data 308 may be generated by scaling up the first submodel data 306. The upscaling of the first submodel data 306 may include: increasing the resolution (e.g. from 512x512 pixels to 2048x2048 pixels, e.g. from 256x256 pixels to 1024x1024 pixels). The upscaling may be based on any type of method capable of increasing the resolution. The upscaling may include interpolation. The upscaling may include a maximum likelihood method, such as super-resolution. An additional neural network or an additional neural subnetwork may be set up to upscale the submodel data, such as the first submodel data 306. The first output data 308 may be stored in the storage device 106. The calculation parameters of the first submodel 304 may be stored in at least one memory of the storage device 106, such as a graphics memory used by a graphics processor during processing. The calculation parameters of the first submodel 304 may include weights and / or activations. According to different embodiments, the first submodel 304 has a first neural subnetwork, wherein the weights and activations of the corresponding layers of the first neural subnetwork are stored in at least one memory of the storage device 106.

[0077] The processor 108 can also be set up to generate second input data 312 based on the digital image 206. The processor 108 can be set up to divide the digital image 206 into multiple data areas 310, such as multiple image areas 310a, 310b, 310c, 310d. The image areas of the multiple image areas 310a, 310b, 310c, 310d can overlap, that is, one image point in the multiple image points of the digital image 206 can be assigned to more than one image area. The multiple image areas 310a, 310b, 310c, 310d can have all the image points in the multiple image points of the digital image 206. These image areas can be generated based on a division key. That is, the digital image 206 can be divided into multiple image areas 310a, 310b, 310c, 310d based on the division key. The digital image 206 can be divided into 16 to 64 image areas, for example. Each of the plurality of image regions 310a, 310b, 310c, 310d may have the same resolution. The resolution of each of the plurality of image regions 310a, 310b, 310c, 310d may correspond to the resolution of the first input data 302. As described above, when the digital image 206 is scaled down, all global, that is, large-scale information of the digital image 206 is retained in the first input data 302. When the digital image 206 is divided into image regions, detail information, that is, local, small-scale information of the digital image 206 is retained in the first input data 302; however, global, large-scale information may be lost. The processor 108 may be configured to generate second input data 312 based on the plurality of image regions 310a, 310b, 310c, 310d, wherein the second input data 312 may have a heap of image regions. The division key may specify the order of the image regions in the heap of image regions. The second input data 312 may be stored in the storage device 106.

[0078] The processor 108 can implement at least a portion of the second submodel 314. The second submodel 314 can have a matrix multiplication, a principal component analysis, or a neural network. The second submodel 314 can be an autoencoder network. The second submodel 314 can be set up to process the second input data 312. The second submodel 314 can be set up to generate a corresponding output data area for the data area of ​​the second input data 312 that is fed to the second submodel 314. These output data areas can be stored in the storage device 106. The second submodel 314 can be set up to process each image area in a plurality of image areas 310a, 310b, 310c, 310d of the second input data 312 and generate one output data area in a plurality of output data areas based on the image area. In other words, each output data area can be assigned exactly one image area. The image areas in the plurality of image areas 310a, 310b, 310c, 310d can be processed in the order specified by the stack of image areas. The resolution of each of the plurality of output data areas may correspond to the resolution of the assigned image area of ​​the plurality of image areas 310a, 310b, 310c, 310d. The second submodel 314 may be configured to generate second submodel data 316 for the supplied second input data 312, wherein a corresponding output data area of ​​the plurality of output data areas is generated for each of the plurality of data areas. In other words, the second input data 312 may include each data area of ​​the digital image 206, and the second submodel 314 may generate a corresponding output data area for each data area. In the following, the embodiments are described based on the second submodel data 316, that is, the processing of each of the plurality of data areas (e.g., the plurality of image areas). However, it should be noted that only a portion of the data areas of the digital image 206 may also be processed. The calculation parameters of the second submodel 314 may be stored in at least one memory of the storage device 106 (e.g., a graphics memory used by a graphics processor during processing). The calculation parameters of the second submodel 314 may have weighting and / or activation. According to various specific embodiments, second sub-model 314 has a second neural sub-network, wherein weights and activations of corresponding layers of the second neural sub-network are stored in at least one memory of storage device 106 .

[0079] The processor 108 may also be configured to combine output data areas, such as the output data areas of the second sub-model data 316, into second output data 318. The plurality of output data areas may be combined based on a partitioning key. The plurality of output data areas may be combined based on an order specified by the stack of image areas and the partitioning key. The resolution of the second output data 318 may correspond to the resolution of the first output data 308. Each of the plurality of image points of the second output data 318 may be assigned exactly one image point of the digital image 206. The second output data 318 may be stored in the storage device 106.

[0080] Processor 108 may be configured to combine first output data 308 and second output data 318 into third output data 320. Third output data 320 may be generated by adding first output data 308 and second output data 318, for example, by weighted addition. Third output data 320 may be generated by adding each of the plurality of image points of the first output data to an assigned image point of the plurality of image points of the second output data (for example, by weighted addition). That is, the third output data has both large-scale information contained in first output data 308 and small-scale information contained in second output data 318. If radar sensor data is used instead of imaging sensor data, this has the following effect: both reflection data with small-scale information and distance-dependent noise characteristics of the radar sensor with large-scale information can be processed by the model; and the third output data has this information.

[0081] Figure 4A processing system 400 for training a model according to various embodiments is shown. The processing system 400 may essentially correspond to the processing system 300, wherein the processor 108 is further configured to train the first submodel 304 and the second submodel 314, that is to say to adapt the first submodel 304 and the second submodel 314. The processor 108 may be configured to train the first submodel 304 with respect to the digital data, such as the digital image 206, using the provided target data 402 and the first output data 308, which may be assigned to the digital data. The processor 108 may be configured to adapt the first submodel 304 by comparing the first output data 308 with the target data 402. The resolution of the first output data 308 may correspond to the resolution of the target data 402. The resolution of the target data 402 may correspond to the resolution of the digital image 206. The processor 108 may be configured to determine the first loss value 404 by comparing the first output data 308 with the target data 402. The first loss value 404 may be determined based on a loss function. The loss function may be any type of loss function, such as a per-image point (per-pixel) loss function (e.g., based on color value differences). The loss function may have a plurality of different loss functions or may be extended with these different loss functions. The loss function may have a KL (Kuhlback-Leibler) divergence. The processor 108 may be configured to adapt the first sub-model 304 based on the first loss value 404. The processor 108 may be configured to adapt the first sub-model 304 so that the first loss value 404 is minimized.

[0082] The processor 108 may also be configured to train the second sub-model 314 on digital data, such as the digital image 206, using the target data 402 and the third output data 320. The processor 108 may be configured to adapt the second sub-model 314 by comparing the third output data 320 with the target data 402. The resolution of the third output data 320 may correspond to the resolution of the target data 402. The processor 108 may be configured to determine a second loss value 406 by comparing the third output data 320 with the target data 402. The second loss value 406 may be determined based on a loss function. The loss function may be any type of loss function, such as a pixel-by-pixel loss function (e.g. based on color value differences). The loss function may have a plurality of different loss functions or may be extended from these different loss functions. The loss function may have a KL (Kuhlback-Leibler) divergence. The processor 108 may be configured to adapt the second sub-model 314 based on the second loss value 406. The processor 108 may be configured to adapt the second sub-model 314 such that the second loss value 406 is minimized. The target data 402 may correspond to the digital image 206 .

[0083] According to different embodiments, the processor 108 also implements at least a part of the third submodel. The processor 108 can be set up to divide the digital image 206 into a plurality of second image areas, wherein the plurality of second image areas can be different from the plurality of image areas 310a, 310b, 310c, 310d. The resolution of the plurality of second image areas can be less than the resolution of the plurality of image areas 310a, 310b, 310c, 310d. The processing of the plurality of second image areas can substantially correspond to the processing of the plurality of image areas 310a, 310b, 310c, 310d. The processor can be set up to generate third input data based on the plurality of second image areas. The third submodel can be set up to process the third input data and generate third submodel data for the third input data conveyed, wherein for each second image area in the plurality of second image areas, the third submodel data can have one second output data area in the plurality of second output data areas. The processor 108 can be set up to combine the plurality of second output data areas into fourth output data. The processor 108 may be configured to generate fifth output data by merging the third output data 320 and the fourth output data. The processor 108 may be configured to train the third sub-model on the digital data (e.g., the digital image 206) using the target data and the fifth output data. As already explained above, the processing system 400 may have any number of additional sub-models, wherein each of these additional sub-models may process a plurality of image regions, wherein the resolution of the plurality of image regions of each sub-model is smaller than the resolution of the plurality of image regions of the respective preceding sub-model. In other words, each additional sub-model processes an image region with a lower resolution among the plurality of image regions.

[0084] Figure 5A method 500 for training a model according to various embodiments is shown. The method 500 may include scaling down digital data so that first input data 302 are generated (in 502). The method 500 may include dividing the digital data into a plurality of data regions so that second input data 312 are generated (in 504). The plurality of data regions 310 of digital data may be, for example, a plurality of image regions 310a, 310b, 310c, 310d of a digital image 206. The method 500 may also include generating first sub-model data 306 by means of a first sub-model 304 for the first input data 302 supplied to the first sub-model 304 (in 506). The method 500 may include scaling up the first sub-model data 306 to first output data 308 (in 508). In the method 500, the second sub-model 314 may generate output data regions corresponding to these data regions for the second input data 312 supplied to the second sub-model 314 (in 510). The method 500 may have, at 512, combining the output data regions into the second output data 318. The method 500 may have, at 514, generating the third output data 320. The first output data 308 and the second output data 318 may be combined into the third output data 320. The method 500 may also have, at 516, training the first sub-model 304. The first sub-model 304 may be trained on the digital data by comparing the provided target data 402 assignable to the digital data with the first output data 308. The method 500 may have, at 518, training the second sub-model 314. The second sub-model 314 may be trained on the digital data by comparing the target data 402 with the third output data 320.

[0085] Fig. 6A and Figure 6B A processing system 600 for training a GAN (Generative Adversarial Network) according to various embodiments is shown. Fig. 6AAs shown in , the processing system 600 may have a storage device 106 for storing digital images 206. The processing system 600 may have at least one processor 108. The processor 108 may implement a GAN 602. The GAN 602 may have a generator part 604. The generator part 604 may have a first generator part 606 and a second generator part 608. The first generator part 606 may be set up to process the digital data 104 of the first domain and generate the digital data 104 of the second domain. The first generator part 606 may be set up to process the digital image 206, wherein the digital image 206 may belong to the first domain. The first domain may be different from the second domain. The digital images assigned to a common domain (e.g., the first domain) have a common (e.g., professional) field, a common category, or a common feature. In other words, the digital images assigned to a common domain may be assigned to a common problem range. For example, the domain may be different in terms of weather, wherein the first domain is, for example, a scene on a sunny day and the second domain is, for example, the same scene on a rainy day. For example, the domains may differ with respect to seasons, wherein a first domain is, for example, a scene in summer and a second domain is, for example, the same scene in winter.

[0086] The first generator portion 606 may be configured to generate a second digital image 610 based on the digital image 206, wherein the second digital image 610 may belong to the second domain. The second generator portion 608 may be configured to process the digital data 104 of the second domain and generate the digital data 104 of the first domain. The second generator portion 608 may be configured to process the second digital image 610 and generate a third digital image 612 based on the second digital image 610. The GAN 602 may also have a first discriminator 614 and a second discriminator 618. The first discriminator 614 may be configured to adapt the GAN 602 based on the digital image 206 and the third digital image 612. As in Figure 6B As shown in , the storage device 106 may also store a fourth digital image 620, wherein the fourth digital image 620 may belong to the second domain. The second generator portion 608 may generate a fifth digital image 622 based on the fourth digital image 620, wherein the fifth digital image 622 may belong to the first domain. The first generator portion 606 may generate a sixth digital image 624 based on the fifth digital image 622, wherein the sixth digital image 624 may belong to the second domain. The second discriminator 618 may be set up to make the GAN 602 adaptive based on the fourth digital image 620 and the sixth digital image 624.

[0087] Figure 7A generator portion 700 of a GAN network according to various embodiments is shown. The generator portion 700 may substantially correspond to the generator portion 604 of the processing system 600, wherein the first generator portion 606 has the first sub-model 304 and the second sub-model 314.

[0088] In the following, the embodiments are described with reference to a neural network as model 330. However, it should be pointed out that other models may also be used, for example models with matrix multiplication or principal component analysis.

[0089] The first sub-model 304 may be a first neural sub-network and may have a first encoder portion 704 and a first decoder portion 708. The first encoder portion 704 may be configured to process a scaled-down version of the digital data of the first domain 740 and output a code in the first latent space 706. The first decoder portion 708 may be configured to process a code generated by the first encoder portion 704 and present in the first latent space 706 and generate the first sub-model data 306 based on the code. The second sub-model 314 may be a second neural sub-network and may have a second encoder portion 710 and a second decoder portion 714. The second encoder portion 710 may be configured to process a data region of the digital data of the first domain 740 and output a code in the second latent space 712. The second decoder portion 714 may be configured to process a code generated by the second encoder portion 710 and present in the second latent space 712 and generate the second sub-model data 316 based on the code. The processor 108 may be configured to generate the third output data 320 based on the first sub-model data 306 and the second sub-model data 316 as described in conjunction with the processing system 300. The third output data 320 may be digital data of the second domain 750.

[0090] The second generator part 608 may have a third neural sub-network 720 and a fourth neural sub-network 722. The third neural sub-network 720 may substantially correspond to the first neural sub-network, that is, the first sub-model 304, wherein the third encoder part 724 may be configured to process a scaled-down version of the digital data of the second domain 750 and output a code in the third latent space 726. The third decoder part 728 may be configured to process the code generated by the third encoder part 724 and present in the third latent space 726 and to generate third sub-model data based on the code. The fourth neural sub-network 722 may substantially correspond to the second neural sub-network, that is, the second sub-model 314, wherein the fourth encoder part 730 may be configured to process a data region of the digital data of the second domain 750 and output a code in the fourth latent space 732. The fourth decoder part 734 may be configured to process the code generated by the fourth encoder part 730 and present in the fourth latent space 732 and to generate fourth sub-model data based on the code. The processor 108 may be configured to generate digital data of the first domain 740 based on the third sub-model data and the fourth sub-model data.

[0091] Fig. 8A and 8B A generator portion 800 of a GAN network according to various embodiments is shown. The generator portion 800 may substantially correspond to the generator portion 700, wherein the first encoder portion 704 and the third encoder portion 724 are mapped into the first shared latent space 802. In other words, the first encoder portion 704 and the third encoder portion 724 output codes in the first shared latent space 802. Fig. 8A The processing of the digital data of the first domain 740 shown in FIG. 7 may substantially correspond to the processing of the digital data of the first domain 740 shown in FIG. Figure 7 802, and wherein the first decoder portion 708 processes the codes present in the first shared latent space. Figure 8B The processing of the digital data of the second domain 750 shown in FIG. 7 may substantially correspond to the processing of the digital data of the second domain 750 shown in FIG. Figure 7 , and described in this context, wherein the third encoder portion 724 outputs codes in the first shared latent space 802, and wherein the third decoder portion 728 processes the codes present in the first shared latent space.

[0092] Fig.9A and 9BA generator portion 900 of a GAN network according to various embodiments is shown. Generator portion 900 may substantially correspond to generator portion 800, wherein second encoder portion 710 and fourth encoder portion 730 are mapped into second shared latent space 902. In other words, second encoder portion 710 and fourth encoder portion 730 output codes in second shared latent space 902. That is, first encoder portion 704 and third encoder portion 724 are mapped into first shared latent space 802, while second encoder portion 710 and fourth encoder portion 730 are mapped into second shared latent space 902. Fig.9A The processing of the digital data of the first domain 740 shown in FIG. 7 may substantially correspond to the processing of the digital data of the first domain 740 shown in FIG. Fig. 8A The processing of digital data of the first domain 740 shown in and described in this context, wherein the second encoder portion 710 outputs codes in the second shared latent space 902, and wherein the second decoder portion 714 processes the codes present in the second shared latent space. Fig. 9B The processing of the digital data of the second domain 750 shown in FIG. 7 may substantially correspond to the processing of the digital data of the second domain 750 shown in FIG. Figure 8B , and described in this context, wherein the fourth encoder portion 730 outputs codes in the second shared latent space 902, and wherein the fourth decoder portion 734 processes the codes present in the second shared latent space.

[0093] Fig.10 A vehicle 1000 according to one embodiment is shown. The vehicle 1000 may be a vehicle with an internal combustion engine, an electric vehicle, a hybrid vehicle or a combination thereof. The vehicle 1000 may also be a car, a truck (LKW), a ship, an unmanned aerial vehicle, an airplane, etc. The vehicle 1000 may have at least one sensor 1002, such as an imaging sensor, a speed sensor, an angle sensor, a temperature sensor or a sensor for detecting an accelerator pedal position, an injection amount or a rotational speed (e.g., sensor 102). The vehicle 1000 may have a driver assistance system 1004. The driver assistance system 1004 may have a storage device 106. The driver assistance system 1004 may have a processor 108. The processor 108 may implement a model 330, which is implemented as a neural network, for example. The driver assistance system 1004 may have a processing system ( 300 , 400 , 600 ), wherein the processing system ( 300 , 400 , 600 ) may process the digital data 104 and wherein the processing system ( 300 , 400 , 600 ) may provide third output data 320 .

[0094] According to different embodiments, the model 330 is trained according to the method 500 for training the model, so that the trained model requires less memory requirements for internal calculations in the process of processing digital data. That is, the trained model 330 can also process digital data with higher storage requirements, such as higher data quality or larger image size. Therefore, one aspect is to provide a vehicle that can process digital data with higher storage requirements. The driving assistance system 1004 can be set up to control the vehicle 1000 based on the processed digital data. In other words, the driving assistance system 1004 can be set up to process the digital data 104 and can output at least one control instruction to one or more actuators of the vehicle 1000 based on the third output data 320. That is, the driving assistance system 1004 can affect the current driving behavior based on the processed digital data, for example, it can maintain or change the current driving behavior. The change to the driving behavior can be, for example, an intervention in the driving behavior for safety reasons, such as emergency braking.

[0095] According to different embodiments, the digital data includes digital image data, which is provided by an imaging sensor, such as a camera sensor or a video sensor. The driver assistance system 1004 may include a segmentation network. The segmentation network may be configured to process the digital image data output by the processing system (300, 400, 600) based on the digital image data and generate segmented image data. The digital image data may include at least one object, and the segmented image data may include segmentation of the at least one object. The driver assistance system 1004 may be configured to influence the current driving behavior based on the segmented image data. For example, the imaging sensor may provide a high-resolution image that shows a pedestrian in front of the vehicle 1000 at night. The processing system (300, 400, 600) having the model 330 may be configured to process high-resolution images, the segmentation network may segment the image generated by the processing system (300, 400, 600), and the driver assistance system 1004 may be configured to control the vehicle based on the segmented image.

[0096] According to different embodiments, model 330 can be implemented as a generator part (700, 800, 900) of a GAN. The GAN can be set up to process digital image data provided by an imaging sensor, wherein the digital image data belongs to a first domain, such as a road scene on a sunny day. The GAN can also be set up to generate digital image data belonging to a second domain, such as a road scene when it rains. The driving assistance system 1004 may have an additional model. The additional model can be trained using the digital image data generated by the GAN. In other words, the additional model can be trained using the digital image data belonging to the second domain. The trained additional model can be set up to output digital image data, and the driving assistance system 1004 can be set up to control the vehicle based on the digital image data output by the trained additional model.

Claims

1. A method for training a model in a computer-implemented manner, the model having a first sub-model (304) and a second sub-model (314): • wherein the digital data (104) is scaled down such that first input data (302) is generated; • wherein said digital data (104) is divided into a plurality of data areas (310) such that second input data (312) is generated; • wherein the first sub-model (304) generates first sub-model data (306) for first input data (302) supplied to the first sub-model (304); • wherein the first sub-model data (306) is scaled up into first output data (308); • wherein the second sub-model (314) generates an output data area (316) corresponding to the data area (310) for second input data (312) supplied to the second sub-model (314); • wherein said output data areas (316) are combined into second output data (318); • wherein the first output data (308) and the second output data (318) are combined into third output data (320); • wherein the first sub-model (304) is trained by comparison of provided target data (402) assigned to the digital data (104) with the first output data (308); and • wherein the second sub-model (314) is trained by comparing the target data (402) with the third output data (320).

2. The method according to claim 1, wherein the first sub-model (304) generates first sub-model data (306) for the supplied first input data (302), the first sub-model data having the same resolution as the first input data (302).

3. The method according to claim 1 or 2, wherein the first sub-model (304) has a first autoencoder.

4. The method according to claim 1 , wherein the second sub-model ( 314 ) generates second sub-model data for the supplied second input data ( 312 ), wherein for each data area in the plurality of data areas ( 310 ), a corresponding output data area in the plurality of output data areas ( 316 ) is generated.

5. The method according to any one of claims 1 to 4, wherein the second sub-model (314) has a second autoencoder.

6. The method according to any one of claims 1 to 5, wherein the model is a generative model, wherein the digital data (104) belongs to a first domain and wherein the third output data (320) belongs to a second domain, the method further comprising: • Generate training data through the trained model; • Train a second model using the generated training data.

7. The method according to claim 6, further comprising: • processing second digital data by means of the second model in the driver assistance system, the second digital data belonging to the second domain; and • Controlling the vehicle based on the processed second digital data. 8 . A training device which is configured to carry out the method according to claim 1 .

9. A vehicle comprising: at least one sensor configured to provide digital data (104); and A driver assistance system having a model trained according to any one of claims 1 to 5, wherein the model is configured to process the digital data (104), and wherein the driver assistance system is configured to control the vehicle based on the processed digital data.

10. A vehicle comprising: at least one sensor configured to provide second digital data, wherein the second digital data belongs to a second domain; and A driver assistance system having a second model trained according to claim 7, wherein the second model is configured to process the second digital data, and wherein the driver assistance system is configured to control the vehicle based on the processed digital data.

Images