Method and apparatus for training a neural network

Abstract

A method and apparatus for training a neural network are disclosed, wherein the method for training a neural network comprises: training a first sub-neural network with first digital training data (302) describing a first context, wherein the first sub-neural network is set up as an autoencoder network and comprises a first encoder section (306) and a first decoder section, and wherein the first encoder section (306) provides a mapping of the first digital training data (302) to a first latent space (308); training a first mapping (404) of first digital data (402) to the first latent space (308) using first digital training data (302) mapped into the first latent space (308) by means of the trained first sub-neural network, wherein the first digital data is semantically associated with the first digital training data (302); training a second sub-neural network with second digital training data (322) describing a second context, wherein the second context is different from the first context, wherein the second sub-neural network is set up as an autoencoder network and comprises a second encoder section and a second decoder section (330), and wherein the second encoder section provides a mapping of the second digital training data (322) to a second latent space; training a second mapping (424) of second digital data (422) to the second latent space (328) using second digital training data (322) mapped into the second latent space (328) by means of the trained second sub-neural network, wherein the second digital data is semantically associated with the second digital training data (322); training a third mapping (502) of digital latent data from the first latent space (308) to the second latent space (328) using third digital training data and third digital data, wherein the third digital training data comprises: digital training data describing the first context, and digital training data describing the second context, wherein the third digital data comprises: digital data semantically associated with digital training data describing the first context; and digital data semantically associated with digital training data describing the second context.

Description

TECHNICAL FIELD

[0001] Different embodiments generally relate to a method and a device for training a neural network. BACKGROUND

[0002] Different neural networks are used, for example, in order to generate output data on the basis of input data and a function that has been learned by training a neural network. The desired output data can depend on a context in which the input data has been generated. It can therefore be necessary to transform data that describes a first context into data that describes a second context. This context is meaningful, for example, in the case of recognizing the intention of a traffic participant.

[0003] A method for transforming video data by means of a neural network is described in Suwajanakorn et al., “Synthesizing Obama: Learning Lip Sync from Audio,” ACM Transactions on Graphics, Vol. 36, No. 4, 2017. SUMMARY

[0004] The method according to the invention and the device according to the invention achieve that a neural network is trained in order to transform digital data from a first context to a second context.

[0005] The context of the digital data, for example the context of the first and second digital training data, can differ in that the digital data has been generated in which context, that is to say in which interrelationships or under which boundary conditions, and / or in which inherent context of the digital data, for example in which environment is described by the digital data. In different embodiments, the context can be distinguished territorially, that is to say the first and second context. For example, the context can differ in respect of urban areas, regions, countries, etc. The context can be distinguished in respect of language and / or in respect of expressions and gestures that are associated with the language and / or the region / country. The context can differ culturally, that is to say the context can be distinguished, for example, territorially, in respect of language, expressions, gestures, etc. According to one example, the digital data is digital image data and the context can be distinguished territorially in that the digital data has been generated in different countries; and / or the context can be distinguished in respect of the inherent in that the digital images are distinguished in respect of expressions and gestures in respect of text that is spoken by a person (that is to say the digital image data is also distinguished in respect of facial movements that are related to the language).

[0006] The language can also be different, for example, in case of digital data that is semantically associated with the digital data describing the first context or the second context. The digital data that is semantically associated with the digital data describing the first context or the second context can be, for example, text data that has a plurality of text strings and the semantic association can describe that exactly one of the plurality of text strings is assigned to each of the digital data describing the first context or the second context. In other words, the digital data describing the context is assigned digital text data and made to describe the digital data. That is: the digital text data can intuitively describe the content of the digital data. The digital text data can have additional information about the digital data. For example, the digital data can be digital image data that shows a scene and the digital text data can describe the scene.

[0007] The first mapping, the second mapping and the third mapping can have sub- neural networks. Each of these sub-neural networks can be an arbitrary neural network, for example an autoencoder network or a convolutional neural network. Each sub- neural network, i.e. the first sub-neural network and the second sub-neural network, can have an arbitrary number of layers and can be trained according to an arbitrary method, such as by Backpropagation. Each encoder section of the autoencoder network can have an arbitrary number of encoder layers, wherein each encoder layer can comprise a convolutional layer with arbitrary properties, for example arbitrary filter parameters, an activation function, for example a ReLU activation function, a pooling layer with arbitrary properties, for example a max-pooling layer with arbitrary strides, and a normalization layer. Each decoder section of the autoencoder network can have an arbitrary number of decoder layers, wherein each decoder layer can comprise a transposed convolutional layer with arbitrary properties, a convolutional layer with arbitrary properties, an activation function, for example a ReLU activation function, and a normalization layer.

[0008] At least a part of the first sub-neural network can be implemented by one or more processors. At least a part of the first mapping can be implemented by one or more processors. At least a part of the second sub-neural network can be implemented by one or more processors. At least a part of the second mapping can be implemented by one or more processors. At least a part of the third mapping can be implemented by one or more processors. The features described in this paragraph constitute a second example in combination with the first example.

[0009] The first and second digital training data can have digital image data. The features described in this paragraph in combination with the first or second example constitute a third example.

[0010] The first sub-neural network can be trained in a manner that the first decoder section reconstructs the first digital training data mapped to the first latent space by the first encoder section and compares the reconstructed first digital training data with the first digital training data. The features described in this paragraph in combination with one or more of the first to third examples constitute a fourth example.

[0011] The comparison of the reconstructed first digital training data with the first digital training data can comprise determining a first loss value. The first loss value can be determined based on a loss function. The features described in this paragraph in combination with the fourth example constitute a fifth example.

[0012] The training of the first sub-neural network can comprise adapting the first encoder section and the first decoder section, wherein adapting the first encoder section and the first decoder section can comprise minimizing the first loss value. That is: the first encoder section of the trained first sub-neural network can output a code that describes the following digital data in the first latent space, the digital data describing a first context; and the first decoder section of the trained first sub-neural network can process the code that describes the digital data describing the first context in the first latent space and output the digital data based on the code. The features described in this paragraph in combination with the fifth example constitute a sixth example.

[0013] The second sub-neural network can be trained in a manner that the second decoder section reconstructs the second digital training data mapped to the second latent space by the second encoder section and compares the reconstructed second digital training data with the second digital training data. The features described in this paragraph in combination with one or more of the first to sixth examples constitute a seventh example.

[0014] The comparison of the reconstructed second digital training data with the second digital training data can comprise determining a second loss value. The second loss value can be determined based on a loss function. The features described in this paragraph in combination with the seventh example constitute an eighth example.

[0015] The training of the second sub-neural network can comprise adapting the second encoder section and the second decoder section, wherein adapting the second encoder section and the second decoder section can comprise minimizing a second loss value. That is: the second encoder section of the trained second sub-neural network can output a code that describes the following digital data in the second latent space, the digital data describing the second context; and the second decoder section of the trained second sub-neural network can process the code that describes the digital data describing the second context in the second latent space and output the digital data based on the code, the features described in this paragraph in combination with the eighth example constituting a ninth example.

[0016] The first mapping can have a third sub-neural network and the second mapping can have a fourth sub-neural network. The features described in this paragraph in combination with one or more of the first to ninth examples constitute a tenth example.

[0017] The first digital data and the second digital data can have a plurality of text strings, wherein the text strings describe the assigned first digital training data or the assigned second digital training data. That is: each of the first digital training data can be assigned to exactly one of the plurality of text strings of the first digital data, and each of the second digital training data can be assigned to exactly one of the plurality of text strings of the second digital data. For example, the first digital training data or the second digital training data can have digital image data that shows a scene, and the first digital data or the second digital data can have a plurality of text strings that describe the shown scene, respectively. The features described in this paragraph in combination with one or more of the first to tenth examples constitute an eleventh example.

[0018] Training the first mapping can comprise comparing the code output by the first encoder section based on the first digital training data with the code output by the first mapping based on the first digital data. The features described in this paragraph in combination with one or more of the first to eleventh examples constitute a twelfth example.

[0019] Comparing the code output by the first encoder section based on the first digital training data with the code output by the first mapping based on the first digital data can comprise determining a first mapping loss value. The first mapping loss value can be determined based on a loss function. The features described in this paragraph in combination with the twelfth example constitute a thirteenth example.

[0020] Training the first mapping can comprise adapting the first mapping, wherein adapting the first mapping can comprise minimizing a first mapping loss value. That is: the trained first mapping can output codes that describe the following digital text data in the first latent space, the digital text data describing a first context, wherein the codes describing the digital text data can be assigned to codes describing digital data in the first latent space. The features described in this paragraph constitute a fourteenth example in combination with the thirteenth example.

[0021] At least a portion of the first digital data can be provided by an additional first neural network, wherein the additional first neural network can process at least a portion of the first digital training data. The features described in this paragraph constitute a fifteenth example in combination with one or more of the first to fourteenth examples.

[0022] Training the second mapping can comprise comparing the codes output by the second encoder section based on the second digital training data with the codes output by the second mapping based on the second digital data. The features described in this paragraph constitute a sixteenth example in combination with one or more of the first to fifteenth examples.

[0023] Comparing the codes output by the second encoder section based on the second digital training data with the codes output by the second mapping based on the second digital data can comprise determining a second mapping loss value. The second mapping loss value can be determined based on a loss function. The features described in this paragraph constitute a seventeenth example in combination with the sixteenth example.

[0024] Training the second mapping can comprise adapting the second mapping, wherein adapting the second mapping can comprise minimizing a second mapping loss value. That is: the trained second mapping can output codes that describe the following digital text data in the second latent space, the digital text data describing a second context, wherein the codes describing the digital text data can be assigned to codes describing digital data in the second latent space. The features described in this paragraph constitute an eighteenth example in combination with the seventeenth example.

[0025] At least a portion of the second digital data can be provided by an additional second neural network, wherein the additional second neural network can process at least a portion of the second digital training data. The features described in this paragraph constitute a nineteenth example in combination with one or more of the first to eighteenth examples.

[0026] The third mapping can have a fifth sub-neural network. The features described in this paragraph constitute a twentieth example in combination with one or more of the first to nineteenth examples.

[0027] The digital training data in the third digital training data that describes the first context can comprise at least a subset of the first digital training data (e.g., all of the first digital training data), and the digital data in the third digital training data that is semantically associated with the digital training data can comprise a subset of the first digital data that is assigned to the subset of the first digital training data. The features described in this paragraph, in combination with one or more of the first through twentieth examples, constitute a twenty-first example.

[0028] The digital training data in the third digital training data that describes the second context can comprise at least a subset of the second digital training data (e.g., all of the second digital training data), and the digital data in the third digital training data that is semantically associated with the digital training data can have a subset of the second digital data that is assigned to the subset of the second digital training data. The features described in this paragraph, in combination with one or more of the first through twenty-first examples, constitute a twenty-second example.

[0029] The third mapping can process: codes in the first latent space that describe digital training data that describes the first context; codes in the first latent space that describe digital data that is semantically associated with the digital training data that describes the first context; and codes in the second latent space that describe digital data that is semantically associated with digital training data that describes the second context, and the third mapping can output codes that describe digital training data in the second latent space. The features described in this paragraph, in combination with one or more of the first through twenty-second examples, constitute a twenty-third example.

[0030] The second decoder segment can process codes that describe digital training data in the second latent space and can output reconstructed digital training data that describes the second context. The features described in this paragraph, in combination with one or more of the first through twenty-third examples, constitute a twenty-fourth example.

[0031] Training the third mapping can include comparing the reconstructed digital training data that describes the second context to digital training data that describes the second context. The features described in this paragraph, in combination with one or more of the first through twenty-fourth examples, constitute a twenty-fifth example.

[0032] The comparing of the reconstructed digital training data that describes the second context to the digital training data that describes the second context can include determining a third mapping loss value. The third mapping loss value can be determined based on a loss function. The features described in this paragraph, in combination with the twenty-fifth example, constitute a twenty-sixth example.

[0033] Training the third mapping can comprise adapting the third mapping, wherein adapting the third mapping can comprise minimizing the third mapping loss value. This has the effect that the trained third mapping is capable of processing: codes in the first latent space that describe digital training data that describes the first context; codes in the first latent space that describe digital data that is semantically related to the digital training data that describes the first context; and codes in the second latent space that describe digital data that is semantically related to digital training data that describes the second context, and that the trained third mapping can output digital training data that describes the second context. The features described in this paragraph constitute a twenty-seventh example in combination with the twenty-sixth example.

[0034] The first transformation network can have: a first encoder section of the trained first sub- neural network; a second decoder section of the trained second sub-neural network; the trained first mapping; the trained second mapping; and the trained third mapping. The first transformation network can process: digital data that describes the first context; digital text data that describes the first context and that is assigned to the digital data that describes the first context; and digital text data that describes the second context, and the first transformation network can output digital data that describes the second context. That is, the first transformation network is capable of transforming digital data that describes the first context into digital data that describes the second context. This has the advantage that if digital data that describes the first context is present on a large scale and if digital data that describes the second context is not present on a large scale, it is possible to generate digital data that describes the second context on the basis of digital data that describes the first context. In other words, it is possible to generate digital data for the second context such that digital data is present on a large scale not only for the first context but also for the second context. The features described in this paragraph constitute a twenty-eighth example in combination with one or more of the first to twenty-seventh examples.

[0035] The second transformation network can have a second encoder section of the trained second sub-neural network, a first decoder section of the trained first sub-neural network, the trained first mapping, the trained second mapping, and an inverse mapping of the trained third mapping. The second transformation network can process digital data describing the second context, digital textual data describing the second context and assigned to the digital data describing the second context, and digital textual data describing the first context, and the second transformation network can output digital data describing the first context. That is, the second transformation network is capable of transforming digital data describing the second context into digital data describing the first context. This has the advantage that digital data can be adapted to the respective context, such that the digital data can be further processed based on the context in which the digital data was generated or based on the inherent context of the digital data. The features described in this paragraph constitute a twenty-ninth example in combination with one or more of the first to twenty-eighth examples.

[0036] The computer program can have program commands which are set up to perform the method according to one or more of the first to twenty-ninth examples when executed by one or more processors. The features described in this paragraph constitute a thirtieth example.

[0037] The computer program can be stored in a machine-readable storage medium. The features described in this paragraph constitute a thirty-first example in combination with the thirtieth example.

[0038] At least a part of the first sub-neural network can be implemented by one or more processors. At least a part of the first mapping can be implemented by one or more processors. At least a part of the second sub-neural network can be implemented by one or more processors. At least a part of the second mapping can be implemented by one or more processors. At least a part of the third mapping can be implemented by one or more processors. The features described in this paragraph constitute a thirty-third example in combination with the thirty-second example.

[0039] The system can have a device according to the thirty-second or thirty-third example. The system can have a sensor, e.g. an imaging sensor, which is set up to provide digital data describing the first or the second context. The features described in this paragraph constitute a thirty-fourth example.

[0040] The system can also have an additional neural network which is set up to generate digital text data having a text string which represents the digital data describing the first or second context based on the digital data describing the first context or the second context. The features described in this paragraph constitute a thirty-fifth example in combination with the thirty-fourth example.

[0041] The imaging sensor can be a camera sensor or a video sensor. The imaging sensor can be a remote sensing sensor such as a radar sensor, a LIDAR sensor or an ultrasound sensor which provides image data by processing sensor signals by means of an imaging method. The features described in this paragraph constitute a thirty-sixth example in combination with the thirty-fourth or the thirty-fifth example.

[0042] The transportation vehicle can have a driving assistance system. The driving assistance system can have a system according to one or more of the thirty-fourth to thirty-sixth examples. The features described in this paragraph constitute a thirty-seventh example.

[0043] The transportation vehicle can have at least one imaging sensor or remote sensing sensor which is set up to provide digital image data. The transportation vehicle can also have a driving assistance system. The driving assistance system can have a first neural transform network according to the twenty-eighth example and / or a second neural transform network according to the twenty-ninth example. The driving assistance system can also be set up to classify and / or segment the digital data output by the first neural transform network or the second neural transform network. The driving assistance system can be set up to control the transportation vehicle based on the classified and / or segmented digital data. That is to say: the driving assistance system can be set up to process the classified and / or segmented digital data and can output at least one control instruction based on the classified and / or segmented digital data. This has the advantage that the driving assistance system is able to influence the driving behavior based on the context of the digital data. For example, the driving assistance system can recognize the intention of a traffic participant based on the context and influence the driving behavior (for example change the driving behavior, for example maintain the driving behavior) in correspondence therewith. The features described in this paragraph constitute a thirty-eighth example. BRIEF DESCRIPTION OF DRAWINGS

[0044] Embodiments of the invention are illustrated in the accompanying drawings and further described in the following description. In the drawings:

[0045] Figure 1 Devices according to different embodiments are shown;

[0046] Figure 2 Imaging devices according to different embodiments are shown;

[0047] Figure 3A processing system for training a first sub-neural network according to different embodiments is shown;

[0048] Figure 3B processing system for training a second sub-neural network according to different embodiments is shown;

[0049] Figure 4A processing system for training a first mapping according to different embodiments is shown;

[0050] Figure 4B processing system for training a second mapping according to different embodiments is shown;

[0051] Figure 5 processing system for training a third mapping according to different embodiments is shown;

[0052] Figure 6 a method for training a neural network according to different embodiments is shown;

[0053] Figure 7A a first processing system for transforming digital data between different contexts according to different embodiments is shown;

[0054] Figure 7B a second processing system for transforming digital data between different contexts according to different embodiments is shown; and Figure 8 a means of transportation according to different embodiments is shown. DETAILED DESCRIPTION

[0055] In one embodiment, a "circuit" can be understood as every type of entity implementing logic, which can be hardware, software, firmware, or a combination thereof. Thus, in one embodiment, a "circuit" can be a hard-wired logic circuit or a programmable logic circuit, such as a programmable processor, such as a microprocessor (e.g. a CISC (Complex Instruction Set Computer) or RISC (Reduced Instruction Set Computer) processor). A "circuit" can also be software implemented or executed by a processor, such as every type of computer program, e.g. using virtual machine code, such as a computer program in Java. Every other implementation of the respective functionality described in more detail below can be understood as a "circuit" in accordance with alternative embodiments.

[0056] The different embodiments illustrate intuitively a method for training a neural network such that the trained neural network can transform digital data, such as digital image data, from a first context to a second context. In other words, the digital data can have context-specific properties, such as country-specific properties, and the trained neural network can convert the digital data to another context.

[0057] Figure 1 A device 100 according to different embodiments is shown. The device 100 can have one or more sensors 102. The sensor 102 can be set up to provide digital data 104. The sensor 102 can be an imaging sensor, such as a camera sensor or a video sensor, or a remote probing sensor, such as a radar sensor, a LIDAR sensor or an ultrasound sensor. According to different embodiments, the sensor 102 can have other types of sensors. According to different embodiments, the digital data 104 has digital image data (detected radar sensor signals, LIDAR sensor signals and ultrasound sensor signals that have been processed by means of imaging methods will also be understood as digital image data in the context of the specification). The sensors of the plurality of sensors can have the same type or different types of sensors.

[0058] The device 100 can also have a storage device 106. The storage device 106 can have a memory. The memory can be used, for example, in the processing by the processor. The memory used in the embodiments can be a volatile memory, such as a DRAM (Dynamic Random Access Memory), or a non-volatile memory, such as a PROM (Programmable Read-Only Memory), an EPROM (Erasable PROM), an EEPROM (Electrically Erasable PROM) or a flash memory, such as a memory device with a floating gate, a memory device that traps electric charges, an MRAM (Magnetic Resistance Direct Access Memory) or a PCRAM (Phase Change Direct Access Memory). The storage device 106 can be set up to store the digital data 104. The device 100 can also have at least one processor 108 (for example exactly one processor, for example two processors, for example more than two processors). The at least one processor 108 can be each type of circuit, i.e. each type of entity that implements logic, as described above. In different embodiments, the at least one processor 108 is set up to process the digital data 104.

[0059] In the following, the embodiments are described in terms of digital image data 204 as digital data 104. However, it should be noted that other context-dependent (digital) data, such as each type of digital sensor data, can also be used.

[0060] Figure 2An imaging device 200 according to different embodiments is shown, wherein the sensor is implemented as an imaging sensor 202. The imaging sensor 202 can be a camera sensor or a video sensor. The imaging sensor 202 can be set up to provide digital image data 204. Within the scope of the present specification, radar sensors, LIDAR sensors and ultrasound sensors that are also set up to provide digital image data 204 are to be understood as imaging sensors 202. These digital image data 204 can have a plurality of digital images 206. The plurality of digital images 206 can show a scene in a respective context. According to different embodiments, the imaging device 200 has a plurality of imaging sensors.

[0061] Figure 3AA processing system 300A for training a first sub-neural network according to different embodiments is shown. The processing system 300A can have a storage device 106 for storing digital image data 204, such as first digital training data 302. The first digital training data 302 can describe a first context. The processing system 300A can further have at least one processor 108. The processor 108 implements at least a part of a first sub-neural network 304. The first sub-neural network 304 is set up for processing the first digital training data 302. The first sub-neural network 304 can be an autoencoder network. The first sub-neural network 304 can have a first encoder section 306. The first encoder section 306 can have at least one encoder and can be set up for showing features of the first digital training data 302 in a first latent space 308, i.e. in a lower dimension than the first digital training data 302. In other words, the first encoder section 306 can output a code having a lower dimension than the first digital training data 302. The first sub-neural network 304 can further have a first decoder section 310. The first decoder section 310 can have at least one decoder and can be set up for processing the code present in the first latent space 308 and outputting first digital output data 312. The dimension of the first digital output data 312 can correspond to the dimension of the first digital training data 302. In other words, the first decoder section 310 can increase the dimension of the code present in the first latent space 308 to the dimension of the first digital training data 302. The first decoder section 310 can reconstruct the first digital training data 302 from the code output by the first encoder section 306. The processor 108 can be set up for determining a first loss value 314 by comparing the first digital output data 312 with the first digital training data 302. The first sub-neural network 304 can be trained in a way that the first encoder section 306 and the first decoder section 310 are adapted. The first sub-neural network 304 can be adapted such that the first loss value 314 is minimized. That is: the trained first sub-neural network 304 can output a code describing digital image data in the first latent space 308, the digital image data describing the first context, and the trained first sub-neural network reconstructs the code describing digital image data in the first latent space 308 into digital image data.

[0062] Figure 3BA processing system 300B for training a second sub-neural network is shown according to different embodiments. This processing system 300B can have a storage device 106 for storing digital image data 204, such as second digital training data 322. Said second digital training data 322 can describe a second context, said second context being different from the first context.

[0063] In different embodiments, the first context and the second context can be distinguished in a territorial and / or intrinsic manner, as mentioned above.

[0064] This processing system 300B can also have said at least one processor 108. Said processor 108 implements at least a part of a second sub-neural network 324. Said second sub-neural network 324 is set up for processing second digital training data 322. Said second sub-neural network 324 can be an autoencoder network. The architecture of the second sub-neural network 324 can substantially correspond to the architecture of said first sub-neural network 304. Said second sub-neural network 324 can have a second encoder section 326 and a second decoder section 330, wherein the second encoder section 326 can generate a code in a second latent space 328 based on the second digital training data 322 and wherein said second decoder section 330 can reconstruct a code present in the second latent space 328. That is: said second decoder section 330 can generate second digital output data 332, wherein the dimensionality of the second digital output data 332 can correspond to the dimensionality of the second digital training data. Said processor 108 can be set up for determining a second loss value 334 by comparing the second digital output data 332 with the second digital training data 322 and for minimizing the second loss value 334 by adapting the second encoder section 326 and the second decoder section 330. That is: the trained second sub-neural network 324 can output a code describing digital image data in the second latent space 328, said digital image data describing a second context, and the trained second sub-neural network 324 can reconstruct the code describing digital image data in the second latent space 328 into digital image data.

[0065] Figure 4AA processing system 400A for training a first mapping is shown in accordance with various embodiments. The processing system 400A can have a storage device 106 for storing first digital training data 302. The storage device 106 can also store first digital data 402. The first digital data 402 can be semantically associated with the first digital training data 302 describing a first context. In accordance with various embodiments, the first digital data 402 has a plurality of text strings, where the text strings describe a scene shown in the first digital training data 302. For example, the first digital training data 302 can have a first digital image 302-1 and a second digital image 302-2, which describe a scene in a first context and the first digital data 402 can have a first text string 402-1 and a second text string 402-2, where the first text string is assigned to the first digital image 302-1 and the second text string is assigned to the second digital image 302-2. For example, the first digital image 302-2 of the first digital training data 302 shows a road, a parked car, and a pedestrian standing on the road and the first text string 402-1 describes the scene as "pedestrian standing on road." The second digital image 302-2 following the first digital image 302-2 shows the scene according to the example, where the pedestrian remains standing on the road and the second text string 402-2 describes the scene as "pedestrian remains standing."

[0066] The processing system 400A can also have at least one processor 108. The processor 108 implements at least a portion of a trained first sub-neural network 304. The first encoder section 306 of the trained first sub-neural network 304 can output a code that describes the first digital training data 302 in a first latent space 308. The processor 108 also implements at least a portion of a first mapping 404. The first mapping 404 can map first digital data 402 to the first latent space 308. In other words, the first mapping 404 can process the first digital data 402 and can output a code that describes the first digital data 402 in the first latent space 308. That is: the first encoder section 306 can output a first digital training data code 406 that describes the first digital training data 302 in the first latent space 308, and the first mapping 404 can output a first digital data code 408 that describes the first digital data 402 in the first latent space 308. In other words, the first mapping 404 assigns the first digital data code 408 in the first latent space 308 to the first digital training data code 406. The processor 108 can be set up to determine a first mapping loss value 410 by comparing the first digital training code 406 to the first digital data code 408. The first mapping loss value 410 can be determined based on a loss function. The loss function can be each type of loss function, such as each type of loss function based on a regression model.

[0067] The first mapping 404 can be trained in a manner that adapts the first mapping 404, where the first mapping can be adapted such that the first mapping loss value 410 is minimized. That is: the trained first mapping 404 can output a code that describes digital text data in the first latent space 308, where the digital text data has a text string, and the code describes digital image data that shows a first context, and where the code that describes the digital text data is assigned to the code that describes digital image data in the first latent space 308. The first mapping 404 can be a third sub-neural network.

[0068] The processing system 400A can also have at least one additional first neural network that is set up to generate at least a portion of the first digital data 402 (e.g., all of the first digital data) using the first digital training data 302 that describes the first context.

[0069] Figure 4BA processing system 400B for training a second mapping is shown in accordance with various embodiments. The processing system 400B can have a storage device 106 for storing the second digital training data 322. The storage device 106 can also store second digital data 422. The second digital data 422 can be semantically associated with the second digital training data 322 describing a second context. In accordance with various embodiments, the second digital data 422 has a plurality of text strings, where the text strings describe scenes shown in the second digital training data 322. For example, the second digital training data 322 can have a first digital image 322-1 and a second digital image 322-2 that describe scenes in a second context and the second digital data 422 can have a first text string 422-1 assigned to the first digital image 322-1 and a second text string 422-2 assigned to the second digital image 302-2. For example, the first digital image 322-2 of the second digital training data 322 shows substantially the same scene as the first digital image 302-2 of the first digital training data 302, that is, a road, a parked car, and a pedestrian standing on the road, and the first text string 422-1 of the second digital data 422 correspondingly describes the scene as "pedestrian standing on road." The second digital image 322-2 following the first digital image 322-2 of the second digital training data 322 shows a scene in the second context according to one example, where the pedestrian is crossing the road and the assigned second text string 422-2 describes the scene as "pedestrian crossing road."

[0070] The processing system 400B can also have at least one processor 108. The processor 108 implements at least a portion of a trained second sub-neural network 324. A second encoder section 326 of the trained second sub-neural network 324 can output a code that describes the second numerical training data 322 in a second latent space 328. The processor 108 also implements at least a portion of a second mapping 424. The second mapping 424 can map second numerical data 422 into the second latent space 328. In other words, the second mapping 424 can process the second numerical data 422 and can output a code that describes the second numerical data 422 in the second latent space 328. That is, the second encoder section 326 can output a second numerical training data code 426 that describes the second numerical training data 322 in the second latent space 328, and the second mapping 424 can output a second numerical data code 428 that describes the second numerical data 422 in the second latent space 328. In other words, the second mapping 424 assigns the second numerical data code 428 in the second latent space 328 to the second numerical training data code 426. The processor 108 can be set up to determine a second mapping loss value 430 by comparing the second numerical training code 426 to the second numerical data code 428. The second mapping loss value 430 can be determined based on a loss function. The second mapping 424 can be trained in a manner that adapts the second mapping 424, where the second mapping 424 is adapted such that the second mapping loss value 430 is minimized. That is: the trained second mapping 424 can output a code that describes numerical text data in the second latent space 328, where the numerical text data has a text string, and the code describes numerical image data, where the numerical image data shows a second context, and where the code that describes the numerical text data is assigned to the code that describes the numerical image data in the second latent space 328. The second mapping 424 can be a fourth sub-neural network.

[0071] The processing system 400B can also have at least one additional second neural network that is set up to generate at least a portion (e.g. all of the second numerical data) of the second numerical data 422 using the second numerical training data 322 that describes the second context.

[0072] Figure 5A processing system 500 for training a third mapping is shown in accordance with various embodiments. The processing system 500 can have a storage device 106 for storing digital image data 204 and digital text data, where the digital text data describes the digital image data 204. The storage device 106 can store third digital training data and third digital data. The third digital training data can have digital training data describing a first context and digital training data describing a second context. The third digital data can have digital data semantically associated with the digital training data describing the first context and digital data semantically associated with the digital training data describing the second context. In accordance with various embodiments, the digital training data describing the first context has at least a subset of the first digital training data 302 (e.g., all of the first digital training data) and the digital data semantically associated with the digital training data can have first digital data 402 assigned to the first digital training data 302. In accordance with various embodiments, the digital training data describing the second context has at least a subset of the second digital training data 322 (e.g., all of the second digital training data) and the digital data semantically associated with the digital training data can have second digital data 422 assigned to the second digital training data 322.

[0073] The processing system 500 can also have at least one processor 108. The processor 108 implements at least a portion of the trained first neural network 304, at least a portion of the trained second neural network 324, at least a portion of the trained first mapping 404, and at least a portion of the trained second mapping 424. The processor 108 also implements at least a portion of the third mapping 502. The third mapping 502 can be a fifth sub-neural network. The trained second mapping 424 can output code describing the second digital data 422 in the second latent space 328. The first encoder section 306 of the trained first sub-neural network 304 can output code describing the first digital training data 302 in the first latent space 308 and the trained first mapping 404 can output code describing the first digital data 402 in the first latent space 308. The third mapping 502 can transform digital latent data of the first latent space 308, i.e., code describing digital training data or digital text data in the first latent space 308, into digital latent data of the second latent space 328, i.e., into code describing the data in the second latent space 328. In other words, the third mapping 502 can map code describing the first digital training data 302 in the first latent space 308 and code describing the first digital data 402 in the first latent space 308 to the second latent space 328.

[0074] The third mapping 502 can process: the code describing the first digital training data 302 in the first latent space 308; the code describing the first digital data 402 in the first latent space 308; and the code describing the second digital data 422 in the second latent space 328, and the third mapping 502 can output code describing digital training data in the second latent space 328. The second decoder section 330 of the second sub-neural network 324 can process the code describing the digital training data in the second latent space 328, wherein the digital training data describes the second context, and the second decoder section can output third digital output data 504, wherein the third digital output data 504 can have dimensions corresponding to the dimensions of the second digital training data 322. The processor 108 can be arranged to determine a third mapping loss value 506 by comparing the third digital output data 504 to the second digital training data 322. The third mapping 502 can be trained by adapting the third mapping 502, wherein the third mapping 502 is adapted such that the third mapping loss value 506 is minimized. That is: the trained third mapping 502 can output code describing training data in the second latent space 328, wherein the second decoder section 330 can output training data after processing the code, which can correspond to the second digital training data 322.

[0075] Figure 6A method 600 for training a neural network according to different embodiments is shown. The method 600 can include training a first sub-neural network 304 (in 602). The first sub-neural network 304 can have a first encoder section 306 and a first decoder section 310 and can be trained based on first digital training data 302 describing a first context. The first encoder section 306 can provide a mapping of the first digital training data 302 to a first latent space 308. The method 600 can include training a first mapping 404 (in 604). The first mapping 404 can map first digital data 402 semantically associated with the first digital training data 302 to the first latent space 308 and can be trained using the first digital training data 302 mapped into the first latent space 308 by the trained first sub-neural network 304. The method 600 can include training a second sub-neural network 324 (in 606). The second sub-neural network 324 can have a second encoder section 326 and a second decoder section 330 and can be trained based on second digital training data 322 describing a second context. The second encoder section 326 can provide a mapping of the second digital training data 322 to a second latent space 328. The method 600 can include training a second mapping 424 (in 608). The second mapping 424 can map second digital data 422 semantically associated with the second digital training data 322 to the second latent space 328 and can be trained using the second digital training data 322 mapped into the second latent space 328 by the trained second sub-neural network 324. The method 600 can also have training a third mapping 502 (in 610). The third mapping 502 can map digital latent data from the first latent space 308 to the second latent space 328. The third mapping 502 can be trained based on third digital training data and third digital data, where the third digital training data can include digital training data describing the first context and digital training data describing the second context, and where the third digital data includes digital data semantically associated with the digital training data describing the first context and digital data semantically associated with the digital training data describing the second context. The digital training data describing the first context can include at least a subset of the first digital training data (e.g., all of the first digital training data), and the digital data semantically associated with the digital training data can include the first digital data 402 assigned to the first digital training data 302. The digital training data describing the second context can include at least a subset of the second digital training data 322 (e.g., all of the second digital training data), and the digital data semantically associated with the digital training data can include the second digital data 422 assigned to the second digital training data 322.

[0076] Figure 7A A processing system 700A is shown for transforming digital data between different contexts according to different embodiments. The processing system 700A can have a storage device 106 for storing digital data 702, such as digital image data 204. The digital data 702 can include first context data 704, where the first context data 704 includes digital image data 204 that describes a first context. The digital data 702 can also include first context text data 706, where the first context text data 706 has a plurality of text strings that describe the first context and are assigned to the first context data 704. The digital data 702 can also include second context text data 708, where the second context text data 708 has a plurality of text strings that describe a second context.

[0077] The processing system 700A can also have at least one processor 108. The processor 108 implements at least a portion of a first neural transform network 710A. The first neural transform network 710A can have at least a portion of the trained first neural network 304 and at least a portion of the trained second neural network 324. The first neural transform network 710A can have the first encoder section 306 of the trained first sub-neural network 304, the second decoder section 330 of the trained second sub-neural network 324, the trained first mapping 404, and the trained second mapping 424. The first neural transform network 710A can also have a third mapping 502. The first encoder section 306 can process the first context data 704 and can output code that describes the first context data 704 in the first latent space 308. The first mapping 404 can process the first context textual data 706 and can output code that describes the first context textual data 706 in the first latent space 308. The second mapping 424 can process the second context textual data 708 and can output code that describes the second context textual data 708 in the second latent space 328. The third mapping 502 can process: code that describes the first context data 704 in the first latent space 308; code that describes the first context textual data 706 in the first latent space 308; and code that describes the second context textual data 708 in the second latent space 328, and the third mapping can output code that describes the numerical data in the second latent space 328. The second decoder section 330 can process the code that describes the numerical data in the second latent space 328 and can output the second context data 712 that describes the second context. That is: the first neural transform network 710A is capable of processing numerical data that describes a first context and numerical textual data that describes a first context and a second context and is capable of outputting numerical data that describes a second context.

[0078] The processor 108 can also be configured to process the second context data 712 and can output classified and / or segmented second context data 714A. The processor 108 can implement a first neural classification network, where the first classification network is configured to classify and / or segment numerical data.

[0079] Figure 7BA second processing system 700B for transforming digital data between different contexts according to different embodiments is shown. The processing system 700B can have a storage device 106 for storing digital data 702, such as digital image data 204. The digital data 702 can include second context data 712, where the second context data 712 includes digital image data 204 that describes a second context. The digital data 702 can also include first context text data 706, where the first context text data 706 has a plurality of text strings that describe a first context. The digital data 702 can also include second context text data 708, where the second context text data 708 has a plurality of text strings that are assigned to the second context data 712 that describes a second context.

[0080] The processing system 700B can also have at least one processor 108. The processor 108 implements at least a portion of a second neural transform network 710B. The second neural transform network 710B can have at least a portion of the trained first neural network 304, at least a portion of the trained second neural network 324, the trained first mapping 404, and the trained second mapping 424. The second neural transform network 706B can have the second encoder section 326 of the trained second sub-neural network 324 and the first decoder section 310 of the trained first sub-neural network 304. The second encoder section 326 can process the second context data 712 and can output a code that describes the second context data 712 in a second latent space 328. The first mapping 404 can process the first context text data 706 and can output a code that describes the first context text data 706 in a first latent space 308. The second mapping 424 can process the second context text data 708 and can output a code that describes the second context text data 708 in a second latent space 328. The second neural transform network 710B can also have at least a portion of an inverse third mapping 716, where the inverse third mapping 716 can correspond to an inverse mapping of the trained third mapping 502. That is, the inverse third mapping 716 can process a code that describes digital data that describes a second context in a second latent space 328, a code that describes digital text data in a second latent space 328, and a code that describes digital text data in a first latent space 308, and the inverse third mapping can output a code that describes digital data in a first latent space 308. In other words, the inverse third mapping 716 can map a code that describes digital data that describes a second context in a second latent space to a first latent space 308 using digital text data that describes a first context and the second context. The second decoder section 330 can process the second context data 712 that describes a second context and can output a code that describes the second context data 712 in a second latent space 328. The inverse third mapping 716 can process a code that describes the second context data 712 in a second latent space 328, a code that describes the first context text data 706 in a first latent space 308, and a code that describes the second context text data 708 in a second latent space 328, and the inverse third mapping 716 can output a code that describes digital data in a first latent space 308. The first decoder section 310 can process the code that describes digital data in a first latent space 308 and can output first context data 704 that describes a first context.That is, the second neural transform network 710B can process digital data describing the second context and digital text data describing the first and second contexts and can output digital data describing the first context.

[0081] The processor 108 can also be configured to process the first context data 704 and can output classified and / or segmented first context data 714B. The processor 108 can implement a second neural classification network, wherein the second classification network is configured to classify and / or segment digital data. The second classification network can correspond to the first classification network.

[0082] Figure 8 A vehicle 800 according to different embodiments is shown. The vehicle 800 can be a vehicle with an internal combustion engine, an electric vehicle, a hybrid vehicle, or a combination thereof. The vehicle 800 can also be a car, a truck, a ship, a drone, an airplane, etc.

[0083] The vehicle 800 can have at least one sensor (e.g., imaging sensor) 802 (e.g., sensor 102). The vehicle 800 can have a driving assistance system 804. The driving assistance system 804 can have a storage device 106. The driving assistance system 804 can have a processor 108. The processor 108 can implement a first neural transform network 710A and / or a second neural transform network 710B. The first neural transform network 710A can be configured to process digital data describing a first context and output digital data describing a second context. The second neural transform network 710B can be configured to process digital data describing the second context and output digital data describing the first context. According to different embodiments, the first neural transform network 710A and / or the second neural transform network 710B have been trained according to the method 600 for training a neural network, such that the first neural transform network 710B or the second neural transform network 710B can transform digital data describing the first or second context into digital data describing the second or first context.

[0084] The processor 108 can also be configured to classify and / or segment digital data output by the first neural transform network 710A and / or second neural transform network 710B.

[0085] The processor 108 can implement a neural classification network configured to classify and / or segment digital data output by the first neural transform network 710A and / or second neural transform network 710B.

[0086] Depending on the different embodiments, the classified and / or segmented digital data 714A, 714B has the intention of the traffic participant as a feature. This in combination with the transformation of the digital image data has for example the advantage that the intention of the traffic participant can be determined depending on the context of the situation, for example with respect to the context on the territory of a city, a region, a country, etc.

[0087] The driving assistance system 804 can be set up for controlling the means of transport 800 based on the classified and / or segmented digital data 714A, 714B. In other words, the driving assistance system 804 can be set up for processing the classified and / or segmented digital data 714A, 714B and can output at least one control instruction to one or more actuators of the means of transport 800 based on the classified and / or segmented digital data 714A, 714B.

[0088] That is, the driving assistance system 804 can influence the current driving behavior based on the context of the digital image data 204 and thus of the classified and / or segmented digital data 714A, 714B, for example can maintain or change the current driving behavior. For example, the driving assistance system 804 can determine that in the context of a situation a pedestrian tries to cross the road and can for example change the driving behavior such that the driving behavior is intervened for safety reasons, for example by emergency braking.

Claims

1. A method for training a neural network, wherein the method is performed by one or more processors, the method comprising: training a first sub-neural network with first digital training data describing a first context, wherein the first sub-neural network is set up as an autoencoder network and comprises a first encoder section and a first decoder section, and wherein the first encoder section provides a mapping of the first digital training data into a first latent space; training a first mapping of first digital data into the first latent space using the first digital training data mapped into the first latent space by means of the trained first sub-neural network, wherein the first digital data is semantically associated with the first digital training data; training a second sub-neural network with second digital training data describing a second context, wherein the second context is different from the first context, wherein the second sub-neural network is set up as an autoencoder network and comprises a second encoder section and a second decoder section, and wherein the second encoder section provides a mapping of the second digital training data into a second latent space; training a second mapping of second digital data into the second latent space using the second digital training data mapped into the second latent space by means of the trained second sub-neural network, wherein the second digital data is semantically associated with the second digital training data; training a third mapping of digital latent data from the first latent space into the second latent space using third digital training data and third digital data, wherein the third digital training data comprises: digital training data describing the first context, and digital training data describing the second context, wherein the third digital data comprises: digital data semantically associated with the digital training data describing the first context; and digital data semantically associated with the digital training data describing the second context, wherein the first digital training data, the second digital training data, and the third digital training data comprise digital image data.

2. The method according to claim 1, wherein the first digital training data, the second digital training data, and the third digital training data comprise digital sensor data.

3. The method according to claim 1, the method further comprising: generating the first digital data using an additional first neural network and first digital training data describing the first context; and / or generating the second digital data using an additional second neural network and second digital training data describing the second context; and / or generating the third digital data using an additional third neural network and digital training data describing the first context and the second context.

4. The method according to claim 1, the method further comprising: transforming digital data describing a first context into digital data describing a second context using a first neural transformation network, wherein the first neural transformation network is constituted by a first encoder section of the trained first sub-neural network, a trained first mapping, a trained second mapping, a trained third mapping for mapping from the second latent space to the first latent space, and a second decoder section of the trained second sub-neural network.

5. The method according to claim 4, further comprising: performing a classification and / or segmentation on the digital data describing the first context.

6. The method according to claim 1, further comprising: transforming digital data describing a second context into digital data describing the first context using a second neural transformation network, wherein the second neural transformation network is constituted by a second encoder section of the trained second sub-neural network, a trained first mapping, a trained second mapping, an inverse mapping of the trained third mapping from the second latent space to the first latent space, and a first decoder section of the trained first sub-neural network.

7. The method according to claim 6, further comprising: performing a classification and / or segmentation on the digital data describing the second context.

8. A device being set up for performing the method according to any one of claims 1 to 7.

9. A system having: the device according to claim 8; and a sensor being set up for providing the device with digital data.

10. A vehicle having: at least one sensor being set up for providing digital data; and a driving assistance system having a neural network trained according to any one of claims 1 to 3, wherein the neural network is set up for classifying and / or segmenting the digital data according to claim 5 or claim 7, and wherein the driving assistance system is set up for controlling the vehicle based on the classified and / or segmented digital data.

Images